none
Norman Scan Engine - Creation of Large number of nse_tmp files in c:\windows\temp on engine updates

    Question

  • Hi all - anyone know the purpose of the temporary files created by the norman scan engine in c:\windows\temp\ ?  Got around 30 of them some up to around 320Mb that are locked by forefront / norman scan engine.  Modified time on files update on a scan engine update.  The Engine updates are stored in the usual path %program directory%\Forefront\Engines\Norman\.

    Could understand having an extract location for the Norman cabs but there seems to be more temporary files than expected?  Also don't seem to see this behavior with the other scan engines.

    Wednesday, May 01, 2013 7:54 AM

All replies

  • Hi,

    I have exactly the same problem, running forefront for exchange 2010 with exchange 2007 on windows 2008.

    nsebin.def.XXXXXXXX.tmp are created on c:\windows\temp

    Each one is about 330mb.

    it's a big problem cause older files are not deleted so server get saturated very quickly.

    I v tried to delete the NORMAN folder and redownload the definitions, it didn't help.

    Please help us !


    • Edited by tomazzzi Thursday, May 02, 2013 11:01 AM
    Thursday, May 02, 2013 11:00 AM
  • Hi,

    Thank you for the post.

    You may delete all Forefront .tmp files that are located in temp folder and rebuild the Norman engine folder.

    Regards,


    Nick Gu - MSFT

    Friday, May 03, 2013 5:20 AM
    Moderator
  • Hi Nick,

    We are experiencing the same issue.

    Can you please give more detail on how to 'rebuild the Norman engine folder' please as I've found this suggested as a solution on a number of posts but can't find any information on how to do this?

    Many thanks,

    R

    Friday, May 03, 2013 8:25 AM
  • Hi Rossyd75,

    Thank you for the update.

    To rebuild the Norman engine folder, just do as below:

    - Rename the folder C:\Program Files (x86)\Microsoft Forefront Protection for Exchange Server\Data\Engines\x86\Norman\ to Norman.old.
    - From the console, select the Norman engine and force an update.

    Regards,


    Nick Gu - MSFT

    Friday, May 03, 2013 8:29 AM
    Moderator
  • Hi Nick,

    Thank you for your reply.  

    For anyone else who, like me, had trouble finding where to update the Norman engine, it's under Policy Management > Global Settings > Advanced Options > Update Scheduling section.  You can right click the Norman engine there and select 'update now'

    Many thanks,

    Ros


    • Edited by Rossyd75 Friday, May 03, 2013 8:56 AM
    Friday, May 03, 2013 8:42 AM
  • Thanks Nick - we can manually intervene on the servers, rebuild norman and delete the tmp files - but when you have a few hundred servers how can we prevent this happening/or ensure that the temporary files are removed automatically?
    Friday, May 03, 2013 11:03 AM
  • Hi Nick,

    I'm sorry but your steps above haven't solved my issue.  There's still a large 330MB temp file (e.g. nsebin.def.98d445b7ccb256ff.tmp) being created in C:\Windows\Temp on my Exchange server every day.

    Any further help anyone can give is greatly appreciated.

    Ros

    Sunday, May 05, 2013 11:31 AM
  • Dear All

    recently i am having the same problem, i dont know whats going on i have more than 10 files of nsebin.def.XXXXX on the TEMP folder and its filing up the C:\ drive , which cause sometimes the mail queue to be on retry status and mails having EventID of DEFER.

    please help us how to solve this problem ASAP.

    Monday, May 06, 2013 6:37 AM
  • Hi,

    Thank you for the post.

    You may delete all Forefront .tmp files that are located in temp folder and rebuild the Norman engine folder.

    Regards,


    Nick Gu - MSFT

    Hi,

    Like i said in my initital post i already did that and it didn't solve the issue.

    THE REBUILD OF THE NORMAN FOLDER DOESN'T SOLVE THIS ISSUE.

    We need a real solution, it's a critical problem, it's filling the C drive very quicly and forefront need at least 1gb to work properlly.

    Im gonna disable NORMAN until a true solution is found.




    • Edited by tomazzzi Monday, May 06, 2013 12:01 PM
    Monday, May 06, 2013 7:07 AM
  • On Mon, 6 May 2013 07:07:38 +0000, tomazzzi wrote:
     
    [ snip ]
     
    >We need a real solution, it's a critical problem, it's filling the C drive very quicly and forefront need at least 1gb to work properlly.
    >
    >Im gonna disable NORMAN until a true solution is found.
     
    Could this be one of those situations where a file-level AV is
    scanning the file at the same tim FPE is trying to remove the file?
     
    If you have a flile-level AV, try excluding the \windows\temp
    directory from being scanned.
     
    If that turns out to be a "solution" I'd like to know how to instruct
    FPE to NOT use the default temp\tmp directories and to use some other
    directory instead!
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    Monday, May 06, 2013 10:13 PM
  • All.

    We started seeing the same issue on 4-25-13. As a temporary workaround are we suggesting changing the "Intelligent Engine management" from "automatic" to "manual" and then disable the "Norman" engine, followed by deleting the tmp files? 

    Have anyone escalated this issue to premier support yet?

    Monday, May 06, 2013 10:15 PM
  • For what it's worth, we're still using Antigen 9.2 on our Exchange 2003 front end server, and it's having exactly the same issue. I've disabled Norman.

    -Peter

    Monday, May 06, 2013 11:03 PM
  • Definitely seems to be a recent change with the Norman update distributions.  Even if you clear out all the temporary files and rebuild norman folder the problem starts to occur again.
    Tuesday, May 07, 2013 2:02 PM
  • I have this same issue occurring with our Exchange 2010/Forefront Protection 2010 for Exchange...   315MB per day consumed on C: drive for every server.

    Subscribing to the discussion for a solution..



    Jason Meyer

    Tuesday, May 07, 2013 2:45 PM
  • Exactly the same problem here with 2010 SP3 & FPE.  Just caught it in time...  Not impressed at all. :|

    Subscribing...

    Tuesday, May 07, 2013 4:07 PM
  • Growing and growing...

     

     


    • Edited by MooEy Tuesday, May 07, 2013 11:54 PM
    Tuesday, May 07, 2013 11:53 PM
  • Same problem here too.  Lots of these files filling up C: drives on Exchange 2010 SP3 servers with fully patched Forefront for Exchange.  Going to completely disable the Norman scanner.
    Wednesday, May 08, 2013 12:55 AM
  • Same issue's on exchange 2007 clusters and hub servers, disabled Norman updates. Workaround by renaming Norman engine folder and removing .tmp files is no long term solution. Anyone in direct contact with MS?
    Wednesday, May 08, 2013 6:00 AM
  • another "me too" client ...

    hope to get a "real solution" here, instead of deleting the temp-files each night by scheduled script...

    Ove

    Wednesday, May 08, 2013 7:39 AM
  • Hello,

    I've just realised that the issue is occurring on my SharePoint 2010 server which has ForeFront Protection for SharePoint as well as my Exchange 2010 server (with ForeFront Protection for Exchange).

    Just wanted to alert you all in case you haven't noticed that your SharePoint server is running out of space on its C:\ drive too!

    What exactly is the Norman engine?  What are the risks of disabling the engine?

    Ros

    Wednesday, May 08, 2013 8:20 AM
  • Hi!

    I run MS exchange 2010 and forefront on a 60gb C:\ drive.
    The first time i realize the problem is 2013. 04. 17.
    I check the update-history, the ForeFront has an update that day, and my TEMP folder is started groving.
    Now we delete the deletable files manually and waiting for correct solution.
    If  we cant find any useful solution, i'll contact the retail support for more information.

    and now.... subscribing.
    Regards,
    Gábor 
    Wednesday, May 08, 2013 8:37 AM
  • I've created a post about this issue in the Microsoft Partner Network Supported Forums.  If anyone in Microsoft comes back with a solution I'll post it here.

    Ros


    • Edited by Rossyd75 Wednesday, May 08, 2013 8:41 AM
    Wednesday, May 08, 2013 8:41 AM
  • Hi there,

    Thank you for the update.

    I am trying to involve senior member familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

    Regards,


    Nick Gu - MSFT

    Wednesday, May 08, 2013 8:46 AM
    Moderator
  • Hi

    i have a script written that delete this files automatically on all my servers. This is the main line of the script:

    get-ChildItem $env:WINDIR\temp\*.* -include nsebin* | foreach ($_) {remove-item $_.fullname}

    flo
    Wednesday, May 08, 2013 8:52 AM
  • I too am experiencing this issue.  Like Rossyd75, I am seeing the issue on our Exchange servers as well as our SharePoint servers......all with Forefront Protection installed.  Disabling Norman as well.
    Wednesday, May 08, 2013 3:26 PM
  • We would definitely appreciate an answer as well as we are seeing issue too.

    tb

    Wednesday, May 08, 2013 6:38 PM
  • Hello Everyone,

    A new Norman engine (version 7.x) was released on 4/26/13 for the Forefront/Antigen products.  We are currently investigating an issue where it appears that Norman's nsebin.def.tmp files are not being removed from the Windows Temp directory after engine updates therefore causing the system drive to increase in space over time.  For the time being you can periodically remove only the older nsebin.def files from the Windows Temp directory (dates older than the most recent 1-2 nsebin.def.tmp files which will currently be in use) to reclaim some disk space or disable the Norman engine entirely and delete all the nsebin.def and nse_tmp.tmp files from the Windows Temp directory.

    In addition, if there are any nvcbin.def files in the Windows Temp directory (these should have dates older than 4/25/13) consuming space you can safely delete these files as well as they are from the previous Norman 6.x version and are no longer in use.

    We will provide an update once we have a better understanding of the issue and a proper resolution.

    Thank you.

    • Proposed as answer by Ed051042 Thursday, May 09, 2013 12:31 AM
    Wednesday, May 08, 2013 7:55 PM
  • Hello Everyone,

    Someone from Microsoft has replied to my post in the Partner Supported Forum.  I've copied the post below.  I think the solution he's suggesting is pretty much the same as what we've all been doing when we've followed Nick's advice to delete the temp files and rebuild the Norman folder, but it's a slight variation on it.  I'm not in the office again until Monday so can't try it out.  If someone else could try following these steps and feed back the results here, I'll reply to the post in the Partner Supported Forum.

    Many thanks,

    Ros

    Reply from MS follows:

    As we can see, this is a BUG and the BUG # is 94372.

     

    Symptom:

    Norman Scan Engine - Creation of Large number of nse_tmp files in c:\windows\temp on engine updates

    Cause:

    Norman Engine

    Resolution:

    1. Stop FSCController / Antigen Service

    2. Verify all scanning processes are down

    3. Delete all of the above Norman related temp files from the C:\Windows\Temp directory

    4. Restart FSCController or Antigen Service

     

    Note that upon start up, the Norman engine will re-create some of the .tmp files, so you will likely see ~1gb or more still in use, but this process will delete any orphaned .tmp files. 

    Thursday, May 09, 2013 8:29 AM
  • Hello Everyone,

    A new Norman engine (version 7.x) was released on 4/26/13 for the Forefront/Antigen products.  We are currently investigating an issue where it appears that Norman's nsebin.def.tmp files are not being removed from the Windows Temp directory after engine updates therefore causing the system drive to increase in space over time.  For the time being you can periodically remove only the older nsebin.def files from the Windows Temp directory (dates older than the most recent 1-2 nsebin.def.tmp files which will currently be in use) to reclaim some disk space or disable the Norman engine entirely and delete all the nsebin.def and nse_tmp.tmp files from the Windows Temp directory.

    In addition, if there are any nvcbin.def files in the Windows Temp directory (these should have dates older than 4/25/13) consuming space you can safely delete these files as well as they are from the previous Norman 6.x version and are no longer in use.

    We will provide an update once we have a better understanding of the issue and a proper resolution.

    Thank you.

    Thanks Ryan - Presumably this issue did not come up in the pre-release integration testing between Microsoft / Norman or was this engine update not notified to MS? 

    Can we get an idea on timescales for a fix?  In an environment running a large number of affected servers - should we carry on firefighting or look to disable Norman in the short term?

    Thursday, May 09, 2013 9:42 AM
  • Done this but only in a non production environment as this will require a restart of the MS Exchange Transport Service.  After completely clearing the Norman files and restarting one NSE_tmp file is created it increasing in size and then is finally deleted and you are left with one nsebin.def file.  However, on checking console Norman def version / engine version shows as 0.0.0 / 0000000.

    Once the console was opened Norman started downloading definition update.  8 temp files of various sizes created.  It is also creating a ton of 0kb av****.tmp files and syb****.tmp files

    It then passes downloading stage and starts show installing (in the GUI) it then creates a further a load more nse_tmp files of various sizes (giving a running total of 20 nse_tmps from 0kb to 326,484kb)

    When the GUI finally stops showing the installing - now have 34 nse_tmp files.  Program log just indicates normal update - nothing unusual.

    Thursday, May 09, 2013 10:40 AM
  • Hi Danger_Mouse1979,

    Thanks for doing that.  I guess we now just have to leave it for a few days to see if it's clearing the daily temp files out of C:\windows\temp or if the issue still exists.  My money is on the issue still existing.

    Ros

    Thursday, May 09, 2013 11:47 AM
  • Adding to the list of people reporting the problem. I'm experiencing it on both my Hub Transports running FPE 2010 on Exchange 2010 SP2.
    Thursday, May 09, 2013 3:21 PM
  • To reply to some responses on my post yesterday:

    1) Yes, testing was done on the new Norman engine.  The issue did not surface during the testing.

    2) We do not have an ETA on the fix but I can assure you that it's being worked on with urgency.

    3) You do not need to cycle services to mitigate this.  You would only need to stop services if you wanted to delete ALL the Norman temp files.  That would include the files currently in use by Norman so they would be locked and you'd have to stop services to remove them.  As I mentioned you can remove the older nsebin.def files to clear disk space as well as any older version 6.x files (nvcbin.def), if they exist, because they are no longer in use and not needed.  You will not be able to remove the most recent nsebin.def file because it's in use by the scanning process(es).  A new nsebin.def file would just be downloaded again anyway on the next Norman update.  You should not be removing any of the nse_tmp files.

    4) The Norman engine WILL have .tmp files in the Windows Temp directory that will consume a decent amount of space.  This is expected.  The number of nse_tmp files will correlate to the amount of scanning processes that you have running on the server.  So if you've increased the process count or if the server has both the mbx and hub role installed there will be more scanning processes running therefore more nse_tmp files.  There should only be one nsebin.def file.  This is the issue that we're trying to resolve.  The problem is the older nsebin.def files are not being properly removed after a new Norman engine definition update is downloaded and after a while the space consumed can grow significantly.  The engine usually updates 1-2 a day and each nsebin.def file is ~326mb so after 2-3 days you have a ~gig of space taken up by the nsebin.def files that are not needed, etc.

    5) Possible mitigations at this time are:

    • Periodically delete the older nsebin.def files.  This will allow you to continue using the engine while you wait for the fix.
    • Disable Norman engine updates as that would cease the new downloads of the problem nsebin.def files.  You'd still want to remove the older nsebin.def files.
    • Disable the Norman engine entirely and wait for the fix.  To disable the engine from use and clear all the Norman .tmp files you would disable the engine from all scan jobs, disable the Norman engine updates, stop services, remove ALL Norman tmp files, start services.

    I'll update this thread when we have news for you all.  Thanks for your patience.

    Thanks,

    -Ryan

    Thursday, May 09, 2013 4:04 PM
  • Same issues on all Exchange servers here, waiting not so patiently for a fix to be released.  Will try the remedy posted by RossyD, but to err on the side of caution, will be performing off-hours.

    Though we will abandon Norman if our disk space is chewed up right away again like Danger_Mouse1979 experienced, and disable it until the problem is resolved.

    Thursday, May 09, 2013 4:28 PM
  • Having the same issue here on my Edge servers. They are virtualized, so I was able to expand the drives to prevent them from going into a backpressure state.

    Hopefully this issue will be resolved soon!

    Thursday, May 09, 2013 7:19 PM
  • I'm also having the same issue. Is there a resource at Microsoft that we all should contact to let them know that the problem is a widespread one?
    Friday, May 10, 2013 1:43 PM
  • I'm also having the same issue. Is there a resource at Microsoft that we all should contact to let them know that the problem is a widespread one?

    The issue has been posted on the Microsoft Partner Network too, there you can mark it as "Me too" so Microsoft can see how many is experiencing it.

    /Jesper

    Friday, May 10, 2013 2:27 PM
  • Thanks Jesper. Can you provide a link to the site, please
    Friday, May 10, 2013 4:33 PM
  • Of course, forgot that in my previous post.

    http://partnersupport.microsoft.com/en-us - log on and search for nsebin, should be the only topic regarding this.

    Friday, May 10, 2013 7:23 PM
  • We (Microsoft) are aware that the issue is widespread.  We do not need any more additional customer information and there are mitigation steps above that can be used while you wait for the fix.

    Once the fix has undergone sufficient testing it will be released and we do not believe that there will be any additional steps that you will need to take.  We don't anticipate there will be a long turn around in getting the fix out.

    Friday, May 10, 2013 7:54 PM
  • Thank you for your support!
    I have the problem too, but additionally I have the problem that the norman scan engine is disabled and even I have the temp files and I can`t delete them.

    More important is, that I have only norman as scan engine. How can I add additional engines? Under Antivirus I have no engines to select, only norman!

    (E2k3 and Antigen 9)

    Thanks!

    Monday, May 13, 2013 9:01 AM
  • So what is the status of this.  I walked in this morning to find our 4 CAS server dangerously low on space.  Two weeks seems like a long time to fix a bug of this magnitude.  What is the ETA?  What is the holdup?
    Monday, May 13, 2013 2:04 PM
  • just disable NORMAN engine while waiting for the fix.
    Monday, May 13, 2013 2:44 PM
  • Now disabled norman across the estate.  However, have one server where I have had to bypass forefront completely - 1000's of sybXXX.tmp files created a minute.
    Monday, May 13, 2013 3:01 PM
  • Damn.

    hopefully for me it's only the NORMAN engine !

    Monday, May 13, 2013 3:06 PM
  • Actually - this problem seems to have been with corrupt wormlist engine.  Pushed the last update back to the server from manager (August 2012) and it stops creating the files.  Interesting though as I thought we had every part of wormlist disabled.
    Monday, May 13, 2013 3:24 PM
  • Thanks Jesper
    Monday, May 13, 2013 4:21 PM
  • So the only nsebin.def file we should leave on our server would be the latest ONE? I've already deleted three of these files but I have two left from yesterday's date. Why is the engine creating two files for the same day? Thanks for your help in advance
    Monday, May 13, 2013 4:42 PM
  • Yes, the only nsebin.def file you need to keep is the most recent nsebin.def file because that is the file that is currently in use by the Norman engine.  You may have more than 1 nsebin.def file for a given day because the engine may have updated more than once that day.
    Monday, May 13, 2013 4:50 PM
  • Any updates yet???
    Tuesday, May 14, 2013 1:05 PM
  • Subscribing as I have the issue as well.
    Tuesday, May 14, 2013 6:43 PM
  • I'm seeing the same thing, hope to see a fix soon.

    Tuesday, May 14, 2013 7:10 PM
  • We are hoping to be able to release the fix tomorrow.  The fix is currently undergoing testing and obviously if something comes up in that testing the release could be further delayed.  We're hopeful that will not occur.

    I'll update this post once the fix is released.

    Thanks,

    -Ryan

    Tuesday, May 14, 2013 7:19 PM
  • Ryan,

    Great thank you for the update. Are you anticipating that we have to manually clean up old temp files or will that get taken care of with the "fix"?

    Tuesday, May 14, 2013 8:02 PM
  • Another me too !

    Thanks Ryan. We are waiting for the fix.

    Wednesday, May 15, 2013 7:18 AM
  • this brought down incoming mail on my edge box until I manually cleaned up the files....hoping for the fix soon.
    Wednesday, May 15, 2013 1:07 PM
  • All,

    About 1.5 hours ago the fix for this issue was put into our production update servers for your consumption.  The new Norman update will contain the fix to remove *almost all* of the previous nsebin.def files that have been created in the Windows\Temp directory since the release of the new engine a little over 2 weeks ago. These are the files that were causing disk space issues in the Temp directory over that time.

    Here are some details for you:

    1. What do you need to do?:
      1. Just wait for the next scheduled Norman engine update or manually initiate one.  In fact, it may have already taken place depending on your current engine update schedule.
      2. Possibly remove a few nsebin.def files manually that remain after the update takes place.  This will be a one-time action after the update and depends on if you’ve disabled the Norman engine updates over the past few days while you waited for the fix.  If you have disabled the Norman engine updates then you should not need to clean up anything manually because it hasn’t been updating and generating newer files.  The fix will remove all of these older files.  If the engine updates were never disabled there will be a few nsebin.def files that were created that you can safely remove.  The fix is unable to remove these few more recent files.  Services do not need to be stopped to remove these older files because they are no longer in use.
      3. Nothing additional should need to be done after that moving forward.
    2. What the fix will do?:
      1. It will create a new directory under the Windows\Temp directory called nsetmp.  This will be the new directory for ALL Norman engine related files moving forward.
      2. As stated above the fix will remove almost all of the older problematic nsebin.def files (~325mb each) from the Windows\Temp directory.  The nsebin.def files from the past day or so will not be removed by the fix BUT are safe for you to delete manually after the update takes place.  If you have not cleaned up any of these files yet then this will be a significant amount of files and disk space that will get cleared. 
      3. No new nsebin.def files will be created in the Windows\Temp directory after this fix is in place.  The only nsebin.def files you should see being created from here on is the one current nsebin.def file that will be in the Windows\Temp\nsetmp directory after each update.  That is the file that will be in use by the engine.  The older nsebin.def files in the nsetmp directory will get removed properly on each subsequent successful Norman update.
      4. If you’re running Windows Server 2003 you will not see the nsetmp directory and the nsebin.def files will continue to be written to the Windows\Temp directory.  You will not need to take any steps as the previous nsebin.def files will be properly removed.
      5. The fix will not remove any Norman version 6.x files if by chance they exist.  If you see any of these files they too are safe to delete at any time.  These would be the nvcbin.def.xxx.tmp files.
    3. How do I know I have the fix in place?
      1. The new Norman engine version will be 7.1.8.  You will see that as the Engine version value for the Norman engine in the UI in FPE.  In FSE you’ll see 7.1 for the engine version in the UI.  You might need to check the details of the nse32.dll in the Norman engine bin directory to confirm that you have the fix.  The details of that .dll will show a version of 7.1.8.0.  However, if the Norman engine has updated successfully any time after this post you can be fairly certain you have the new update.

    I apologize if any of the above is confusing at all.  To sum it up the fix is in place, the next Norman update will correct the issue and will no longer create nsebin.tmp files that do not get removed on new Norman updates.  There *may* be a need for you to manually clean up a few more recent nsebin.tmp files if they exist but this would only be a one-time only action.

    Please let me know if you have any concerns and thank you very much for the patience as we diagnosed the issue and tested the fix.

    Thanks,

    -Ryan



    • Proposed as answer by HUNAL Wednesday, May 15, 2013 8:30 PM
    • Edited by Ryan McGrath - MSFT Wednesday, May 15, 2013 8:35 PM
    Wednesday, May 15, 2013 8:26 PM
  • What about the nse_tmp{xxxxx}.tmp files in C:\Windows\Temp, can those be safely deleted?
    Wednesday, May 15, 2013 8:57 PM
  • Hi,

    one of my servers has the updated Norman engine 7.1.8 and looks good so far. I could delete the tmp files without any problems out of the windows temp directory and no other tmp files have been created.

    Greetings

    Christian


    Christian Groebner MVP Forefront

    Wednesday, May 15, 2013 9:01 PM
  • There will be new nse_tmp{xxxxx}.tmp files in the nsetmp directory.  If the engine has updated successfully and you see the Windows\Temp\nsetmp directory with the new nsebin.def file and newer nse_tmp{xxxxx}.tmp files then the nse_tmp{xxxxx}.tmp files in the Windows\Temp folder would be safe to delete, yes. 

    If you do not see the nse_tmp{xxxxx}.tmp files it might be because nothing has been scanned yet if you recently restarted services.  These nse_tmp{xxxxx}.tmp files are usually created after scanning processes come online.

    Wednesday, May 15, 2013 9:08 PM
  • Great, thanks Christian.
    Wednesday, May 15, 2013 9:14 PM
  • All works great after fix. Thanks so much :)
    Thursday, May 16, 2013 8:02 AM
  • Everything went fine after the update in Windows 2008 Server R2, Exchange 2010 and Forefront Protection 2010 for Exchange Server.

    A nsebin.def file appeared on Windows\temp directory after reenabling and updating Norman and also a nsetmp directory. nsebin.def file on windows\temp could be deleted after a while and at this moment there is only one nsbin.def under nsetmp directory.

    Thank you for this fix.

    David.

    Thursday, May 16, 2013 9:05 AM
  • Fix works. Thanks for addressing this quickly!
    Thursday, May 16, 2013 9:06 AM
  • Thanks for the fix, it works fine. (:
    Thursday, May 16, 2013 9:53 AM
  • Thanks for the fix, however, My Norman Scan engine is enabled and I it still hasn't updated to the newest edition. I have tried to update it manually but no changes. I'd appreciate any assistance or suggestions. Thanks again
    Thursday, May 16, 2013 12:59 PM
  • Is the new Norman engine update also for Antigen 9 Sp2 ? We don't see any updates available yet.


    Georgi Petkov MSFT

    Thursday, May 16, 2013 6:13 PM
  • I have one server that has the update but does not have the 'nsetmp' directory. Is there a service or something hung that needs to be restarted?
    Thursday, May 16, 2013 6:54 PM
  • Thanks for the feedback guys.  Let me address some questions I've seen.

    1. Yes, the new Norman update is applicable to Antigen as well.  It's applicable to all our products using the Norman scan engine.  Just make sure the engine is up to date and check the engine version as I mention above.  There was already a Norman update today so if it checked and updated successfully any time after yesterday afternoon you should have the fix. 
    2. If you're seeing that the Norman engine has not updated it's probably not related to the fix itself but just a general updating issue.  Perhaps the update is timing out or there is a connection issue, etc.  You might need to increase the engine download timeout or there is another issue that is keeping it from updating.  As you can see from other posts quite a few customers are having success and this is probably not the forum for general updating issues.  Please check your logs and/or open a support ticket.
    3. It is possible that you have a Windows 2008 server that is updating and you DO NOT have the nsetmp directory.  This would be if "NetworkService" has rights to the Windows\Temp directory.  If that is the case then you would see the Norman engine files written directly to the Windows\Temp directory.  You would then NOT see the nsetmp directory created at all.  The clean-up will take place correctly in either case.  This was the primary reason for the actual issue we had.  Windows 2008\2008 R2 does not grant NetworkService rights to the Windows\
      Temp folder.  If you changed things so that NetworkService has access to the Windows\Temp directory (or another application changed it) you most likely never saw the issue to begin with.  By default on Windows 2003 NetworkService has rights to the Windows\Temp directory so that is why it was not seen there.

    Thanks,

    -Ryan

    Thursday, May 16, 2013 7:27 PM
  • Ryan,

    In regards to your third point, I have two servers that have the update. Both of these servers are Windows 2008 R2. The "NetworkService" account does not show up in the ACL for Windows\Temp but there is no 'nsetmp' directory. Why would this be?

    Thanks

    Thursday, May 16, 2013 8:18 PM
  • Arkiados06,

    1) What product and version are you running?  Forefront, Antigen? 

    2) What exactly do you see in the Windows\Temp directory?  Do you see Norman engine update files? 

    3) What is the update version for Norman?

    4) Have services been started?

    5) You're not seeing an issue correct?  You just don't see the nsetmp folder?

    Thanks,

    -Ryan

    Thursday, May 16, 2013 8:46 PM
  • Thanks for the quick reply.

    1. Forefront 11.1.401.0

    2. I cleaned up earlier this afternoon but only see one nsebin.def file at this time.

    3. Forefront shows EV 7.1.8, DV 7.1#4384, and nse64.dll 7.1.8

    4. All Forefront services are started.

    5. Not at this time but I am waiting to see what happens when another definition update comes about. No 'nsetmp' folder.

    Thursday, May 16, 2013 9:08 PM
  • Ok, you're running Forefront for SharePoint RU1

    You only see one nsebin.def files.  When is that file dated? Do you see any nse_tmp files?  If not, it doesn't appear that anything has been scanned or Norman hasn't loaded into any scanning process(es).  However, that should not keep the nsetmp folder from being created.  This should be created on the engine update.  Maybe you've hit a scenario where the it has access to the Windows\Temp directory and doesn't need the nsetmp folder.  I would try to make sure that the scanning processes get loaded.  Upload a file to the site to try to load one to see if you get the nse_tmp files.  I'll try to find that particular setup to check out.

    Thursday, May 16, 2013 9:37 PM
  • The only *nse* file in Windows\Temp is 'nsebin.def.b65fc723392b991b.tmp' from 5/16 @ 09:29.
    Thursday, May 16, 2013 9:44 PM
  • The fix doesn't work at all here.

    Sure the folder "nsetmp" is created in c:\windows\temp  but the drive is still filed with a lot of files and get saturated very quickly.

    My drive went from 6gb to less than 500mb in about 5 minutes

    i had to disable NORMAN again.

    We need a prpoper fix :)

    Friday, May 17, 2013 9:22 AM
  • Your drive consumed >5.5 gigs of data in 5 minutes? Was this after a Norman update that this occurred?

    What files is the drive filled with that are taking up this space? What are the names and timestamps of these files?  Are these files in the nsetmp directory of Windows\Temp?

    What version of Forefront, what OS and how many scanning processes are you running with?

    Friday, May 17, 2013 11:42 AM
  • Hi,

    thx for the prompt answer.

    Yeah the drive consumed more than 5gig in 5 minutes just after enabling and updating the NORMAN scanner.

    Im running exchange 2007 (08.03.0298.001) on windows 2008 sp1 with Forefront for exchange 2010 ( 11.0.713.0).

    My processcount is 4.

    I can't tell where those files are located, the windows\temp folder looks stable but the drive is still filled when NORMAN is enabled.

    I m unable to locate those files.

    ( it's not the engine folders cause they are located on another drive )

    I just tried again to reenable it and after less than 10 minutes the drive was full.


    • Edited by tomazzzi Friday, May 17, 2013 12:40 PM
    Friday, May 17, 2013 12:39 PM
  • Is there a way to change the default folder these nse_tmp and def files are created into from windows\temp to elsewhere. Alot of us have had Forefront installed safely off the OS partition and the change in this engines behavior, even after your fix, is killing what used to be a stable OS partition which only has a limited safety margin mainly due to the exchange servers age.

    As it stands I'm leaving norman off my companies customers platforms as its simply not safe to leave an application doing what this one does.

    Friday, May 17, 2013 12:53 PM
  • tomazzzi,

    What you're describing is not the same behavior we've seen due to the underlying issue with the Norman engine even before the fix was released.  The issue on this forum is regarding the orphaned nsebin.def files that would not get deleted after Norman updates.  This would cause the disk space to increase over time, not within minutes.  This does not appear to be related to the issue that this forum post is covering.  

    I'd recommend locating what files are actually consuming the disk space and opening a support ticket if needed.

    -Ryan

    Friday, May 17, 2013 2:58 PM
  • You cannot change the path of the nse_tmp files to a specific location, no.  Obviously, you can change the path of the Windows Temp directory if you'd like to but most people would probably not want to do that.

    With the fix, as I've mentioned, the Norman engine will consume space on your Windows\Temp folder by design.  You'll have the one nsebin.def file as well as the nsetemp files.  The number of these files will correlate to the number of scanning processes that you have. 

    If you're running older systems that do not have enough space on the system drive then yes, you may in fact need to disable the Norman engine. 

    Friday, May 17, 2013 3:07 PM
  • The only *nse* file in Windows\Temp is 'nsebin.def.b65fc723392b991b.tmp' from 5/16 @ 09:29.

    Ryan,
    So of the two servers I am experiencing this on, one is test and the other is production. Test doesn't see much use. Right now test has one nsebin.def file. Production has one nsebin.def file and many nse_tmp files. On both servers the files are in Windows\Temp and all time stamped for today. You said something about looking at this particular setup. Have you had any luck? Still no real issue here but want to make sure we know what's happening.

    Thanks

    Tuesday, May 21, 2013 4:13 PM
  • Arkiados06,

    The only scenario where I see what you're describing would be if NetworkService had rights to the Windows\Temp directory.  You're checking the 'Security' tab in the properties of the Windows\Temp folder, correct?  If Norman determines that NetworkService does not have the required permissions then it will create the Windows\Temp\nsetmp directory and write all files needed there.  If you're not seeing NetworkService having access then I'm not sure why it's writing to the Temp directory directly.  You might want to get a process explorer capture for all file access to the Windows\Temp directory.

    The other behavior you describe sounds perfectly normal.  You're only seeing one nsebin.def file per server, for the current day,  which confirms that the real issue we had previously is resolved.  The fact that you're not seeing the nse_tmp files on the test server is fine too.  That probably means that nothing has been scanned yet or the Norman engine has not been loaded.

    Thanks,

    -Ryan

    Tuesday, May 21, 2013 9:36 PM
  • We do kind of have the same issue, the nsetemp folder contains 137 files with a total size of 5.4 GB. I've monitered the folder for some days now and the filesize and number of files stays pretty much the same.

    The majority of the files are now a few days old but the (single) nsebin.def-file continues to be replaced when new ones are released.

    Here is a list of the files in \temp\nsetmp folder on our Exchange edge server.
    http://shorttext.com/ZPjRsp

    I will continuously monitor the folder and if the number and total size of the files dramatically increases I post a note here.

    Wednesday, May 22, 2013 9:15 AM
  • Thanks for the information Jesper.

    Your nsebin.def file looks good.

    5.4 GB seems pretty large and we anticipated that all of the nse_tmp files would get cleaned up on shutdown and then on startup new nse_tmp files would be created.  It's possible that on your server the scanning process(es) have crashed on a couple of occasions and Norman is not cleaning up the nse_tmp files that were in use when the crash occurred.  It then seemingly does not clean these files up on the next startup.  This seems like it would be a pretty uncommon occurrence.  We'll investigate this scenario with Norman.  In the meantime, you should be able to delete the nse_tmp files that are not marked with today's date from the list you provided above.  These are the files that are consuming the majority of space in the folder.  If the nse_tmp files are in use you will not be able to delete them. 

    Thanks,

    -Ryan

    Wednesday, May 22, 2013 5:35 PM
  • The piling of files seems to coincide with server restarts.

    Norman files left in the nsebin folder:
    2013-05-15 20:51 - Server restart 2013-05-16 03:26 (Automatic restart, updates)
    2013-05-21 06:51 - Server restart 2013-05-21 17:40 (Manual restart, more updates)
    2013-05-21 17:54 - Server restart 2013-05-21 18:48 (Manual restart, added one CPU-core and memory.)

    2013-05-22 18:26 - The files in use now

    I will try to do a manual server restart tonight to see if the pattern continues.

    Thursday, May 23, 2013 6:13 AM
  • Hi Ryan,

    it's me again. I have tried enabling the Norman module on the Antigen 9.0 SP2, but it started again generating the tmp files.

    The version of the nse32.dll is 7.1.8, but the Engine version in the Console is 7.1.0.

    Do you have any suggestions ?

    Thanks in advance,


    Georgi Petkov MSFT

    Thursday, May 23, 2013 7:13 PM
  • Georgi,

    You're going to see tmp files.  This is expected.  The fix is that it will delete the older nsebin.tmp files that were consuming space over time.

    It's ok that you see 7.1.0 as the engine version in the console.  If you've confirmed the version of the .dll is 7.1.8 and it's updating you're good to go.  See #3 above from my post on 5/15.

    -Ryan

    Thursday, May 23, 2013 7:31 PM
  • Georgi,

    You're going to see tmp files.  This is expected.  The fix is that it will delete the older nsebin.tmp files that were consuming space over time.

    It's ok that you see 7.1.0 as the engine version in the console.  If you've confirmed the version of the .dll is 7.1.8 and it's updating you're good to go.  See #3 above from my post on 5/15.

    -Ryan

    We've held off apply this "fix" in production as not confident that it resolves the issue effectively.  With many servers running forefront the installation is safely off the system partition on a drive scaled to grow flexibly.  The system volumes are fixed and limited in space.  To my knowledge the previous norman engine did not create these temporary files, or at least not the same quantity and size otherwise our monitoring would have picked this up.  This fix whilst resolving the issue with the nsebin.def file does not really address the issue of the number and size of the temporary files.  Moving them into a sub folder doesn't really achieve a lot for us.  On one of our pre-production systems that is updating correctly you are still left with over 2.5Gb temporary files. 

    If these temporary files are necessary why are they not created within the program directory structure?  Also is there anyway of calculating how many of these temporary files should be created under normal circumstances, and what the maximum size is likely to be?  e.g. what are the factors that control how many and size of the temporary files?

    Friday, May 31, 2013 10:45 AM
  • The fix referred to in this thread was specifically geared to address the Norman engine not deleting the old nsebin.def files from the temp directory after each Norman engine signature update.  This were the files that were taking up the majority of space.

    You're correct this is new behavior for the engine and an upcoming release of the Norman engine will further improve the behavior with the nse_tmp files. 

    The nse_tmp{GUID}.tmp files should correlate to the number of scanning processes that you have running and these files will vary in size.  We've seen situations where these nse_tmp files are not properly cleaned up and therefore can begin to take up some disk space after some time.  It is safe to remove those older nse_tmp files so if you need to free up space you can do so by deleting the stale nse_tmp files.

    We are working with Norman to improve the implementation so that the final release of their new engine resolves this issue so that both the nsebin.def files AND the nse_tmp files are properly cleaned up so that excessive disk space is not consumed after some running time.  Once that is accomplished it should be much easier to predict and manage disk space consumption by the Norman engine.

    If you cannot currently manage disk space issues due to the Norman engine I would recommend keeping the engine disabled until the final release of the new engine. 

     

    Friday, June 07, 2013 4:28 PM
  • Hi Sorry I might be waking up ghosts here. I'm a college of Georgi and together we work on an old Exchange 2003 environment running Antigen.

    Ever since the problems with creating the tmp file issue had started we disabled Norman Antivirus. The question I have is the following...  has there been any change in the nse_tmp file creation behavior over the past few months or in other words, were you successful in working with Norman on resolving the issue.

    Thank you.

    Best Regards

    Yavor Tsanev  

    Wednesday, August 14, 2013 10:21 AM
  • The new Norman engine is currently undergoing testing.  Once this is completed the engine will be released publicly.

    I will update this thread once that is done.  I don't think it should be much longer.

    Thanks,

    -Ryan

    Thursday, August 22, 2013 11:21 AM
  • Ryan,

    Have you any update for us on the final release of the engine?

    We hadn't realised Norman was back to haunt us until we patched some HT servers last week. After a couple of reboots one ran out of disk space because of orphaned tmp definitions.

    Thanks

    Thursday, September 05, 2013 12:35 PM