none
URL Filtering & Categorisation : Is it actually any good!?

    Question

  • Hello.

    I am currently testing the TMG 2010 as a candidate to replace our current Internet Access Proxy. 

    Although it works great and performs well for Malware scanning etc. I am perplexed by how poorly many websites have been categorised.   Does anybody else notice this? 

    I have had to feed in category overrides for the most obvious of sites;

    www.facebook.com is by default in the Blogs/Wiki category.   WRONG : Should be in Online Community.

     - - I want to allow Blogs and Wiki as sites like this very one are useful and relatively harmless. 

    bbcimg.co.uk is in the Media Sharing category.  WRONG : Should be in the NEWS category. 

     - - BBC News and Sport pages are not Media Sharing, that is surely reserved for SkyDrive etc?

    *.social.s-msft.com is in Web Email category.  WRONG : Should be in Technical information. 

     - - This very website displays incorrectly when the above is left at default.  We obviously want to block access to Hotmail services and so allowing Web Email is a no-no.

     

    The examples above are the most obvious I have come across after one day of testing between three users.  Basically we get fragmented pages, images missing or layout is completely screwed unless I override the categories as above. 

    So - how accurate is the MSoft Reputation Services and who actually categorises this stuff?!? 

     


    Ash Cox
    • Edited by AshCox Thursday, February 02, 2012 5:42 PM
    Thursday, February 02, 2012 5:42 PM

Answers

  • I read your post and got worried as we are switching over to TMG from Websense.  But then I found out that you can enable non-primary URL filtering categorizations.  Which would probably fix your problem.  You see many sites have multiple classifications, so that site that hosts mail might come up as Education and Web Mail.  Since Education is higher on TMG's  precedence list (http://blogs.technet.com/b/isablog/archive/2010/08/03/tmg-url-filtering-category-precedence.aspx) it will filter based on Education not Web Mail.  By-passing your Deny rule.  However, if you turn on non-primary URL filtering categorizations (http://technet.microsoft.com/en-us/library/ff966399.aspx) then if any of the categories a site is label as is part of a Deny rule it will match. In this case it would be Web Mail.

    I've been running test between Websense and TMG and comparing how it categorizes sites.  They are very similar and some cases TMG does a better job.  So I'm convinced with non-primary URL filtering categorizations turned on you should be OK.


    • Edited by Anthony Meluso Friday, March 23, 2012 5:21 AM Typo
    • Marked as answer by AshCox Wednesday, July 18, 2012 1:31 PM
    Friday, March 23, 2012 5:20 AM

All replies

  • Hi,

     

    Thank you for the post.

     

    “how accurate is the MSoft Reputation Services and who actually categorises this stuff?!? ” – you can read this blog: http://blogs.technet.com/b/isablog/archive/2009/06/10/url-filtering-is-here.aspx. If you want to update the URL database with your own overrides.  Please look at this: http://blogs.technet.com/b/isablog/archive/2010/01/07/scripting-url-overrides-in-forefront-tmg.aspx.

     

    Regards,


    Nick Gu - MSFT
    Friday, February 03, 2012 6:42 AM
  • Hi,

    Please check this link, on Common Q&A about TMG URL Filtering database,

    http://blogs.technet.com/b/isablog/archive/2010/11/15/common-q-amp-a-about-tmg-url-filtering-database.aspx

    URL filtering troubleshooting flow,

    http://technet.microsoft.com/en-us/library/ff358603.aspx

    I hope this you help you.

     

    Thanks,


    Best Regards, ----Naresh Man Maharjan,Nepal---- www.msserverpro.com
    Friday, February 03, 2012 7:38 AM
  • Ok...  Do this:

    Filter to DENY all Web Email.

    Now search www.google.com for Web Email.  Go down the list, trying all the links.   You will see that MANY of the URLs are incorrectly categorised as "Education" or "Technical Information" etc - not Web Email.  Even when the URL contains http://mail.xxx.xxx.com or http://webmail.xxx.com or http://www.xxx.com/webmail.

    Ideally I'd like to add a category override for a URL string, such as mail.* or webmail.*, but this is not allowed as you have to re-categorise full domain names only. Not great.

    So, back to my question:  Is the URL categorisation actually trustworthy?  Do many people out there actually pay to use and solely rely on the Reputation Services subscription?  Or is TMG 2010 only used for its VPN, Gateway and Reverse Proxy by most companies?


    Ash Cox


    • Edited by AshCox Wednesday, February 15, 2012 5:52 PM
    Wednesday, February 15, 2012 5:51 PM
  • Hi,

    Thanks for your questions.

    Q. 1. Is the URL categorisation actually trustworthy?

    Ans.  The URL categorisation actully trustworthy because we have no any more option it's a BIG BAD INTERNET. New websites spring into operation every minute ( or second). Traditional protection based on key words aren’t effective. Therefore, it helps to save you time figuring out what to block and what not to block. But problem is that URL Categorisation is not 100% perfect. You will face same problems in third party UTM also. Therefore, Microsoft give us URL Category Overrides Tab for this problems. Therefore, I finally suggest to use URL categorisation .Cyber attacks seem to come in waves, and when in rains, it pours. So manully adding web sites is not possible to protect our systems.

    Q. 2. TMG 2010 only used for its VPN Gateway and Reverse Proxy by most companies?

    Ans. In many deployment scenarios, Forefront TMG 2010 is used solely for forward and reverse proxy functionality.Due to some limitations of networking features in previous versions of Microsoft Firewall (ISA Server 2004/2006), large organizations used hardware firewall at the edge of network(This has been the trend over last ten years). For this please check this site, http://www.msserverpro.com and check article on Deploying Forefront TMG 2010 Server as a Reverse Proxy in an Existing Firewall DMZ.

    I hope this you help you.

    Thanks,

     

     


    Best Regards, ----Naresh Man Maharjan,Nepal---- www.msserverpro.com

    Wednesday, February 15, 2012 6:47 PM
  • The main benefit I've seen is the malware inspection. You will be suprised at just how many things get blocked...little javascripts and advertisements are blocked every second that contain potential threats.

    The URL catergorisation: I've only blocked the ugly groups such as botnet, spyware/adware and phishing. When blocking more mainstream catergories, yuo get a lot of false positives, or a multitude of exceptions are reguired for every man and his dog. I've found this though with any reputation based service, not just TMG.

    Wednesday, February 15, 2012 11:19 PM
  • I read your post and got worried as we are switching over to TMG from Websense.  But then I found out that you can enable non-primary URL filtering categorizations.  Which would probably fix your problem.  You see many sites have multiple classifications, so that site that hosts mail might come up as Education and Web Mail.  Since Education is higher on TMG's  precedence list (http://blogs.technet.com/b/isablog/archive/2010/08/03/tmg-url-filtering-category-precedence.aspx) it will filter based on Education not Web Mail.  By-passing your Deny rule.  However, if you turn on non-primary URL filtering categorizations (http://technet.microsoft.com/en-us/library/ff966399.aspx) then if any of the categories a site is label as is part of a Deny rule it will match. In this case it would be Web Mail.

    I've been running test between Websense and TMG and comparing how it categorizes sites.  They are very similar and some cases TMG does a better job.  So I'm convinced with non-primary URL filtering categorizations turned on you should be OK.


    • Edited by Anthony Meluso Friday, March 23, 2012 5:21 AM Typo
    • Marked as answer by AshCox Wednesday, July 18, 2012 1:31 PM
    Friday, March 23, 2012 5:20 AM
  • I second that. Enabling non-primary URL categories is essential for a filter that works as expected.

    If it still catches stuff you want to allow or visa versa, use the category override.

    Saturday, March 24, 2012 11:06 PM
  • Have revisted this with the suggested option for tighter URL filtering - a feature that was introduced in an update to the TMG.  I can confirm the same - the filtering is much better and reliable - if anything it's the other way, too restrictive but at least we only have to work to open up nominated sites rather than trying to lock down any number of them.

    Thanks for all responses they have helped.


    Ash Cox

    Tuesday, March 27, 2012 8:24 AM