locked
UAG DirectAccess: impossible to start NLB RRS feed

  • Question

  • Hi,

    On a deployment from scratch of two Forefront UAG servers for DirectAccess there is no possibility to start the NLB through the Web Monitor. In fact on the Synchronization Status there is an Error notification and on detailed event the error message is:

    ·          ID: 133

    ·          Type: Cannot reach remote machine

    ·          Category: System

    On both servers the following patch are deployed:

    ·     Forefront UAG Update 1

    ·     Forefront TMG SP1

    ·     KB977342

    Network configuration of the Array master:

    ·     LAN: 10.xx.xx.12 /24

    ·     WAN: 82.xx.xx.32 /24

    Network configuration of the Array member:

    ·     LAN: 10.xx.xx.13 /24

    ·     WAN: 82.xx.xx.33 /24

    NLB configuration done through UAG:

    ·     Unicast mode

    ·     Internal: 10.xx.xx.11 /24

    ·     External: 82.xx.xx.30 /24

    ·     External: 82.xx.xx.31 /24

    Any help could be appreciated.


    Follow me on Twitter http://www.twitter.com/liontux
    My Blog (French) : http://security.sakuranohana.fr/search/label/FR
    My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    Wednesday, September 8, 2010 9:40 AM

Answers

  • See if you can find updated drivers. If that doesn't work, you might try putting in some non-broadcom NICs and see what happens.

    HTH,

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    • Marked as answer by Erez Benari Thursday, October 7, 2010 12:21 AM
    Friday, September 24, 2010 2:50 PM

All replies

  • I don't see the internal VIPs.

    Check out the Test Lab Guide: Demonstrate UAG DirectAccess Network Load Balancing and Array Configuration at http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=7fb64cad-5dac-471a-9fbf-a6c9d03ffbad for details.

    HTH,

    Tom

     

     


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Wednesday, September 8, 2010 2:27 PM
  • As mentionned on the  Test Lab Guide: Demonstrate UAG DirectAccess Network Load Balancing and Array Configuration the internal VIP configured on the NLB is 10.xx.xx.11 /24.


    Follow me on Twitter http://www.twitter.com/liontux
    My Blog (French) : http://security.sakuranohana.fr/search/label/FR
    My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    Wednesday, September 8, 2010 2:32 PM
  • How are the NICs configured?

    Are these physical or virtual?

    If physical, how are they network connected?


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    Wednesday, September 8, 2010 3:41 PM
  • There is two physical NIC:

    • One connected to the LAN
    • On connected to the WAN

    Follow me on Twitter http://www.twitter.com/liontux
    My Blog (French) : http://security.sakuranohana.fr/search/label/FR
    My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    Wednesday, September 8, 2010 4:49 PM
  • Hi Lionel,

    I was hoping for a little more detail than that!!!!

    I also meant, are the SERVERS physical or virtual, not the NICs ;)

    NLB is often senesitve to the networking environment so we need to understand your deployment to help. Also, NLB and virtualisation like VMware and Hyper-V need to be configured to integrate correctly...

    Cheers

    JJ


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    Wednesday, September 8, 2010 10:14 PM
  • Hi Lionel,

     

    Jason is right. Do you enable the MAC sproofing feature for the network card with NLB configured?

     

    Have a nice day.


    BenoitS - Simple by Design http://danstoncloud.com/blogs/simplebydesign/default.aspx
    Thursday, September 9, 2010 5:20 AM
  • Sorry I misunderstanding the question. Both servers are physical without any teaming on network card.
    Follow me on Twitter http://www.twitter.com/liontux My Blog (French) : http://security.sakuranohana.fr/search/label/FR My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    Thursday, September 9, 2010 8:03 AM
  • Hey Lionel,

    My car won't start; it is blue in colour and is made by Audi - any idea how to fix it ;)

    Cheers

    JJ

    [Hint: Fixing problems with no detail when you can't see the important bits is kinda hard to do :)] 


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    Thursday, September 9, 2010 10:07 PM
  • Sorry for the lack of information I'll try to do my best in this reply don't hesitate to ask for the missing ones.

    Array master server:

    • Physical server
    • Forefront UAG with update 1, Forefront TMG SP1 and KB977342 installed
    • Two Broadcom BMC5709C NIC : on dedicated for LAN and another one for the WAN with following configuration as mentioned on the first post
    • both NIC are connected to a CISCO catalyst switch
    • Private certificate for ipsec is installed
    • Public certificate for IP-HTTPS is installed
    Array member server:
    • Physical server
    • Forefront UAG with update 1, Forefront TMG SP1 and KB977342 installed
    • Two Broadcom BMC5709C NIC : on dedicated for LAN and another one for the WAN with following configuration as mentioned on the first post
    • both NIC are connected to a CISCO catalyst switch
    • Private certificate for ipsec is installed
    • Public certificate for IP-HTTPS is installed

    The servers are located on a DMZ and all the traffic are authorized from and to the internal network.

    After configuring trough the UAG console the NLB feature on unicast mode with one private VIP on the same subnet, and two public VIP on the same subnet on the array master the configuration is properly activated on both servers.

    When I access to the UAG Web monitor to start NLB on all node here the status displayed:

    • NLB Status: unknown
    • Synchronization status: Error
    • No possibility then to start NLB

    UAG Activation monitor display that NLB configuration successfully done, on Forefront TMG Alerts Tab NLB is "started".

    From UAG Best practice analyzer no error is raise up for NLB configuration.

    When analysing network trace between all the servers nothing seems to be blocked.


    Follow me on Twitter http://www.twitter.com/liontux
    My Blog (French) : http://security.sakuranohana.fr/search/label/FR
    My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    Friday, September 10, 2010 7:07 PM
  • Yes, much better ;)

    The services tab of TMG shows NLB as "running" on both servers?

    As both NICs are connected to a single switch, I assume you are using VLANs?

    Can you ping the NLB VIPs?

    Have you defined the VIPs on the same subnet as the dedciated IP addresses?

    What do you see in NLB manager on each host? (be aware NLB manager will only be able to see the localhost properties not both)

    Cheers

    JJ


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    Friday, September 10, 2010 11:08 PM
  • The services tab of TMG shows NLB as "running" on both servers?

    Yes.

    As both NICs are connected to a single switch, I assume you are using VLANs?

    Each server is connected to is own switch (one different for each server):

    • The NICs connected to the LAN use the same VLAN "A" with trunk
    • The NICs connected to the WAN use the same VLAN "B" with trunk

    Can you ping the NLB VIPs?

    Ping work for the VIP associated to the LAN it's not comming on my mind to test for the WAN VIP.

    Have you defined the VIPs on the same subnet as the dedciated IP addresses?

    Yes as mentionned on the first post.

    What do you see in NLB manager on each host? (be aware NLB manager will only be able to see the localhost properties not both)

    On each host in NLB manager I see the following errors for the remote host (no errors for localhost):

    • Host unreachable
    • RPC unavailable

    Follow me on Twitter http://www.twitter.com/liontux
    My Blog (French) : http://security.sakuranohana.fr/search/label/FR
    My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    Saturday, September 11, 2010 6:21 PM
  • Hi,

    I'm running a similar environment and got the same behavior:

    When I access to the UAG Web monitor to start NLB on all node here the status displayed:

    • NLB Status: unknown
    • Synchronization status: Error
    • No possibility then to start NLB

    UAG Activation monitor display that NLB configuration successfully done, on Forefront TMG Alerts Tab NLB is "started".

    It's possible to start NLB from TMG Service Management on both nodes, but NLB stops automatically after a few seconds. There are no NLB related error messages in event logs.

     

    Monday, September 13, 2010 8:46 AM
  • Here's some additionnal information from the wlbs display command line from the Array Master:

    WLBS Cluster Control Utility V2.6 (c) 1997-2007 Microsoft Corporation.

    Cluster 10.xx.xx.11

     

    === Configuration: ===

     

    Current time                = 13/09/2010 11:50:54
    ParametersVersion           = 6
    CurrentVersion              = V2.6
    EffectiveVersion            = 00000201
    InstallDate                 = 0x4C88D428
    HostPriority                = 2
    ClusterName                 = Internal

    ClusterIPAddress            = 10.xx.xx.11
    ClusterNetworkMask          = 255.255.255.0
    DedicatedIPAddresses/       = 10.xx.xx.12/255.255.255.0
    DedicatedNetworkMasks      
    McastIPAddress              = 0.0.0.0
    ClusterNetworkAddress       = xx-yy-zz-aa-bb-cc
    IPToMACEnable               = ENABLED
    MulticastSupportEnable      = DISABLED
    IGMPSupport                 = DISABLED
    MulticastARPEnable          = ENABLED
    MaskSourceMAC               = ENABLED
    AliveMsgPeriod              = 1000
    AliveMsgTolerance           = 5
    MaxConnectionDescriptors    = 262144
    FilterICMP                  = DISABLED
    ClusterModeOnStart          = STOPPED
    PersistedStates             = SUSPENDED
    NBTSupportEnable            = ENABLED
    UnicastInterHostCommSupport = ENABLED
    BDATeaming                  = YES
    TeamID                      = {5601BF8D-2D28-46D2-B4DC-0983B2B6532E}
    Master                      = YES
    ReverseHash                 = NO
    IdentityHeartbeatPeriod     = 10000

    NumberOfRules (1):

          VIP       Start  End  Prot   Mode   Pri Load Affinity
    --------------- ----- ----- ---- -------- --- ---- --------
    ALL                 0 65535 Both Multiple      Eql Single

     


    === Event messages: ===

     

    Could not open event log due to:

    The operation completed successfully.

    In bold the lines when something seems to be weird.

     

    And from a trace.hta on the Array master server I got the following line:

    [10]14A4.29BC 09/13/2010-10:24:47.774 [MonitorHelper PingAggregator.get_Data xxxxxxxxxxxxx_xxxxx] TF_NOISE: GetSyncAndNLBStatusAggregator Data [Array_Master_Hostname,10.xx.xx.12,Error;Array_Member_Hostname,10.xx.xx.13,Error]


    Follow me on Twitter http://www.twitter.com/liontux
    My Blog (French) : http://security.sakuranohana.fr/search/label/FR
    My Blog (English) : http://security.sakuranohana.fr/search/label/EN
    • Edited by Lionel LEPERLIER Monday, September 13, 2010 1:38 PM Add trace.hta log extract
    Monday, September 13, 2010 12:02 PM
  • Hi Lionel,

    I think the NLB stopped setting is normal, so that NLB can be controlled by TMG/UAG as opposed to starting automatically.

    NLB manager will also show cosmetic errors as the MMC communications between nodes are blocked by TMG by default.

    So, just to recap...

    NLB doesn't show as converged in Web Monitor, but looks ok in TMG services? Can you paste the result you get from wlbs query?

    Web Monitor shows a sync error, yet activation completes successfully? Can you past the result you get from the Monitoring=>Configuration tab in TMG?

    Cheers

    JJ


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    • Marked as answer by Erez Benari Wednesday, September 15, 2010 12:12 AM
    • Unmarked as answer by Lionel LEPERLIER Wednesday, September 15, 2010 8:03 AM
    Monday, September 13, 2010 10:18 PM
  • From the array manager the result of wlbs query:

    WLBS Cluster Control Utility V2.6 (c) 1997-2007 Microsoft Corporation.

    Cluster 10.xx.xx.11

    Host 2 is stopped and does not know convergence state of the cluster.


    Cluster 82.xx.xx.30

    Host 2 is a slave to cluster 10.xx.xx.11.

    And from the array manager the result of Configuration Tab on TMG:

    - Array_Manager: Synced | Server configuration matches the stored configuration. 

    - Array_Member: Synced | Server configuration matches the stored configuration.


    Follow me on Twitter http://www.twitter.com/liontux | My Blog (French/English) : http://security.sakuranohana.fr/
    Wednesday, September 15, 2010 8:08 AM
  • So, that tells me that TMG configuration is happy across the standalone array, but NLB appears to be broken.

    If NLB is not converging, you must be getting event log errors to this effect?

    Can you run the wlbs query on the array member - do you get the same thing?

    Cheers

    JJ

    P.S. I wish I could see the setup :(


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    Wednesday, September 15, 2010 6:14 PM
  • As asked here the array member result of wlbs query:

    WLBS Cluster Control Utility V2.6 (c) 1997-2007 Microsoft Corporation.

    Cluster 10.xx.xx.11

    Host 3 is stopped and does not know convergence state of the cluster.


    Cluster 82.xx.xx.30

    Host 3 is a slave to cluster 10.xx.xx.11.


    Follow me on Twitter http://www.twitter.com/liontux | My Blog (French/English) : http://security.sakuranohana.fr/
    Thursday, September 16, 2010 8:27 AM
  • So, something is obviously preventing convergence, I would start by troubleshooting NLB in it's native form and then work up to TMG/UAG. This may help: http://technet.microsoft.com/en-us/library/cc781160(WS.10).aspx or this: http://support.microsoft.com/kb/812870

    Without going through every possible reason for NLB not converging, I am starting to get a little stumped :(

    Cheers

    JJ


    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk
    Thursday, September 16, 2010 12:24 PM
  • Actually a ticket support is open for this issue, from Microsoft  here the first way to resolve it:

    1. Restart the Array master
    2. On the Event Viewer search for this entry

    Log Name:      System

    Source:        Service Control Manager

    Date:          15/09/2010 10:52:14

    Event ID:      7022

    Task Category: None

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      hostaname.domain.ext

    Description:

    The Microsoft Forefront TMG Managed Control service hung on starting.

    Event Xml:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

      <System>

        <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />

        <EventID Qualifiers="49152">7022</EventID>

        <Version>0</Version>

        <Level>2</Level>

        <Task>0</Task>

        <Opcode>0</Opcode>

        <Keywords>0x8080000000000000</Keywords>

        <TimeCreated SystemTime="2010-09-15T08:52:14.400011100Z" />

        <EventRecordID>9178</EventRecordID>

        <Correlation />

        <Execution ProcessID="564" ThreadID="568" />

        <Channel>System</Channel>

        <Computer>hostaname.domain.ext</Computer>

        <Security />

      </System>

      <EventData>

        <Data Name="param1">Microsoft Forefront TMG Managed Control</Data>

      </EventData>

    </Event>

    If it's the case a request for support is required (known problem for Microsoft).

    Unfortunately the solution doesn't work for me, as soon as the problem is solved I'll post how to solve this kind of issue.


    Follow me on Twitter http://www.twitter.com/liontux | My Blog (French/English) : http://security.sakuranohana.fr/
    Thursday, September 16, 2010 1:20 PM
  • Hi Lionel,

    Did you happen to install the NLB hotfix for ISATAP?

    There is a known issue when it's installed after NLB is already configured with 2 clusters. NLB simply won't start

    Thursday, September 16, 2010 2:32 PM
  • If you talk about the KB977342 yes it's installed on both servers before the installation of Forefront UAG and configuring NLB.
    Follow me on Twitter http://www.twitter.com/liontux | My Blog (French/English) : http://security.sakuranohana.fr/
    Thursday, September 16, 2010 2:36 PM
  • I know the some Cisco switches may have an issue with unicast NLB. Are you aware of this issue?

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Wednesday, September 22, 2010 2:00 PM
  • The reference of the cisco switche is Catalyst 2960. I only found this information :

    • For NLB Multicast some configuration is needed but non appopriate in this case
    • For NLB Unicast the flood blocking feature must be disable (it's the case)

     


    Follow me on Twitter http://www.twitter.com/liontux | My Blog (French/English) : http://security.sakuranohana.fr/
    Wednesday, September 22, 2010 3:46 PM
  • Any chance that you are using Broadcom NICs?

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Thursday, September 23, 2010 1:57 PM
  • Yes you are right: two Broadcom NetXtreme II 5709c Gigabit Ethernet without teaming on a Dell PowerEdge 11G R610 for each server.
    Follow me on Twitter http://www.twitter.com/liontux | My Blog (French/English) : http://security.sakuranohana.fr/
    Thursday, September 23, 2010 2:03 PM
  • See if you can find updated drivers. If that doesn't work, you might try putting in some non-broadcom NICs and see what happens.

    HTH,

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    • Marked as answer by Erez Benari Thursday, October 7, 2010 12:21 AM
    Friday, September 24, 2010 2:50 PM
  • Bumping this old thread - I'm seeing the same thing at a customer site. I did update the NIC drivers although it didn't have any effect. I am not sure what trace flags affect this although I set everything for error/warning to enabled and got nothing interesting.
    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Tuesday, December 14, 2010 7:28 AM
  • Hi Brian,

    Is this a physical or virtual deployment?

    Check out some of our new troubleshooting information:

    http://blogs.technet.com/b/tomshinder/archive/2010/12/14/new-uag-directaccess-troubleshooting-content-on-the-technet-wiki.aspx

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Tuesday, December 14, 2010 6:24 PM
  • Hi Brian,

    Is this a physical or virtual deployment?

    Check out some of our new troubleshooting information:

    http://blogs.technet.com/b/tomshinder/archive/2010/12/14/new-uag-directaccess-troubleshooting-content-on-the-technet-wiki.aspx

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx

    It's a physical environment. I'm actually not using DA - just straight HTTP/HTTPS trunks but it's the same error and symptoms.
    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Tuesday, December 14, 2010 11:15 PM
  • Hi Brian,

    When you configured NLB, did you activate the configuration?

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Wednesday, December 15, 2010 2:40 PM
  • Hi Brian,

    When you configured NLB, did you activate the configuration?

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx

    Yes.
    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Thursday, December 16, 2010 12:02 AM
  • Well, this doesn't look like an easy one.

    Have you gone through the steps over at http://technet.microsoft.com/en-us/library/cc781160(WS.10).aspx

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Thursday, December 16, 2010 4:34 PM
  • Well, this doesn't look like an easy one.

    Have you gone through the steps over at http://technet.microsoft.com/en-us/library/cc781160(WS.10).aspx

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx

    I don't see anything obvious in that list that would apply. My current thought process is getting some tracing but I'm not sure what trace flags would apply.
    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Thursday, December 16, 2010 11:50 PM
  • Well, this doesn't look like an easy one.

    Have you gone through the steps over at http://technet.microsoft.com/en-us/library/cc781160(WS.10).aspx

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx

    I don't see anything obvious in that list that would apply. My current thought process is getting some tracing but I'm not sure what trace flags would apply.
    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Thursday, December 16, 2010 11:51 PM
  • For what it's worth, this is exactly the error I saw when the UAG Array manager account wasn't in the Administrators group on the second node.  Might be worth a check.  It's easy to forget if you built it as a single node and added the second later, and it didn't make itself obvious by interfering with any other part of the configuration process.
    Wednesday, February 9, 2011 7:05 PM
  • Hi Doug,

    Nice tip!

    Thanks!

    Tom


    MS ISDUA/UAG DA Anywhere Access Team Get yourself some Test Lab Guides! http://blogs.technet.com/b/tomshinder/archive/2010/07/30/test-lab-guides-lead-the-way-to-solution-mastery.aspx
    Thursday, February 10, 2011 12:32 PM