Thursday, September 23, 2010 6:18 PM
We've been trying to upgrade from ISA 2006 (running on Server 2003 Std) to TMG 2010 SP1 (running on 2008 R2 Std) without much success. All publishing, routing, etc works fine except for client VPN access. Specifically L2TP/IPSec clients. PPTP works fine everytime. When attempting to connect using L2TP/IPSec, we receive Error: 789 "The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer". The kicker is, however, it's not consistent. Sometimes the client connects, sometimes it does not. Sometimes we can connect, disconnect, and reconnect twice within several minutes without issue. Other times, we can't connect even once. After some random amount of time we may be able to connect again. If we connect to TMG using PPTP it will work fine every time. If we put our original ISA box back in service, the L2TP/IPSec VPNs will work again every time.
We've tried to address this by rebuilding the box from scratch, configuring the rules and objects from scratch (assuming there may be a problem with corrupt objects in the config file exported from ISA), configuring the NAT-T reg hack (KB926179). Still no luck. The config is the same between boxes. Is there something different with how Server 2008R2 and/or TMG handles IPSec? Is this even a TMG problem or a problem w/RRAS on 2008?
Any help anyone can offer is greatly appreciated!
Thursday, October 07, 2010 4:33 PMAnswerer
Have you installed the TMGBPA from http://ISABPA.com yet? Once installed, if you use the TMG Data Packager to troubleshoot when using a repro mode. What types of clients are getting the error? What is being logged into the System and Security Event logs that is related to the error? Doing a search on other issues relates to the error you are getting, I found one that was resolved by modifying the registry. Please note that modifying the registry can lead to disastrous results if done incorrectly.
1. Open the registry and navigate to HKLM\System\CurrentControlSet\Services\Rasman\Parameters
2. Create a new REG_DWORD if this is not there - AllowL2TPWeakCrypto
3. Set the value to 1.
4. Reboot the system and test.
- Proposed As Answer by Brennan CroweEditor Thursday, October 07, 2010 4:35 PM
- Marked As Answer by Nick Gu - MSFTMicrosoft Contingent Staff, Moderator Friday, October 08, 2010 2:24 AM