More ISA Site-to-Site IPSec VPN Configuration

The last two months, traffic on this site has increased by almost 50%.  Honestly, I don’t know specifically what it’s related to, but my number one search item is “ISA” followed by #4 which is “2006”.  Therefore, I thought I’d post a little bit more, since there are still some issues that I had run into.

First thing is the issue I discussed last time, which was about IPsec not creating the filters it needed to.  I think I may have found the solution to that, but I haven’t verified it (which I may do this week while I’m on the bench, in between various training).  Now because of the fact that ISA relies on Windows 2003 IPsec, there are some pretty awful problems.  The first being “adjacent ranges” in your IPsec rules.  Windows IPsec does not allow you to have two adjacent IPSec policies.  Instead it believes it should be one continuous policy.  When you attempt to create an adjacent policy in ISA 2006 you will recieve the error message below.

As an example, let’s say that you need access to the individual hosts 10.10.10.150 and 10.10.10.151 at the remote site (or two ranges like 10.10.10.0/24 and 10.10.11.0/24).  Now, if the remote site happened to have a Cisco Concentrator, they would be able to publish each of those hosts (or subnets) as separate IPsec policies.  However, with ISA, they have to be in the same remote networks range.

Many times, there isn’t a problem, especially with a small number of hosts or ranges like this.  However, the problem comes into play with subnetting.  Typically hosts are designated with a 32 bit mask (255.255.255.255).  However, since we’ve now created a range, we may see a different mask (255.255.255.254).  It’s when the different, unexpected mask comes into play, that we have issues.  If the mask is wrong, Phase II negotiations fail, and you’ll not be able to create a Phase II tunnel.  However, if you don’t put the the hosts into the range and ignore the warning that ISA gives, the IPsec policies won’t be created, and you’ll have to manually create them whenever the IPsec service restarts (specifically if/when the machine restarts).

Finally, there’s yet another IPsec issue with Windows 2003, that again manifests itself with ISA.  There are multiple ways you may see this.  One way is that no matter what you set as your Phase II timeout policy from within ISA, you’re seeing Phase II rekeying happen about every 300 seconds.  Another way is that you IPsec Site-to-Site VPN connections drop a lot and in the logs you see the error “0xc0040014 FWX_E_FWE_SPOOFING_PACKET_DROPPED”.

The first thing I tried was to disable IP Spoof Detection.  However, that didn’t seem to fix it, plus, since this is an external firewall, I wanted to keep the spoof features on to do application filtering.  The part that was really frustrating was that rekeying was happening every 300 seconds, instead of the 3600 I had specified in ISA.

Well, it turns out it’s a bug with ISA and/or Windows 2003 IPsec.  This was actually a bug with ISA 2004, but apparently it wasn’t deemed big enough to fix with 2006, since there’s a workaround that works well.  Microsoft KB article 917025 goes over exactly what to do, but the gist of it is, is that you need to edit the SAIdleTime registry key and change it to 3600 (default is 300).  The downside is that 3600 is the max (trust me, I’ve tried to set it higher, it doesn’t work at all), so plan your IPsec Site-to-Site VPNs accordingly (let your peer know that the max will be 3600 seconds).

Hopefully those two nuggets will help anyone having other issues.  I’m sure I’ll post more things too, as they come up.