We are setting up a Win2k8 Enterprise x64 three node fileserver cluster. Each of the machines is a virtual server hosted on Citrix Xen. After creating the cluster (with default firewall settings) everything works fine.
Ideally, we would leave the firewalls off and use IPSec (without encryption) to lock down the machines to only our subnets, but if we enable any IPSec rules the cluster fails immediately.
Since IPSec isn't working, we tried locking down windows firewall to only allow cluster traffic on our subnets, but as soon as I edit the "Failover Clusters - Cluster Service (RPC)" or "Failover Clusters - Cluster Service (TCP-In)" rules the cluster fails. It only seems to work if there are no specific IP or Subnet definitions in the allow list for these two rules. Most of the other rules will allow subnet definitions without affecting the cluster.
Has anyone else had this issue? Am I overlooking something?
Thanks for your help.