Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Cory Von Wallenstein]

Hi folks,

Stephen Spector suggested these questions may be best answered by the dev email list.

My team and I have been working diligently since early this year working on automation and deployment of Xen for a new VPS service at DynDNS. Along the way, we decided to have IPv6 as one of our features, and had to make a handful of changes to the Xen network scripts to successfully and safely do so in a VPS environment.

While our priorities have first and foremost getting our Spring Server VPS service out the door (which as of a few weeks ago, it is!), now I'd like to see if the community could benefit from this work.

a) Have people already solved and dealt with IPv6 in Xen successfully (i.e., is it a non-issue at this point)? If not, I'd be happy to submit the changes and a guide to making it work and work well.

Along the way, we also ran into some issues where domUs were able to:

1) "steal" IP addresses through IP aliasing (e.g., domU has 1.2.3.4, and domU root does "ifconfig eth0:0 1.2.3.5/32" in Linux, and now has two working IPs),
2) and more importantly, were able to impact the network connectivity of another domU by aliasing or assigning its in-use IP address,
3) and MOST importantly, were able to impact the network connectivity for all domUs on a subnet by aliasing a gateway IP address (e.g., in Linux "ifconfig 1.2.3.1" for a typical /24 subnet). 
4) Also, sending out invalid or poisoned ARP packets from one domU were able to introduce network connectivity problems for other domUs.

We were able to make a handful of changes to the Xen scripts to resolve these issues as well for safe and secure operation (especially for a VPS environment, where individual owners of domUs are likely unrelated to each other).

b) As above, have folks already addressed these issues for stealing IPs/ARP poisoning? Have they just not encountered them yet? Would it be useful to submit these modifications for review by the community?

We're more than happy to help, just don't want to duplicate work or step on anyone's toes for work they already have in progress.

Best regards,

Cory von Wallenstein
Spring Server Engineer
Dynamic Network Services
http://www.dyndns.com

Re: Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Stefan de Konink]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Cory Von Wallenstein schreef:
> We're more than happy to help, just don't want to duplicate work or step on anyone's toes for work they already have in progress.

I think people here are more than happy to receive the patches! Anyway
I'm interested ;)



Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREKAAYFAkjZTpcACgkQYH1+F2Rqwn29BQCfU5umxeMeaGOsc9dBPGVsO7OW
/JAAninukDBoU69JQP9TBK9liV42GIvl
=ghle
-----END PGP SIGNATURE-----

Re: Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Luke S Crawford]

Cory Von Wallenstein <cvonwallenstein@dyn-inc.com> writes:


> a) Have people already solved and dealt with IPv6 in Xen successfully (i.e., is it a non-issue at this point)? If not, I'd be happy to submit the changes and a guide to making it work and work well.

Guides would be awesome.  Also, if you made anti-spoof work for IPv6, that 
would also be awesome.

I've had IPv6 going for a while with no problems (though the xen antispoof 
rules only work with IPv4.)  - but I didn't  even need to enable IPv6 in the 
Dom0;  it's a layer 2 bridge, so it just works.  In fact, stateless 
autoconfiguration even worked.   I asked for an IPv6 allocation from
my provider and my customers noticed it was working before I got around to
doing any setup at all.

what problems did you hit with IPv6?

> Along the way, we also ran into some issues where domUs were able to:
> 
> 1) "steal" IP addresses through IP aliasing (e.g., domU has 1.2.3.4, and domU root does "ifconfig eth0:0 1.2.3.5/32" in Linux, and now has two working IPs),
> 2) and more importantly, were able to impact the network connectivity of another domU by aliasing or assigning its in-use IP address,
> 3) and MOST importantly, were able to impact the network connectivity for all domUs on a subnet by aliasing a gateway IP address (e.g., in Linux "ifconfig 1.2.3.1" for a typical /24 subnet). 

See the Xen antispoof functionality.  you specify the IPv4 address 
for the DomU in the xm config file, and it adds firewall rules to drop IPv4 
packets from the DomU that are not from its assigned address.  it works
well.  (you need to make your firewall put your bridge packets
through the forward chain, but after that it works well)  

If you have this working with IPv6, though, that would be really awesome.

> 4) Also, sending out invalid or poisoned ARP packets from one domU were able to introduce network connectivity problems for other domUs.

The Xen networking scripts, I believe, don't currently have anything that
stops mac spoofing/arp cache poisioning.  Something like that would be
pretty nice, though. (It would allow me to trust my interface counters
again)

> b) As above, have folks already addressed these issues for stealing IPs/ARP poisoning? Have they just not encountered them yet? Would it be useful to submit these modifications for review by the community?

the antispoof stuff in xen works fine for IPv4, but we've got nothing
for ARP poisoning, or for antispoof in IPv6.  both would be nice.

Re: Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Teck Choon Giam]

> I think people here are more than happy to receive the patches! Anyway
> I'm interested ;)

Me too :)

Re: Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Cory Von Wallenstein]

[-- Attachment #1: Type: text/plain, Size: 3854 bytes --]

I apologize folks for not getting back to the list in a timely matter. Alas, duty called.

I've put together a guide, as well as my patches, for:

a) Getting IPv6 anti-spoofing to work.
b) Preventing ARP poisoning attacks that can bring down IPv4 communication on a subnet.
c) Preventing IPv4 packet sniffing.

The guide walking through the "method to my madness" is on my engineering blog:

http://www.standingonthebrink.com/index.php/ipv6-ipv4-and-arp-on-xen-for-vps/

I walk through the specific iptables, ip6tables, and arptables rules I found necessary there.

The modified networking scripts are vif-common.sh, vif-bridge, and network-bridge. Diffs are attached. By no means do I consider these "THE answer", but have worked well for what we have in the field, and I welcome suggestions for improvements. 

Hope these help folks looking to use Xen for VPS hosting.

-- 
Cory von Wallenstein 
Dynamic Network Services Inc. 
P: +1.508.340.0958 
http://dynamicnetworkservices.com/ 

----- Original Message -----
From: "Luke S Crawford" <lsc@prgmr.com>
To: "Cory Von Wallenstein" <cvonwallenstein@dyn-inc.com>
Cc: xen-devel@lists.xensource.com, "Stephen Spector" <stephen.spector@citrix.com>
Sent: Friday, September 26, 2008 9:58:19 PM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-devel] Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

Cory Von Wallenstein <cvonwallenstein@dyn-inc.com> writes:


> a) Have people already solved and dealt with IPv6 in Xen successfully (i.e., is it a non-issue at this point)? If not, I'd be happy to submit the changes and a guide to making it work and work well.

Guides would be awesome.  Also, if you made anti-spoof work for IPv6, that 
would also be awesome.

I've had IPv6 going for a while with no problems (though the xen antispoof 
rules only work with IPv4.)  - but I didn't  even need to enable IPv6 in the 
Dom0;  it's a layer 2 bridge, so it just works.  In fact, stateless 
autoconfiguration even worked.   I asked for an IPv6 allocation from
my provider and my customers noticed it was working before I got around to
doing any setup at all.

what problems did you hit with IPv6?

> Along the way, we also ran into some issues where domUs were able to:
> 
> 1) "steal" IP addresses through IP aliasing (e.g., domU has 1.2.3.4, and domU root does "ifconfig eth0:0 1.2.3.5/32" in Linux, and now has two working IPs),
> 2) and more importantly, were able to impact the network connectivity of another domU by aliasing or assigning its in-use IP address,
> 3) and MOST importantly, were able to impact the network connectivity for all domUs on a subnet by aliasing a gateway IP address (e.g., in Linux "ifconfig 1.2.3.1" for a typical /24 subnet). 

See the Xen antispoof functionality.  you specify the IPv4 address 
for the DomU in the xm config file, and it adds firewall rules to drop IPv4 
packets from the DomU that are not from its assigned address.  it works
well.  (you need to make your firewall put your bridge packets
through the forward chain, but after that it works well)  

If you have this working with IPv6, though, that would be really awesome.

> 4) Also, sending out invalid or poisoned ARP packets from one domU were able to introduce network connectivity problems for other domUs.

The Xen networking scripts, I believe, don't currently have anything that
stops mac spoofing/arp cache poisioning.  Something like that would be
pretty nice, though. (It would allow me to trust my interface counters
again)

> b) As above, have folks already addressed these issues for stealing IPs/ARP poisoning? Have they just not encountered them yet? Would it be useful to submit these modifications for review by the community?

the antispoof stuff in xen works fine for IPv4, but we've got nothing
for ARP poisoning, or for antispoof in IPv6.  both would be nice.


[-- Attachment #2: network-bridge.diff --]
[-- Type: application/octet-stream, Size: 668 bytes --]

--- xen-unstable.hg/network-bridge	2008-10-08 13:50:15.000000000 -0400
+++ dynspring/network-bridge	2008-10-08 13:00:50.000000000 -0400
@@ -188,9 +179,18 @@
 antispoofing () {
     iptables -P FORWARD DROP
     iptables -F FORWARD
-    iptables -A FORWARD -m physdev --physdev-in ${pdev} -j ACCEPT
+    
+    ip6tables -P FORWARD DROP
+    ip6tables -F FORWARD
+    
+    arptables -P FORWARD DROP
+    arptables -F FORWARD    
+
+    # Below commented out by CvW on 08-24-2008 because I don't think we need it.
+    #iptables -A FORWARD -m physdev --physdev-in ${pdev} -j ACCEPT
 }
 
+
 # Usage: show_status dev bridge
 # Print ifconfig and routes.
 show_status () {

[-- Attachment #3: vif-bridge.diff --]
[-- Type: application/octet-stream, Size: 389 bytes --]

--- xen-unstable.hg/vif-bridge	2008-10-08 13:50:15.000000000 -0400
+++ dynspring/vif-bridge	2008-10-08 13:01:07.000000000 -0400
@@ -93,8 +93,15 @@
 
 handle_iptable
 
+# Added by CvW on 08-24-2008
+handle_ip6table
+
+# Added by CvW on 09-09-2008
+handle_arptable
+
 log debug "Successful vif-bridge $command for $vif, bridge $bridge."
 if [ "$command" == "online" ]
 then
   success
 fi
+

[-- Attachment #4: vif-common.sh.diff --]
[-- Type: application/octet-stream, Size: 8729 bytes --]

--- xen-unstable.hg/vif-common.sh	2008-10-08 13:50:15.000000000 -0400
+++ dynspring/vif-common.sh	2008-10-09 15:01:22.000000000 -0400
@@ -48,11 +48,14 @@
 ip=${ip:-}
 ip=$(xenstore_read_default "$XENBUS_PATH/ip" "$ip")
 
+# Added by CvW on 09-09-2008
+mac=${mac:-}
+mac=$(xenstore_read_default "$XENBUS_PATH/mac" "$mac")
+
 # Check presence of compulsory args.
 XENBUS_PATH="${XENBUS_PATH:?}"
 vif="${vif:?}"
 
-
 vifname=$(xenstore_read_default "$XENBUS_PATH/vifname" "")
 if [ "$vifname" ]
 then
@@ -64,7 +67,47 @@
 fi
 
 
-frob_iptable()
+# CvW: Duplicate of frob_iptable_physdevout, except ip6tables is used
+frob_ip6table_physdevout()
+{
+  if [ "$command" == "online" ]
+  then
+    local c="-A"
+  else
+    local c="-D"
+  fi
+
+  ip6tables "$c" FORWARD -m physdev --physdev-out "$vif" "$@" -j ACCEPT \
+    2>/dev/null ||
+    [ "$c" == "-D" ] ||
+    log err \
+     "ip6tables $c FORWARD -m physdev --physdev-out $vif $@ -j ACCEPT failed.
+If you are using iptables, this may affect networking for guest domains."
+}
+
+# CvW: Duplicate of frob_iptable, except ip6tables is used
+frob_ip6table()
+{
+  if [ "$command" == "online" ]
+  then
+    local c="-A"
+  else
+    local c="-D"
+  fi
+
+  # This is the default iptables edit that was included with the base
+  # Xen scripts
+  ip6tables "$c" FORWARD -m physdev --physdev-in "$vif" --physdev-out peth0 "$@" -j ACCEPT \
+    2>/dev/null ||
+    [ "$c" == "-D" ] ||
+    log err \
+     "ip6tables $c FORWARD -m physdev --physdev-in $vif --physdev-out peth0 $@ -j ACCEPT failed.
+If you are using iptables, this may affect networking for guest domains."
+}
+
+# CvW: This was added 8-24-2008 to add rules for physdev-out, which
+# means packets leaving the bridge and into the domU.
+frob_iptable_physdevout()
 {
   if [ "$command" == "online" ]
   then
@@ -73,14 +116,53 @@
     local c="-D"
   fi
 
-  iptables "$c" FORWARD -m physdev --physdev-in "$vif" "$@" -j ACCEPT \
+  iptables "$c" FORWARD -m physdev --physdev-out "$vif" "$@" -j ACCEPT \
     2>/dev/null ||
     [ "$c" == "-D" ] ||
     log err \
-     "iptables $c FORWARD -m physdev --physdev-in $vif $@ -j ACCEPT failed.
+     "iptables $c FORWARD -m physdev --physdev-out $vif $@ -j ACCEPT failed.
 If you are using iptables, this may affect networking for guest domains."
 }
 
+# CvW: This is modeled after frob_iptable.
+frob_arptable()
+{
+  if [ "$command" == "online" ]
+  then
+    local c="-A"
+  else
+    local c="-D"
+  fi
+
+  arptables "$c" FORWARD "$@" -j ACCEPT \
+    2>/dev/null ||
+    [ "$c" == "-D" ] ||
+    log err \
+     "arptables $c FORWARD $@ -j ACCEPT failed.
+If you are using arptables, this may affect networking for guest domains."
+}
+
+# CvW: This is the default iptable handling method, but it only configures
+# physdev-in, meaning packets leaving the domU and heading into the bridge
+# (i.e., out to the network). 
+frob_iptable()
+{
+  if [ "$command" == "online" ]
+  then
+    local c="-A"
+  else
+    local c="-D"
+  fi
+
+  # This is the default iptables edit that was included with the base
+  # Xen scripts
+  iptables "$c" FORWARD -m physdev --physdev-in "$vif" --physdev-out peth0 "$@" -j ACCEPT \
+    2>/dev/null ||
+    [ "$c" == "-D" ] ||
+    log err \
+     "iptables $c FORWARD -m physdev --physdev-in $vif --physdev-out peth0 $@ -j ACCEPT failed.
+If you are using iptables, this may affect networking for guest domains."
+}
 
 ##
 # Add or remove the appropriate entries in the iptables.  With antispoofing
@@ -105,17 +187,129 @@
       local addr
       for addr in $ip
       do
-        frob_iptable -s "$addr"
+          # Enable anti-spoof and anti-sniffing rules 
+          frob_iptable -s "$addr"
+          frob_iptable_physdevout -d "$addr"
       done
 
+      # Disabled by CvW on 8-24-08 since we don't use DHCP
       # Always allow the domain to talk to a DHCP server.
-      frob_iptable -p udp --sport 68 --dport 67
+      # frob_iptable -p udp --sport 68 --dport 67
+  else
+      # No IP addresses have been specified, so log an error.
+      log err "No IP address was specified. Default behavior would have been
+to allow anything, but this changed on 8-24-08 by CvW to throw this error and
+enable nothing."
+  fi
+}
+
+# Added by CvW
+handle_ip6table()
+{ 
+  # Check for a working ip6tables installation.  Checking for the ip6tables
+  # binary is not sufficient, because the user may not have the appropriate
+  # modules installed.  If ip6tables is not working, then there's no need to do
+  # anything with it, so we can just return.
+  if ! ip6tables -L -n >&/dev/null
+  then
+    log err "The ip6tables binary was not found." 
+    return 
+  fi
+
+  # Currently limiting IPv6 to just the public interface VIF0, which will map to eth0 in the domUs
+  VIF0_REGEX="vif(.*)\.0"
+  if [[ "$vif" =~ "$VIF0_REGEX" ]]; then
+      log debug "Will configure $vif for IPv6 in ip6tables."
+  else
+      log debug "Will NOT configure $vif for IPv6 in ip6tables."
+      return
+  fi
+
+  if [ "$command" == "online" ]
+  then
+      # For bringing the interface online, we'll need to get the IPv6 address from the
+      # /etc/xen/domU.cfg file, which we've stored in a proprietary "#ipv6=" comment.
+
+      # We need the domain ID to get the domain name to get the config file... Ugh!   
+      DOMID_REGEX="backend/vif/(.*)/" 
+
+      if [[ "$XENBUS_PATH" =~ "$DOMID_REGEX" ]]; then
+          DOMID=${BASH_REMATCH[1]}
+      else 
+          log err "Failed to hack domID from XENBUS_PATH ${XENBUS_PATH} for setting the ip6tables rules."
+          return
+      fi
+
+      DOMU_NAME=$(xenstore_read "/local/domain/$DOMID/name")
+      log debug "Preparing to search for IPv6 addresses for domU ${DOMU_NAME}"
+
+      DOMU_CONFIG=`cat /etc/xen/${DOMU_NAME}.cfg`
+      IPV6_REGEX="#ipv6=([a-fA-F0-9]{0,4}:[a-fA-F0-9]{0,4}:[a-fA-F0-9]{0,4}:[a-fA-F0-9]{0,4}:[a-fA-F0-9]{0,4}:[a-fA-F0-9]{0,4}:[a-fA-F0-9]{0,4}:[a-fA-F0-9]{4})"
+
+      if [[ "$DOMU_CONFIG" =~ "$IPV6_REGEX" ]]; then
+          IPV6_ADDRESS=${BASH_REMATCH[1]}
+      else
+          log info "No IPv6 address provided in config for ${DOMU_NAME}. Will not configure ip6tables."
+          return
+      fi
+      $(xenstore_write "$XENBUS_PATH/springipv6" "$IPV6_ADDRESS")
   else
-      # No IP addresses have been specified, so allow anything.
-      frob_iptable
+      # For anything else, we can't guarantee access to the name of the domU in this code
+      # to retrieve the IPv6 address from the file. Instead, we'll need to read it from
+      # the xenstore, where we placed it during the "online" command.
+      IPV6_ADDRESS=$(xenstore_read_default "$XENBUS_PATH/springipv6" "")
+      if [ "$IPV6_ADDRESS" == ""]; then
+          log err "Failed to retrieve the IPv6 address from the hackish springipv6 xenstore entry. Probably leaving ip6tables rules floating arround."
+          return
+      fi
   fi
+      
+  log info "Configuring ${IPV6_ADDRESS} in ip6tables for command ${command}." 
+
+  frob_ip6table -s "$IPV6_ADDRESS"
+  frob_ip6table_physdevout -d "$IPV6_ADDRESS"
+  
+  log info "Done configuring ${IPV6_ADDRESS} in ip6tables." 
 }
 
+# Added by CvW on 09-09-2008
+handle_arptable()
+{
+  # Check for a working arptables installation.  Checking for the arptables
+  # binary is not sufficient, because the user may not have the appropriate
+  # modules installed.  If arptables is not working, then there's no need to do
+  # anything with it, so we can just return.
+  if ! arptables -L -n >&/dev/null
+  then
+    return
+  fi
+  
+  # This method is called once per virtual interface. This means we're guaranteed
+  # to only see a single MAC address. It is possible, however, to see multiple
+  # IP addresses for this MAC address
+
+  if [ "$mac" != "" ] &&
+     [ "$ip" != "" ]  
+  then
+      # Restrict arp traffic, such that all requests pass successfully, but
+      # this virtual interface can only provide a reply if it is an approved
+      # MAC/IP pair.
+      frob_arptable --opcode Request --in-interface "$vif"
+      frob_arptable --opcode Request --out-interface "$vif"
+      frob_arptable --opcode Reply --out-interface "$vif" --in-interface peth0
+
+	  # The rule restricting replies must be done for each valid IP address.
+      local addr
+      for addr in $ip
+      do
+          frob_arptable --opcode Reply --in-interface "$vif" --source-ip "$addr" --source-mac "$mac"
+      done
+  else
+      # No MAC or IP addresses have been specified, so log an error.
+      log err "No MAC or IP address was specified. Default DROP policy applies to this domU, and
+no ARP traffic will be allowed through the bridge."
+  fi
+}
 
 ##
 # ip_of interface
@@ -149,3 +343,4 @@
   fi
   echo "$result"
 }
+

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Teck Choon Giam]

On Fri, Oct 10, 2008 at 4:09 AM, Cory Von Wallenstein
<cvonwallenstein@dyn-inc.com> wrote:
> I apologize folks for not getting back to the list in a timely matter. Alas, duty called.

No apology is needed.  Greatly appreciate your sharing/contribution :)

> I've put together a guide, as well as my patches, for:
>
> a) Getting IPv6 anti-spoofing to work.
> b) Preventing ARP poisoning attacks that can bring down IPv4 communication on a subnet.
> c) Preventing IPv4 packet sniffing.
>
> The guide walking through the "method to my madness" is on my engineering blog:
>
> http://www.standingonthebrink.com/index.php/ipv6-ipv4-and-arp-on-xen-for-vps/

Nice blog and many thanks for your effort to put up such informative
blog related to Xen.

>
> The modified networking scripts are vif-common.sh, vif-bridge, and network-bridge. Diffs are attached. By no means do I consider these "THE answer", but have worked well for what we have in the field, and I welcome suggestions for improvements.

I have a suggestion since I just tried your patches in one of my test
server.  All my servers are with two NICs at least and running with
xen-3.3.0.  I encountered domU can't be started and when I check
/var/log/xen/xen-hotplug.log as below:

/etc/xen/scripts/vif-common.sh: line 261: [: : unary operator expected
Nothing to flush.
cat: /etc/xen/domU.cfg: No such file or directory
Nothing to flush.
/etc/xen/scripts/vif-common.sh: line 261: [: : unary operator expected
Nothing to flush.
Nothing to flush.
cat: /etc/xen/domU.cfg: No such file or directory
/etc/xen/scripts/vif-common.sh: line 261: [: : unary operator expected

So in vif-common.sh I modified from yours as below:

DOMU_CONFIG=`cat /etc/xen/${DOMU_NAME}.cfg`

to:

if [ -f "/etc/xen/${DOMU_NAME}" ] ; then
    DOMU_CONFIG=`cat /etc/xen/${DOMU_NAME}`
elsif [ -f "/etc/xen/${DOMU_NAME}.cfg" ] ; then
    DOMU_CONFIG=`cat /etc/xen/${DOMU_NAME}.cfg`
else
    DOMU_CONFIG=""
fi

After the above changes, I am able to start my domU as per normal.  I
am using /etc/xen/${DOMU_NAME} instead of with .cfg extension for my
domU config files.

Once again, thanks.

Kindest regards,
Giam Teck Choon

Re: Successful IPv6 Xen Deployment; Protection Against IPv4 ARP Poisoning Attacks

[Teck Choon Giam]

> elsif [ -f "/etc/xen/${DOMU_NAME}.cfg" ] ; then

Sorry typo should be:
elif [ -f "/etc/xen/${DOMU_NAME}.cfg" ] ; then

Kindest regards,
Giam Teck Choon