Stimulating domains to send gratuitous ARPs

Stimulating domains to send gratuitous ARPs

[Graham, Simon]

[-- Attachment #1.1: Type: text/plain, Size: 1369 bytes --]

Currently, netfront will send a gratuitous ARP when resumed at the end
of a live migration/resume op. However, there are other situations when
it is necessary to send a gratuitous ARP; one good example is when you
bond NICs in Dom0 for availability and plug the bond into the bridge; in
this case, when a failover from primary to backup NIC occurs, you need
to send ARPs for all the domains VIFs on the backup link to get traffic
routed correctly.

 

Now, there are a number of ways of doing this (including using ebtables
in Dom0 to track IP-MAC address translations) but it seems to me that
the simplest would be to have a way to poke the netfront driver to send
an ARP on demand (especially given that fact that it already needs/has
this functionality).

 

I've prototyped something fairly simple & gross based on having netfront
setup a xenstore watch on a leaf in the domains vif directory in
xenstore (that is, device/vif/send_arp); when the watch fires, all
netfront instances send the fake ARP just like they do at the end of
migration/resume. All you need to do from dom0 then is modify the leaf
with xenstore-write/xenstore-rm. 

 

I accept that this is not very nice and we should probably have a more
organized/documented way to do this but I'm not sure what would be
appropriate; any suggestions?

 

Thanks

Simon


[-- Attachment #1.2: Type: text/html, Size: 3495 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Stimulating domains to send gratuitous ARPs

[Ian Campbell]

On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> I accept that this is not very nice and we should probably have a more
> organized/documented way to do this but I’m not sure what would be
> appropriate; any suggestions?

A usespace tool to send such an ARP would be a very useful thing to
have, if it doesn't already exist.

The gratuitous arping in netfront is not very popular with the upstream
Linux folks so this functionality is not in the paravirt_ops patches
which are going upstream, so we'd need such a tool in the end anyway.

Ian.


 

RE: Stimulating domains to send gratuitous ARPs

[Graham, Simon]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]

There are existing tools for sending arps -- the problem is knowing what IP address to use in the ARP -- it's actually pretty hard to know this in Dom0 (which is where you know that the ARP needs to be sent) and if you install it in DomU you need a way to kick the domain to actually call the utility... (plus you have to install a service in DomU which is something I think is a bad idea to require).

Whilst I might agree that from a purist point of view having netfront do this is a violation of layering, it's already accepted in some other upstream drivers -- bonding for example; this virtual network interface driver will send gratuitous ARPs for the IP address associated with the bond and for any VLANs layered directly on top of the bond when failing over (it just wont send ARPs for things above the bridge if it is plugged into a bridge).

Simon

> -----Original Message-----
> From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> Sent: Thursday, July 05, 2007 3:46 PM
> To: Graham, Simon
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> 
> On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > I accept that this is not very nice and we should probably have a
> more
> > organized/documented way to do this but I’m not sure what would be
> > appropriate; any suggestions?
> 
> A usespace tool to send such an ARP would be a very useful thing to
> have, if it doesn't already exist.
> 
> The gratuitous arping in netfront is not very popular with the upstream
> Linux folks so this functionality is not in the paravirt_ops patches
> which are going upstream, so we'd need such a tool in the end anyway.
> 
> Ian.
> 
> 
> 
> 


[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Stimulating domains to send gratuitous ARPs

[Ian Pratt]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

> There are existing tools for sending arps -- the problem is knowing what
> IP address to use in the ARP -- it's actually pretty hard to know this
> in Dom0 (which is where you know that the ARP needs to be sent) and if
> you install it in DomU you need a way to kick the domain to actually
> call the utility... (plus you have to install a service in DomU which is
> something I think is a bad idea to require).
> 
> Whilst I might agree that from a purist point of view having netfront do
> this is a violation of layering, it's already accepted in some other
> upstream drivers 

It's pretty clear it's not going to be accepted in the netfront that goes into upstream Linux, at least not without sedating various LKML folk first :)

For bond case you described, you don't need to send ARPs, you just need to send any packet with the right src MAC to cause the switches to reconfigure. Since the bond driver likely already maintains a list of MAC addrs its operating on behalf of this is easy.

Ian


> Simon
> 
> > -----Original Message-----
> > From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> > Sent: Thursday, July 05, 2007 3:46 PM
> > To: Graham, Simon
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> >
> > On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > > I accept that this is not very nice and we should probably have a
> > more
> > > organized/documented way to do this but I’m not sure what would be
> > > appropriate; any suggestions?
> >
> > A usespace tool to send such an ARP would be a very useful thing to
> > have, if it doesn't already exist.
> >
> > The gratuitous arping in netfront is not very popular with the
> upstream
> > Linux folks so this functionality is not in the paravirt_ops patches
> > which are going upstream, so we'd need such a tool in the end anyway.
> >
> > Ian.
> >
> >
> >
> >


[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Stimulating domains to send gratuitous ARPs

[Zhu Han]

On 7/5/07, Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
> For bond case you described, you don't need to send ARPs, you just need to send any packet with the right src MAC to cause the switches to reconfigure. Since the bond driver likely already maintains a list of MAC addrs its operating on behalf of this is easy.

Ian, that's right.

But you can't depend on it just like you can't depend on the VM
continues to talk with the right mac address after the migration. :-)

>
> Ian
>
>
> > Simon
> >
> > > -----Original Message-----
> > > From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> > > Sent: Thursday, July 05, 2007 3:46 PM
> > > To: Graham, Simon
> > > Cc: xen-devel@lists.xensource.com
> > > Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> > >
> > > On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > > > I accept that this is not very nice and we should probably have a
> > > more
> > > > organized/documented way to do this but I'm not sure what would be
> > > > appropriate; any suggestions?
> > >
> > > A usespace tool to send such an ARP would be a very useful thing to
> > > have, if it doesn't already exist.
> > >
> > > The gratuitous arping in netfront is not very popular with the
> > upstream
> > > Linux folks so this functionality is not in the paravirt_ops patches
> > > which are going upstream, so we'd need such a tool in the end anyway.
> > >
> > > Ian.
> > >
> > >
> > >
> > >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>


-- 
best regards,
hanzhu

RE: Stimulating domains to send gratuitous ARPs

[Graham, Simon]

[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]

> > Whilst I might agree that from a purist point of view having netfront
> do
> > this is a violation of layering, it's already accepted in some other
> > upstream drivers
> 
> It's pretty clear it's not going to be accepted in the netfront that
> goes into upstream Linux, at least not without sedating various LKML
> folk first :)
> 

Fine (well, not fine at all really, but I understand ;-).

> For bond case you described, you don't need to send ARPs, you just need
> to send any packet with the right src MAC to cause the switches to
> reconfigure. Since the bond driver likely already maintains a list of
> MAC addrs its operating on behalf of this is easy.
> 

Well, the bonding module itself doesn't track this at all actually; when a failover is done, it sends gratuitous ARPs for the following:
1. The IP address associated with the bond itself
2. The IP addresses associated with an VLANs above the bond (suitably encapsulated in the VLAN header).

In addition, I don’t think that 'any old frame' will work -- it needs to be a link-level broadcast frame to ensure that all the switches update their forwarding tables -- in the example I cited of bonding for availability, it's entirely possible (in fact it's desirable) to plug the two NICs that are bonded into different switches. 

This is really exactly the same as the migration case (which also doesn't need to be an ARP from the point of view of updating ARP caches, it just needs to be a link level broadcast frame). So I think we both need a Dom0 based mechanism!

Assuming the netfront approach is dead, we need to identify a link level broadcast frame that can be safely sent to update the bridge forwarding tables with no other side effects... perhaps a RARP rq since this is deprecated anyway now, so it will probably get no answer and even if it does, the answer should be ignored???

Simon

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Stimulating domains to send gratuitous ARPs

[Ian Pratt]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

> Well, the bonding module itself doesn't track this at all actually; when
> a failover is done, it sends gratuitous ARPs for the following:
> 1. The IP address associated with the bond itself
> 2. The IP addresses associated with an VLANs above the bond (suitably
> encapsulated in the VLAN header).

The Linux bond module isn't well suited to the L2 bridged case anyway. To get the effect you want I think you'll have to add MAC addr tracking anyhow.

> In addition, I don’t think that 'any old frame' will work -- it needs to
> be a link-level broadcast frame to ensure that all the switches update
> their forwarding tables 

Sure, it needs to be a link-layer broadcast, but it doesn't particularly even need to be a valid IP/ARP packet. Broadcast arp reply works as well as anything, and has the additional benefit of also helping folk who do happen to be using a routed rather than L2 setup.

Best,
Ian

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Stimulating domains to send gratuitous ARPs

[Jeremy Fitzhardinge]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Ian Pratt wrote:
>> Whilst I might agree that from a purist point of view having netfront do
>> this is a violation of layering, it's already accepted in some other
>> upstream drivers 
>>     
>
> It's pretty clear it's not going to be accepted in the netfront that goes into upstream Linux, at least not without sedating various LKML folk first :)
>   

Stephen Hemminger posted a patch to add this capability generically so 
it wouldn't be a netfront-specific piece of behaviour.  I have not 
submitted it for upstream yet, mainly because nobody has 
confirmed/tested that it really does what we need.  But it looks 
reasonable, and it should be accepted once we confirm its correct.

    J

[-- Attachment #2: ARP-notify-option.patch --]
[-- Type: text/x-patch, Size: 3093 bytes --]

Subject: ARP notify option

This adds another inet device option to enable gratuitous ARP
when device is brought up or address change. This is handy for
clusters or virtualization.

Tested on a normal device (not Xen).

Signed-off-by: Stephen Hemminger <shemminge@linux-foundation.org>

---
 Documentation/networking/ip-sysctl.txt |    6 ++++++
 include/linux/inetdevice.h             |    2 ++
 include/linux/sysctl.h                 |    1 +
 net/ipv4/devinet.c                     |   16 ++++++++++++++++
 4 files changed, 25 insertions(+)

===================================================================
--- net-2.6.22.orig/Documentation/networking/ip-sysctl.txt
+++ net-2.6.22/Documentation/networking/ip-sysctl.txt
@@ -732,6 +732,12 @@
 	The max value from conf/{all,interface}/arp_ignore is used
 	when ARP request is received on the {interface}
 
+arp_notify - BOOLEAN
+	Define mode for notification of address and device changes.
+	0 - (default): do nothing
+	1 - Generate gratuitous arp replies when device is brought up
+	    or hardware address changes.
+
 arp_accept - BOOLEAN
 	Define behavior when gratuitous arp replies are received:
 	0 - drop gratuitous arp frames
===================================================================
--- net-2.6.22.orig/include/linux/inetdevice.h
+++ net-2.6.22/include/linux/inetdevice.h
@@ -26,6 +26,7 @@
 	int	arp_announce;
 	int	arp_ignore;
 	int	arp_accept;
+	int	arp_notify;
 	int	medium_id;
 	int	no_xfrm;
 	int	no_policy;
@@ -84,6 +85,7 @@
 #define IN_DEV_ARPFILTER(in_dev)	(ipv4_devconf.arp_filter || (in_dev)->cnf.arp_filter)
 #define IN_DEV_ARP_ANNOUNCE(in_dev)	(max(ipv4_devconf.arp_announce, (in_dev)->cnf.arp_announce))
 #define IN_DEV_ARP_IGNORE(in_dev)	(max(ipv4_devconf.arp_ignore, (in_dev)->cnf.arp_ignore))
+#define IN_DEV_ARP_NOTIFY(in_dev)	(ipv4_devconf.arp_notify || (in_dev)->cnf.arp_notify)
 
 struct in_ifaddr
 {
===================================================================
--- net-2.6.22.orig/include/linux/sysctl.h
+++ net-2.6.22/include/linux/sysctl.h
@@ -495,6 +495,7 @@
 	NET_IPV4_CONF_ARP_IGNORE=19,
 	NET_IPV4_CONF_PROMOTE_SECONDARIES=20,
 	NET_IPV4_CONF_ARP_ACCEPT=21,
+	NET_IPV4_CONF_ARP_NOTIFY=22,
 	__NET_IPV4_CONF_MAX
 };
 
===================================================================
--- net-2.6.22.orig/net/ipv4/devinet.c
+++ net-2.6.22/net/ipv4/devinet.c
@@ -1089,6 +1089,14 @@
 			}
 		}
 		ip_mc_up(in_dev);
+		/* fall through */
+	case NETDEV_CHANGEADDR:
+		if (IN_DEV_ARP_NOTIFY(in_dev))
+			arp_send(ARPOP_REQUEST, ETH_P_ARP,
+				 in_dev->ifa_list->ifa_address,
+				 dev,
+				 in_dev->ifa_list->ifa_address,
+				 NULL, dev->dev_addr, NULL);
 		break;
 	case NETDEV_DOWN:
 		ip_mc_down(in_dev);
@@ -1495,6 +1503,14 @@
 			.proc_handler	= &proc_dointvec,
 		},
 		{
+			.ctl_name	= NET_IPV4_CONF_ARP_NOTIFY,
+			.procname	= "arp_notify",
+			.data		= &ipv4_devconf.arp_notify,
+			.maxlen		= sizeof(int),
+			.mode		= 0644,
+			.proc_handler	= &proc_dointvec,
+		},
+		{
 			.ctl_name	= NET_IPV4_CONF_NOXFRM,
 			.procname	= "disable_xfrm",
 			.data		= &ipv4_devconf.no_xfrm,

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Stimulating domains to send gratuitous ARPs

[Graham, Simon]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

> 
> Stephen Hemminger posted a patch to add this capability generically so
> it wouldn't be a netfront-specific piece of behaviour.  I have not
> submitted it for upstream yet, mainly because nobody has
> confirmed/tested that it really does what we need.  But it looks
> reasonable, and it should be accepted once we confirm its correct.
> 

-sigh- ... doesn't really help unfortunately for a couple of reasons:

0. It's in some as yet unreleased kernel version -- doesn't help with backlevel DomUs
1. You still have to configure this in every DomU since the default is to not do it.
2. There's no way to stimulate this from Dom0 as needed (e.g. at the end of migration
   or when some other change occurs in Dom0).

Right now, I'm experimenting with a solution that sends an invalid broadcast RARP frame... if this works for me, it'll also work for the migration/resume case...

Simon

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Stimulating domains to send gratuitous ARPs

[Zhu Han]

On 7/5/07, Graham, Simon <Simon.Graham@stratus.com> wrote:
> Whilst I might agree that from a purist point of view having netfront do this is a violation of layering, it's already accepted in some other upstream drivers -- bonding for example; this virtual network interface driver will send gratuitous ARPs for the IP address associated with the bond and for any VLANs layered directly on top of the bond when failing over (it just wont send ARPs for things above the bridge if it is plugged into a bridge).
>

I guess the user mode helper should be inside DOM0 unless you wanna
ask the user to intall some extra tools as vmware does, considering
about HVM domain.

The user mode helper in DOM0 should watch the connect status between
netback and netfront. If it becomes connected, then sends out an fake
ethernet frame, not an fake arp reply because you can't know the IP
address.

Xentore can provide the mac address of netfront but netfront has to
register set_mac_address method to update it when the user try to
modify it.

> Simon
>
> > -----Original Message-----
> > From: Ian Campbell [mailto:Ian.Campbell@xensource.com]
> > Sent: Thursday, July 05, 2007 3:46 PM
> > To: Graham, Simon
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Stimulating domains to send gratuitous ARPs
> >
> > On Thu, 2007-07-05 at 14:41 -0400, Graham, Simon wrote:
> > > I accept that this is not very nice and we should probably have a
> > more
> > > organized/documented way to do this but I'm not sure what would be
> > > appropriate; any suggestions?
> >
> > A usespace tool to send such an ARP would be a very useful thing to
> > have, if it doesn't already exist.
> >
> > The gratuitous arping in netfront is not very popular with the upstream
> > Linux folks so this functionality is not in the paravirt_ops patches
> > which are going upstream, so we'd need such a tool in the end anyway.
> >
> > Ian.
> >
> >
> >
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>


-- 
best regards,
hanzhu

Re: Stimulating domains to send gratuitous ARPs

[Zhu Han]

On 7/5/07, Graham, Simon <Simon.Graham@stratus.com> wrote:
>
> I agree it's not good. However, in the example I quoted with a bond
> underneath the bridge, the netbridge code doesn't even know there are
> two paths or that it should send an update (in fact, there would be no
> update since the port config hasn't changed as far as the bridge code is
> concerned).

It's not a problem only for Xen.  So, you may find a solution between
bond device driver and bridge code.

So, the bond device driver broadcast a netlink message and then the
usr mode helper tells bridge to broadcast the mac address in its
forwarding table. :-)

You may need help from linux net-devel.

>
> Simon
>


-- 
best regards,
hanzhu