lost gARP after live migration

lost gARP after live migration

[Laszlo Ersek]

Hi,

with reference to RHBZ#713585:

It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the 
gratuitous ARP packet is not forwarded to the affected "networking 
equipment". The netback vif is added to a routed bridge in the host(s) 
and external hosts are expeted to have connection to the guest at all 
times, no matter the current Xen host.

I experimented a bit with tcpdump, and the gARP does appear on the 
netfront interface. It also appears on the host bridge if sufficient 
time passes between completing the xenbus handshake and sending the gARP.

When the guest queues eg. three gARPs in rapid succession, a variable 
number of them gets lost. (When all such packets disappear, then the 
migrated guest becomes invisible to the outside world, until it 
initiates network traffic on its own.)

When the guest waits for about half a second before sending (queueing), 
the very first gARP packet successfully appears on the host bridge.

I suspect it's a timing race against the netback vif being added to the 
host bridge. What would be a good countermeasure?

- Adding two modparams to xen-netfront (gARP requeue count & number of 
msecs to wait between queueing the gARPs).
- (Paolo's idea:) watching the "hotplug-status" xenstore node and 
sending a single gARP when the watch fires with "connected". This node 
belongs to the backend xenstore subtree, thus watching it from the guest 
doesn't please the architecture astronaut in me.
- Something else.

Sorry for the naivety / verbiage.

Thanks,
lacos

Re: lost gARP after live migration

[Paolo Bonzini]

On 06/28/2011 03:01 PM, Laszlo Ersek wrote:
> - (Paolo's idea:) watching the "hotplug-status" xenstore node and
> sending a single gARP when the watch fires with "connected". This node
> belongs to the backend xenstore subtree, thus watching it from the guest
> doesn't please the architecture astronaut in me.

Note that watching the backend and reading its information is quite 
common.  In fact, that's how the state of the backend is observed in the 
first place.  Of course you cannot write to the backend tree, but you do 
not have to do that.

Paolo

Re: lost gARP after live migration

[Ian Campbell]

On Tue, 2011-06-28 at 14:01 +0100, Laszlo Ersek wrote:
> Hi,
> 
> with reference to RHBZ#713585:
> 
> It seems when a RHEL-6.1 or F-15 Xen PV guest is live migrated, the 
> gratuitous ARP packet is not forwarded to the affected "networking 
> equipment". The netback vif is added to a routed bridge in the host(s) 
> and external hosts are expeted to have connection to the guest at all 
> times, no matter the current Xen host.
> 
> I experimented a bit with tcpdump, and the gARP does appear on the 
> netfront interface. It also appears on the host bridge if sufficient 
> time passes between completing the xenbus handshake and sending the gARP.
> 
> When the guest queues eg. three gARPs in rapid succession, a variable 
> number of them gets lost. (When all such packets disappear, then the 
> migrated guest becomes invisible to the outside world, until it 
> initiates network traffic on its own.)
> 
> When the guest waits for about half a second before sending (queueing), 
> the very first gARP packet successfully appears on the host bridge.
> 
> I suspect it's a timing race against the netback vif being added to the 
> host bridge. What would be a good countermeasure?
> 
> - Adding two modparams to xen-netfront (gARP requeue count & number of 
> msecs to wait between queueing the gARPs).
> - (Paolo's idea:) watching the "hotplug-status" xenstore node and 
> sending a single gARP when the watch fires with "connected". This node 
> belongs to the backend xenstore subtree, thus watching it from the guest 
> doesn't please the architecture astronaut in me.

netback already waits (or should...) for hotplug-status to fire with
"connected" before moving to state XenbusStateConnected. See
hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need
either the netback in upstream or something newer than 43223efd9bfd (C
Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit
fixes pretty much the issue you describe.

I expected that netfront waited for the backend to hit
XenbusStateConnected before sending the grat ARP but instead I find it
happens when the backend hits XenbusStateInitWait. I'm not sure if that
is a problem -- it appears to have been done this way since forever
(even back in the classic Xen kernels) and I've never noticed a gARP go
missing in the way you describe, but perhaps something isn't quite
matching up any more.

Ian.

> - Something else.
> 
> Sorry for the naivety / verbiage.
> 
> Thanks,
> lacos
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: lost gARP after live migration

[Ben Hutchings]

On Tue, 2011-06-28 at 15:01 +0200, Laszlo Ersek wrote:
[...]
> When the guest waits for about half a second before sending (queueing), 
> the very first gARP packet successfully appears on the host bridge.
> 
> I suspect it's a timing race against the netback vif being added to the 
> host bridge. What would be a good countermeasure?
> 
> - Adding two modparams to xen-netfront (gARP requeue count & number of 
> msecs to wait between queueing the gARPs).

Note that peer notifications are indirected through netdev notifiers and
now include IPv6 NAs as well as ARPs.  If repeated notifications are
commonly necessary then this should probably be handled in the protocol
(or in the networking core).  However this sounds like a workaround
whereas your other option would be a proper fix:

> - (Paolo's idea:) watching the "hotplug-status" xenstore node and 
> sending a single gARP when the watch fires with "connected". This node 
> belongs to the backend xenstore subtree, thus watching it from the guest 
> doesn't please the architecture astronaut in me.
[...]

-- 
Ben Hutchings, Senior Software Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

[PATCH] xen-netfront: delay gARP until backend switches to Connected

[Laszlo Ersek]

On 06/28/11 15:33, Ian Campbell wrote:

> netback already waits (or should...) for hotplug-status to fire with
> "connected" before moving to state XenbusStateConnected. See
> hotplug_status_changed in drivers/net/xen-netback/xenbus.c. You need
> either the netback in upstream or something newer than 43223efd9bfd (C
> Feb 2010) if you are using e.g. xen.git#xen/next-2.6.32. That commit
> fixes pretty much the issue you describe.
> 
> I expected that netfront waited for the backend to hit
> XenbusStateConnected before sending the grat ARP but instead I find it
> happens when the backend hits XenbusStateInitWait. I'm not sure if that
> is a problem -- it appears to have been done this way since forever
> (even back in the classic Xen kernels) and I've never noticed a gARP go
> missing in the way you describe, but perhaps something isn't quite
> matching up any more.

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the
following in the RHEL-6 guest, in order to fix the network outage after live
migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest
(without the patch), and the backport to the host side alone was not sufficient
there either.

Thanks for the help and for considering this one.

Signed-off-by: Laszlo Ersek <lersek@redhat.com>
---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 3f71199..214c3a4 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
-- 
1.7.4.4

Re: [PATCH] xen-netfront: delay gARP until backend switches to Connected

[Konrad Rzeszutek Wilk]

> migration. The patch is for xen/next-2.6.32. I also tested a Fedora-15 guest
> (without the patch), and the backport to the host side alone was not sufficient
> there either.

I am having a hard time parsing that. It sounds like this patch is also
neccessary for upstream kernel? Or do you have another one for the upstream
kernel that you are going to submit soon?
(If so, please CC David Miller on it).

> 
> Thanks for the help and for considering this one.
> 
> Signed-off-by: Laszlo Ersek <lersek@redhat.com>
> ---
>  drivers/net/xen-netfront.c |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index 3f71199..214c3a4 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -1732,7 +1732,6 @@ static void netback_changed(struct xenbus_device *dev,
>  	case XenbusStateInitialised:
>  	case XenbusStateReconfiguring:
>  	case XenbusStateReconfigured:
> -	case XenbusStateConnected:
>  	case XenbusStateUnknown:
>  	case XenbusStateClosed:
>  		break;
> @@ -1743,6 +1742,9 @@ static void netback_changed(struct xenbus_device *dev,
>  		if (xennet_connect(netdev) != 0)
>  			break;
>  		xenbus_switch_state(dev, XenbusStateConnected);
> +		break;
> +
> +	case XenbusStateConnected:
>  		netif_notify_peers(netdev);
>  		break;
>  
> -- 
> 1.7.4.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[PATCH v2] xen-netfront: delay gARP until backend switches to Connected

[Laszlo Ersek]

In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the
following in the RHEL-6 guest, in order to fix the network outage after live
migration. I also tested a Fedora-15 guest (without the patch), and the
backport to the host side alone was not sufficient there either. Therefore I
think it could be useful to apply this guest-side patch to upstream as well.

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585

v1->v2: "port" to Linus's tree; add RHBZ ref.

Signed-off-by: Laszlo Ersek <lersek@redhat.com>
---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index d29365a..f033656 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
-- 
1.7.4.4

Re: [PATCH v2] xen-netfront: delay gARP until backend switches to Connected

[Konrad Rzeszutek Wilk]

On Wed, Jul 13, 2011 at 01:44:47PM +0200, Laszlo Ersek wrote:
> In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed the
> following in the RHEL-6 guest, in order to fix the network outage after live
> migration. I also tested a Fedora-15 guest (without the patch), and the

Laszlo,

This description is .. well, pointless for upstream patches. Just succinctly
describe the problem, how to reproduce it, and what this patch does.

And do mention more about 43223efd9bfd, like "this is the front
counterpart of patch xen: netback: wait for hotplug scripts to complete before
signalling connected to frontend (43223efd9bfd)"

> backport to the host side alone was not sufficient there either. Therefore I
> think it could be useful to apply this guest-side patch to upstream as well.
> 
> Reference: https://bugzilla.redhat.com/show_bug.cgi?id=713585
> 
> v1->v2: "port" to Linus's tree; add RHBZ ref.
> 
> Signed-off-by: Laszlo Ersek <lersek@redhat.com>
> ---
>  drivers/net/xen-netfront.c |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index d29365a..f033656 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
>  	case XenbusStateInitialised:
>  	case XenbusStateReconfiguring:
>  	case XenbusStateReconfigured:
> -	case XenbusStateConnected:
>  	case XenbusStateUnknown:
>  	case XenbusStateClosed:
>  		break;
> @@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
>  		if (xennet_connect(netdev) != 0)
>  			break;
>  		xenbus_switch_state(dev, XenbusStateConnected);
> +		break;
> +
> +	case XenbusStateConnected:
>  		netif_notify_peers(netdev);
>  		break;
>  
> -- 
> 1.7.4.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[PATCH v3] xen-netfront: delay gARP until backend switches to Connected

[Laszlo Ersek]

On 07/13/11 15:29, Konrad Rzeszutek Wilk wrote:
> On Wed, Jul 13, 2011 at 01:44:47PM +0200, Laszlo Ersek wrote:
>> In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed
>> the following in the RHEL-6 guest, in order to fix the network outage after
>> live migration. I also tested a Fedora-15 guest (without the patch), and the
>
> Laszlo,
>
> This description is .. well, pointless for upstream patches. Just succinctly
> describe the problem, how to reproduce it, and what this patch does.

I avoided writing up the commit message myself because in my first posting [1]
I quoted two paragraphs from Ian Campbell [2]. I intended those two cited
paragraphs verbatim as the commit message (along with my short remarks),
because they describe the problem exactly. Anyway:

After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP
message, so that networking hardware on the target host's subnet can take
notice, and public routing to the guest is re-established. However, if the
packet appears on the backend interface before the backend is added to the
target host's bridge, the packet is lost, and the migrated guest's peers become
unable to talk to the guest.

A sufficient two-parts condition to prevent the above is:

(1) ensure that the backend only moves to Connected xenbus state after its
hotplug scripts completed, ie. the netback interface got added to the bridge;
and

(2) ensure the frontend only queues the gARP when it sees the backend move to
Connected.

These two together provide complete ordering. Sub-condition (1) is satisfied
by pvops commit 43223efd9bfd.

In general, the full condition is sufficient, not necessary, because, according
to [2], live migration has been working for a long time without satisfying
sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5
host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch
intends to provide (2) for upstream.

[1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html
[2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html

Signed-off-by: Laszlo Ersek <lersek@redhat.com>
---
 drivers/net/xen-netfront.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index d29365a..f033656 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
 	case XenbusStateInitialised:
 	case XenbusStateReconfiguring:
 	case XenbusStateReconfigured:
-	case XenbusStateConnected:
 	case XenbusStateUnknown:
 	case XenbusStateClosed:
 		break;
@@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
 		if (xennet_connect(netdev) != 0)
 			break;
 		xenbus_switch_state(dev, XenbusStateConnected);
+		break;
+
+	case XenbusStateConnected:
 		netif_notify_peers(netdev);
 		break;
 
-- 
1.7.4.4

Re: [PATCH v3] xen-netfront: delay gARP until backend switches to Connected

[Ian Campbell]

On Wed, 2011-07-13 at 17:11 +0100, Laszlo Ersek wrote:
> On 07/13/11 15:29, Konrad Rzeszutek Wilk wrote:
> > On Wed, Jul 13, 2011 at 01:44:47PM +0200, Laszlo Ersek wrote:
> >> In addition to backporting 43223efd9bfd to the RHEL-5 host side, we needed
> >> the following in the RHEL-6 guest, in order to fix the network outage after
> >> live migration. I also tested a Fedora-15 guest (without the patch), and the
> >
> > Laszlo,
> >
> > This description is .. well, pointless for upstream patches. Just succinctly
> > describe the problem, how to reproduce it, and what this patch does.
> 
> I avoided writing up the commit message myself because in my first posting [1]
> I quoted two paragraphs from Ian Campbell [2]. I intended those two cited
> paragraphs verbatim as the commit message (along with my short remarks),
> because they describe the problem exactly.

Please don't expect/require that maintainers trawl around and construct
a commit message for you, always propose the full text you would like to
see committed in each patch posting.

>  Anyway:
> 
> After a guest is live migrated, the xen-netfront driver emits a gratuitous ARP
> message, so that networking hardware on the target host's subnet can take
> notice, and public routing to the guest is re-established. However, if the
> packet appears on the backend interface before the backend is added to the
> target host's bridge, the packet is lost, and the migrated guest's peers become
> unable to talk to the guest.
> 
> A sufficient two-parts condition to prevent the above is:
> 
> (1) ensure that the backend only moves to Connected xenbus state after its
> hotplug scripts completed, ie. the netback interface got added to the bridge;
> and
> 
> (2) ensure the frontend only queues the gARP when it sees the backend move to
> Connected.
> 
> These two together provide complete ordering. Sub-condition (1) is satisfied
> by pvops commit 43223efd9bfd.
> 
> In general, the full condition is sufficient, not necessary, because, according
> to [2], live migration has been working for a long time without satisfying
> sub-condition (2). However, after 43223efd9bfd was backported to the RHEL-5
> host to ensure (1), (2) still proved necessary in the RHEL-6 guest. This patch
> intends to provide (2) for upstream.

I expect that 43223efd9bfd just reduces the window (enough to prevent it
occuring most of the time) but doesn't actually close it. This change
seems good to me.
 
> [1] http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00327.html
> [2] http://lists.xensource.com/archives/html/xen-devel/2011-06/msg01969.html
> 
> Signed-off-by: Laszlo Ersek <lersek@redhat.com>

Reviewed-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  drivers/net/xen-netfront.c |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index d29365a..f033656 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -1646,7 +1646,6 @@ static void netback_changed(struct xenbus_device *dev,
>  	case XenbusStateInitialised:
>  	case XenbusStateReconfiguring:
>  	case XenbusStateReconfigured:
> -	case XenbusStateConnected:
>  	case XenbusStateUnknown:
>  	case XenbusStateClosed:
>  		break;
> @@ -1657,6 +1656,9 @@ static void netback_changed(struct xenbus_device *dev,
>  		if (xennet_connect(netdev) != 0)
>  			break;
>  		xenbus_switch_state(dev, XenbusStateConnected);
> +		break;
> +
> +	case XenbusStateConnected:
>  		netif_notify_peers(netdev);
>  		break;
>