[Qemu-devel] [PATCH v6 0/2] vhost user: Add live migration

[Qemu-devel] [PATCH v6 0/2] vhost user: Add live migration

[Thibaut Collet]

v5->v6
1. First patch: remove a warning log
2. Second patch: rename some functions to be more explicit on the purpose of
   these functions.

The first patch provides limited live migration:
- guest without GUEST_ANNOUNCE capabilities does not announce migration ending
and peers talking to the migrated guest can suffer important network outage.
- Some packets sent by remote peers to the guest can be lost during migration.

The second patch fixes limitation for guest without GUEST_ANNOUNCE capabilities
and patches from Marc Andre Lureau fix potential packet's lost during migration.

Thibaut Collet (2):
  vhost user: add support of live migration
  vhost user: add rarp sending after live migration for legacy guest

 docs/specs/vhost-user.txt         |   15 +++++++++++++++
 hw/net/vhost_net.c                |   18 ++++++++++++++++++
 hw/virtio/vhost-backend.c         |    3 ++-
 hw/virtio/vhost-user.c            |   32 ++++++++++++++++++++++++++++++--
 include/hw/virtio/vhost-backend.h |    2 ++
 include/net/vhost_net.h           |    1 +
 net/vhost-user.c                  |   31 +++++++++++++++++++++++++++++--
 7 files changed, 97 insertions(+), 5 deletions(-)

-- 
1.7.10.4

[Qemu-devel] [PATCH v6 1/2] vhost user: add support of live migration

[Thibaut Collet]

Some vhost user backends are able to support live migration.
To provide this service the following features must be added:
1. Add the VIRTIO_NET_F_GUEST_ANNOUNCE capability to vhost-net when netdev
   backend is vhost-user.
2. Provide a nop receive callback to vhost-user.
   This callback is called by:
    *  qemu_announce_self after a migration to send fake RARP to avoid network
       outage for peers talking to the migrated guest.
         - For guest with GUEST_ANNOUNCE capabilities, guest already sends GARP
           when the bit VIRTIO_NET_S_ANNOUNCE is set.
           => These packets must be discarded.
         - For guest without GUEST_ANNOUNCE capabilities, migration termination
           is notified when the guest sends packets.
           => These packets can be discarded.
    * virtio_net_tx_bh with a dummy boot to send fake bootp/dhcp request.
      BIOS guest manages virtio driver to send 4 bootp/dhcp request in case of
      dummy boot.
      => These packets must be discarded.

Signed-off-by: Thibaut Collet <thibaut.collet@6wind.com>
---
 hw/net/vhost_net.c |    2 ++
 net/vhost-user.c   |   12 ++++++++++--
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index c864237..9850520 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -85,6 +85,8 @@ static const int user_feature_bits[] = {
     VIRTIO_NET_F_CTRL_MAC_ADDR,
     VIRTIO_NET_F_CTRL_GUEST_OFFLOADS,
 
+    VIRTIO_NET_F_GUEST_ANNOUNCE,
+
     VIRTIO_NET_F_MQ,
 
     VHOST_INVALID_FEATURE_BIT
diff --git a/net/vhost-user.c b/net/vhost-user.c
index 93dcecd..1b1c626 100644
--- a/net/vhost-user.c
+++ b/net/vhost-user.c
@@ -65,6 +65,15 @@ static void vhost_user_stop(VhostUserState *s)
     s->vhost_net = 0;
 }
 
+static ssize_t vhost_user_receive(NetClientState *nc, const uint8_t *buf,
+                                  size_t size)
+{
+    /* Discard the request that is received and managed by backend
+     * by an other way.
+     */
+    return size;
+}
+
 static void vhost_user_cleanup(NetClientState *nc)
 {
     VhostUserState *s = DO_UPCAST(VhostUserState, nc, nc);
@@ -90,6 +99,7 @@ static bool vhost_user_has_ufo(NetClientState *nc)
 static NetClientInfo net_vhost_user_info = {
         .type = NET_CLIENT_OPTIONS_KIND_VHOST_USER,
         .size = sizeof(VhostUserState),
+        .receive = vhost_user_receive,
         .cleanup = vhost_user_cleanup,
         .has_vnet_hdr = vhost_user_has_vnet_hdr,
         .has_ufo = vhost_user_has_ufo,
@@ -143,8 +153,6 @@ static int net_vhost_user_init(NetClientState *peer, const char *device,
 
     s = DO_UPCAST(VhostUserState, nc, nc);
 
-    /* We don't provide a receive callback */
-    s->nc.receive_disabled = 1;
     s->chr = chr;
 
     qemu_chr_add_handlers(s->chr, NULL, NULL, net_vhost_user_event, s);
-- 
1.7.10.4

[Qemu-devel] [PATCH v6 2/2] vhost user: add rarp sending after live migration for legacy guest

[Thibaut Collet]

A new vhost user message is added to allow QEMU to ask to vhost user backend to
broadcast a fake RARP after live migration for guest without GUEST_ANNOUNCE
capability.

This new message is sent only if the backend supports the new
VHOST_USER_PROTOCOL_F_RARP protocol feature.
The payload of this new message is the MAC address of the guest (not known by
the backend). The MAC address is copied in the first 6 bytes of a u64 to avoid
to create a new payload message type.

This new message has no equivalent ioctl so a new callback is added in the
userOps structure to send the request.

Upon reception of this new message the vhost user backend must generate and
broadcast a fake RARP request to notify the migration is terminated.

Signed-off-by: Thibaut Collet <thibaut.collet@6wind.com>
---
 docs/specs/vhost-user.txt         |   15 +++++++++++++++
 hw/net/vhost_net.c                |   16 ++++++++++++++++
 hw/virtio/vhost-backend.c         |    3 ++-
 hw/virtio/vhost-user.c            |   32 ++++++++++++++++++++++++++++++--
 include/hw/virtio/vhost-backend.h |    2 ++
 include/net/vhost_net.h           |    1 +
 net/vhost-user.c                  |   23 +++++++++++++++++++++--
 7 files changed, 87 insertions(+), 5 deletions(-)

diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
index c2d2e2a..8c05301 100644
--- a/docs/specs/vhost-user.txt
+++ b/docs/specs/vhost-user.txt
@@ -140,6 +140,7 @@ Protocol features
 -----------------
 
 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 0
+#define VHOST_USER_PROTOCOL_F_RARP      1
 
 Message types
 -------------
@@ -308,6 +309,20 @@ Message types
       invalid FD flag. This flag is set when there is no file descriptor
       in the ancillary data.
 
+ * VHOST_USER_SEND_RARP
+
+      Id: 17
+      Equivalent ioctl: N/A
+      Master payload: u64
+
+      Ask vhost user backend to broadcast a fake RARP to notify the migration
+      is terminated for guest that does not support GUEST_ANNOUNCE.
+      Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
+      VHOST_USER_GET_FEATURES and protocol feature bit VHOST_USER_PROTOCOL_F_RARP
+      is present in VHOST_USER_GET_PROTOCOL_FEATURES.
+      The first 6 bytes of the payload contain the mac address of the guest to
+      allow the vhost user backend to construct and broadcast the fake RARP.
+
 Migration
 ---------
 
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index 9850520..cbcb6dd 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -383,6 +383,17 @@ void vhost_net_cleanup(struct vhost_net *net)
     g_free(net);
 }
 
+int vhost_net_notify_migration_done(struct vhost_net *net, char* mac_addr)
+{
+    const VhostOps *vhost_ops = net->dev.vhost_ops;
+    int r = -1;
+
+    if (vhost_ops->vhost_backend_migration_done)
+        r = vhost_ops->vhost_backend_migration_done(&net->dev, mac_addr);
+
+    return r;
+}
+
 bool vhost_net_virtqueue_pending(VHostNetState *net, int idx)
 {
     return vhost_virtqueue_pending(&net->dev, idx);
@@ -456,6 +467,11 @@ void vhost_net_virtqueue_mask(VHostNetState *net, VirtIODevice *dev,
 {
 }
 
+int vhost_net_notify_migration_done(struct vhost_net *net)
+{
+    return -1;
+}
+
 VHostNetState *get_vhost_net(NetClientState *nc)
 {
     return 0;
diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
index 4d68a27..09a5d67 100644
--- a/hw/virtio/vhost-backend.c
+++ b/hw/virtio/vhost-backend.c
@@ -46,7 +46,8 @@ static const VhostOps kernel_ops = {
         .backend_type = VHOST_BACKEND_TYPE_KERNEL,
         .vhost_call = vhost_kernel_call,
         .vhost_backend_init = vhost_kernel_init,
-        .vhost_backend_cleanup = vhost_kernel_cleanup
+        .vhost_backend_cleanup = vhost_kernel_cleanup,
+        .vhost_backend_migration_done = NULL
 };
 
 int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType backend_type)
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index fe75618..aa65cd5 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -10,6 +10,7 @@
 
 #include "hw/virtio/vhost.h"
 #include "hw/virtio/vhost-backend.h"
+#include "hw/virtio/virtio-net.h"
 #include "sysemu/char.h"
 #include "sysemu/kvm.h"
 #include "qemu/error-report.h"
@@ -27,8 +28,9 @@
 
 #define VHOST_USER_F_PROTOCOL_FEATURES 30
 
-#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x1ULL
+#define VHOST_USER_PROTOCOL_FEATURE_MASK 0x3ULL
 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 0
+#define VHOST_USER_PROTOCOL_F_RARP      1
 
 typedef enum VhostUserRequest {
     VHOST_USER_NONE = 0,
@@ -48,6 +50,7 @@ typedef enum VhostUserRequest {
     VHOST_USER_SET_VRING_ERR = 14,
     VHOST_USER_GET_PROTOCOL_FEATURES = 15,
     VHOST_USER_SET_PROTOCOL_FEATURES = 16,
+    VHOST_USER_SEND_RARP = 17,
     VHOST_USER_MAX
 } VhostUserRequest;
 
@@ -409,9 +412,34 @@ static int vhost_user_cleanup(struct vhost_dev *dev)
     return 0;
 }
 
+static int vhost_user_migration_done(struct vhost_dev *dev, char* mac_addr)
+{
+    VhostUserMsg msg = { 0 };
+    int err;
+
+    assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_USER);
+
+    /* If guest supports GUEST_ANNOUNCE do nothing */
+    if (__virtio_has_feature(dev->acked_features, VIRTIO_NET_F_GUEST_ANNOUNCE))
+        return 0;
+
+    /* if backend supports VHOST_USER_PROTOCOL_F_RARP ask it to send the RARP */
+    if (__virtio_has_feature(dev->protocol_features, VHOST_USER_PROTOCOL_F_RARP)) {
+        msg.request = VHOST_USER_SEND_RARP;
+        msg.flags = VHOST_USER_VERSION;
+        memcpy((char*) &msg.u64, mac_addr, 6);
+        msg.size = sizeof(m.u64);
+
+        err = vhost_user_write(dev, &msg, NULL, 0);
+        return err;
+    }
+    return -1;
+}
+
 const VhostOps user_ops = {
         .backend_type = VHOST_BACKEND_TYPE_USER,
         .vhost_call = vhost_user_call,
         .vhost_backend_init = vhost_user_init,
-        .vhost_backend_cleanup = vhost_user_cleanup
+        .vhost_backend_cleanup = vhost_user_cleanup,
+        .vhost_backend_migration_done = vhost_user_migration_done
         };
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index e472f29..5c8a440 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -24,12 +24,14 @@ typedef int (*vhost_call)(struct vhost_dev *dev, unsigned long int request,
              void *arg);
 typedef int (*vhost_backend_init)(struct vhost_dev *dev, void *opaque);
 typedef int (*vhost_backend_cleanup)(struct vhost_dev *dev);
+typedef int (*vhost_backend_migration_done)(struct vhost_dev *dev, char* mac_addr);
 
 typedef struct VhostOps {
     VhostBackendType backend_type;
     vhost_call vhost_call;
     vhost_backend_init vhost_backend_init;
     vhost_backend_cleanup vhost_backend_cleanup;
+    vhost_backend_migration_done vhost_backend_migration_done;
 } VhostOps;
 
 extern const VhostOps user_ops;
diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h
index 840d4b1..8e5dfd4 100644
--- a/include/net/vhost_net.h
+++ b/include/net/vhost_net.h
@@ -26,5 +26,6 @@ void vhost_net_ack_features(VHostNetState *net, uint64_t features);
 bool vhost_net_virtqueue_pending(VHostNetState *net, int n);
 void vhost_net_virtqueue_mask(VHostNetState *net, VirtIODevice *dev,
                               int idx, bool mask);
+int vhost_net_notify_migration_done(VHostNetState *net, char* mac_addr);
 VHostNetState *get_vhost_net(NetClientState *nc);
 #endif
diff --git a/net/vhost-user.c b/net/vhost-user.c
index 1b1c626..2489eeb 100644
--- a/net/vhost-user.c
+++ b/net/vhost-user.c
@@ -68,9 +68,28 @@ static void vhost_user_stop(VhostUserState *s)
 static ssize_t vhost_user_receive(NetClientState *nc, const uint8_t *buf,
                                   size_t size)
 {
-    /* Discard the request that is received and managed by backend
-     * by an other way.
+    /* In case of RARP (message size is 60) notify backup to send a fake RARP.
+       This fake RARP will be sent by backend only for guest
+       without GUEST_ANNOUNCE capability.
      */
+    if (size == 60) {
+        VhostUserState *s = DO_UPCAST(VhostUserState, nc, nc);
+        int r;
+        static int display_rarp_failure = 1;
+        char mac_addr[6];
+
+        /* extract guest mac address from the RARP message */
+        memcpy(mac_addr, &buf[6], 6);
+
+        r = vhost_net_notify_migration_done(s->vhost_net, mac_addr);
+
+        if ((r != 0) && (display_rarp_failure)) {
+            fprintf(stderr,
+                    "Vhost user backend fails to broadcast fake RARP\n");
+            fflush(stderr);
+            display_rarp_failure = 0;
+        }
+    }
     return size;
 }
 
-- 
1.7.10.4

Re: [Qemu-devel] [PATCH v6 0/2] vhost user: Add live migration

[Michael S. Tsirkin]

On Thu, Aug 06, 2015 at 10:45:07AM +0200, Thibaut Collet wrote:
> v5->v6
> 1. First patch: remove a warning log
> 2. Second patch: rename some functions to be more explicit on the purpose of
>    these functions.
> 
> The first patch provides limited live migration:
> - guest without GUEST_ANNOUNCE capabilities does not announce migration ending
> and peers talking to the migrated guest can suffer important network outage.
> - Some packets sent by remote peers to the guest can be lost during migration.
> 
> The second patch fixes limitation for guest without GUEST_ANNOUNCE capabilities
> and patches from Marc Andre Lureau fix potential packet's lost during migration.
> 
> Thibaut Collet (2):
>   vhost user: add support of live migration
>   vhost user: add rarp sending after live migration for legacy guest

I think these patches need to be rebased on top of Marc Andre's ones,
and use protocol flags to negotiate capabilities.
Right?


>  docs/specs/vhost-user.txt         |   15 +++++++++++++++
>  hw/net/vhost_net.c                |   18 ++++++++++++++++++
>  hw/virtio/vhost-backend.c         |    3 ++-
>  hw/virtio/vhost-user.c            |   32 ++++++++++++++++++++++++++++++--
>  include/hw/virtio/vhost-backend.h |    2 ++
>  include/net/vhost_net.h           |    1 +
>  net/vhost-user.c                  |   31 +++++++++++++++++++++++++++++--
>  7 files changed, 97 insertions(+), 5 deletions(-)
> 
> -- 
> 1.7.10.4

Re: [Qemu-devel] [PATCH v6 0/2] vhost user: Add live migration

[Marc-André Lureau]

Hi

On Wed, Aug 12, 2015 at 9:25 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> I think these patches need to be rebased on top of Marc Andre's ones,
> and use protocol flags to negotiate capabilities.
> Right?

Correct. His patches should be applied before my migration tests, though.

-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v6 0/2] vhost user: Add live migration

[Ouyang, Changchun]

> -----Original Message-----
> From: Marc-André Lureau [mailto:marcandre.lureau@gmail.com]
> Sent: Wednesday, August 12, 2015 6:07 PM
> To: Michael S. Tsirkin
> Cc: Thibaut Collet; QEMU; stefanha@redhat.com; Jason Wang; Paolo Bonzini;
> Linhaifeng; Ouyang, Changchun
> Subject: Re: [PATCH v6 0/2] vhost user: Add live migration
> 
> Hi
> 
> On Wed, Aug 12, 2015 at 9:25 AM, Michael S. Tsirkin <mst@redhat.com>
> wrote:
> > I think these patches need to be rebased on top of Marc Andre's ones,
> > and use protocol flags to negotiate capabilities.
> > Right?
> 
> Correct. His patches should be applied before my migration tests, though.
> 

Good point, do we have plan how to make these patches applied serially?
M.s.t's protocol feature patch,  marc andre's patch, Thibaut's patch for adding live migration, and my patch for vhost-user multi queue. 

> --
> Marc-André Lureau