From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53329) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WKu5y-0005i3-7m for qemu-devel@nongnu.org; Tue, 04 Mar 2014 13:29:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WKu5i-0006Cb-OQ for qemu-devel@nongnu.org; Tue, 04 Mar 2014 13:29:38 -0500 Received: from mail-qa0-x235.google.com ([2607:f8b0:400d:c00::235]:41004) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WKu5i-0006CW-HD for qemu-devel@nongnu.org; Tue, 04 Mar 2014 13:29:22 -0500 Received: by mail-qa0-f53.google.com with SMTP id w8so3719879qac.26 for ; Tue, 04 Mar 2014 10:29:22 -0800 (PST) Sender: Paolo Bonzini Message-ID: <53161B7C.9010903@redhat.com> Date: Tue, 04 Mar 2014 19:29:16 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <1393957383-16685-1-git-send-email-a.motakis@virtualopensystems.com> In-Reply-To: <1393957383-16685-1-git-send-email-a.motakis@virtualopensystems.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v9 00/20] Vhost and vhost-net support for userspace based backends List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: snabb-devel@googlegroups.com, qemu-devel@nongnu.org Cc: lukego@gmail.com, Antonios Motakis , tech@virtualopensystems.com, n.nikolaev@virtualopensystems.com, mst@redhat.com Il 04/03/2014 19:22, Antonios Motakis ha scritto: > In this patch series we would like to introduce our approach for putting a > virtio-net backend in an external userspace process. Our eventual target is to > run the network backend in the Snabbswitch ethernet switch, while receiving > traffic from a guest inside QEMU/KVM which runs an unmodified virtio-net > implementation. > > For this, we are working into extending vhost to allow equivalent functionality > for userspace. Vhost already passes control of the data plane of virtio-net to > the host kernel; we want to realize a similar model, but for userspace. > > In this patch series the concept of a vhost-backend is introduced. > > We define two vhost backend types - vhost-kernel and vhost-user. The former is > the interface to the current kernel module implementation. Its control plane is > ioctl based. The data plane is realized by the kernel directly accessing the > QEMU allocated, guest memory. > > In the new vhost-user backend, the control plane is based on communication > between QEMU and another userspace process using a unix domain socket. This > allows to implement a virtio backend for a guest running in QEMU, inside the > other userspace process. For this communication we use a chardev with a Unix > domain socket backend. Vhost-user is client/server agnostic regarding the > chardev, however it does not support the 'nowait' and 'telnet' options. > > A reconnection in 'server' mode is supported, but the backend's exposed virtio > features need to be compatible with the first connected slave. > > We change -mem-path to QemuOpts and add prealloc and share as properties > to it. HugeTLBFS is required for this option to work. > > The data path is realized by directly accessing the vrings and the buffer data > off the guest's memory. > > The current user of vhost-user is only vhost-net. We add new netdev backend > that is intended to initialize vhost-net with vhost-user backend. > > Example usage: > > qemu -m 1024 -mem-path /hugetlbfs,share=on \ > -chardev socket,id=chr0,path=/path/to/ > socket \ > -netdev type=vhost-user,id=net0,chardev=chr0 \ > -device virtio-net-pci,netdev=net0 > > On non-MSIX guests the vhost feture can be forced using a special option: > > ... > -netdev type=vhost-user,id=net0,chardev=chr0,vhostforce > ... > > In order to use ioeventfds, kvm should be enabled. > > This code can be pulled from git@github.com:virtualopensystems/qemu.git vhost-user-v9 > A simple functional test is available in tests/vhost-user-test.c > > A reference vhost-user slave for testing is also available from git@github.com:virtualopensystems/vapp.git Hi, did you see the series I posted today? It would be great if you tested vhost-user on top of it. The replacement for the above -mem-path incantation should be the following: -object memory-file,id=mem,path=/hugetlbfs,share=on \ -numa node,memdev=mem Do the ideas behind vhost-user work if you have multiple hugetlbfs files, one per node? Paolo > Changes from v8: > - Removed prealloc property from the -mem-path refactoring > - Added and use new function - kvm_eventfds_enabled > - Add virtio_queue_get_avail_idx used in vhost_virtqueue_stop to > get a sane value in case of VHOST_GET_VRING_BASE failure > - vhost user uses kvm_eventfds_enabled to check whether the ioeventfd > capability of KVM is available > - Added flag VHOST_USER_VRING_NOFD_MASK to be set when KICK, CALL or ERR file > descriptor is invalid or ioeventfd is not available > > Changes from v7: > - Slave reconnection when using chardev in server mode > - qtest vhost-user-test added > - New qemu_chr_fe_get_msgfds for reading multiple fds from the chardev > - Mandatory features in vhost_dev, used on reconnect to verify for conflicts > - Add vhostforce parameter to -netdev vhost-user (for non-MSIX guests) > - Extend libqemustub.a to support qemu-char.c > > Changes from v6: > - Remove the 'unlink' property of '-mem-path' > - Extend qemu-char: blocking read, send fds, monitor for connection close > - Vhost-user uses chardev as a backend > - Poll and reconnect removed (no VHOST_USER_ECHO). > - Disconnect is deteced by the chardev (G_IO_HUP event) > - vhost-backend.c split to vhost-user.c > > Changes from v5: > - Split -mem-path unlink option to a separate patch > - Fds are passed only in the ancillary data > - Stricter message size checks on receive/send > - Netdev vhost-user now includes path and poll_time options > - The connection probing interval is configurable > > Changes from v4: > - Use error_report for errors > - VhostUserMsg has new field `size` indicating the following payload length. > Field `flags` now has version and reply bits. The structure is packed. > - Send data is of variable length (`size` field in message) > - Receive in 2 steps, header and payload > - Add new message type VHOST_USER_ECHO, to check connection status > > Changes from v3: > - Convert -mem-path to QemuOpts with prealloc, share and unlink properties > - Set 1 sec timeout when read/write to the unix domain socket > - Fix file descriptor leak > > Changes from v2: > - Reconnect when the backend disappears > > Changes from v1: > - Implementation of vhost-user netdev backend > - Code improvements > > Antonios Motakis (20): > Convert -mem-path to QemuOpts and add share property > Add kvm_eventfds_enabled function > Add chardev API qemu_chr_fe_read_all > Add chardev API qemu_chr_fe_set_msgfds > Add chardev API qemu_chr_fe_get_msgfds > Add G_IO_HUP handler for socket chardev > vhost_net should call the poll callback only when it is set > Refactor virtio-net to use generic get_vhost_net > Add new virtio API virtio_queue_get_avail_idx > Gracefully handle ioctl failure in vhost_virtqueue_stop > vhost_net_init will use VhostNetOptions to get all its arguments > Add vhost_ops to vhost_dev struct and replace all relevant ioctls > Add mandatory_features to vhost_dev > Add vhost-backend and VhostBackendType > Add vhost-user as a vhost backend. > Add new vhost-user netdev backend > Add the vhost-user netdev backend to the command line > Add vhost-user protocol documentation > libqemustub: add stubs to be able to use qemu-char.c > Add qtest for vhost-user > > docs/specs/vhost-user.txt | 261 ++++++++++++++++++++++++++++ > exec.c | 21 ++- > hmp-commands.hx | 4 +- > hw/net/vhost_net.c | 141 ++++++++++----- > hw/net/virtio-net.c | 29 +--- > hw/scsi/vhost-scsi.c | 20 ++- > hw/virtio/Makefile.objs | 2 +- > hw/virtio/vhost-backend.c | 71 ++++++++ > hw/virtio/vhost-user.c | 356 ++++++++++++++++++++++++++++++++++++++ > hw/virtio/vhost.c | 58 +++---- > hw/virtio/virtio.c | 5 + > include/exec/cpu-all.h | 1 - > include/hw/virtio/vhost-backend.h | 38 ++++ > include/hw/virtio/vhost.h | 9 +- > include/hw/virtio/virtio.h | 1 + > include/net/vhost-user.h | 17 ++ > include/net/vhost_net.h | 13 +- > include/sysemu/char.h | 43 ++++- > include/sysemu/kvm.h | 11 ++ > kvm-all.c | 4 + > kvm-stub.c | 1 + > net/Makefile.objs | 2 +- > net/clients.h | 3 + > net/hub.c | 1 + > net/net.c | 25 +-- > net/tap.c | 18 +- > net/vhost-user.c | 273 +++++++++++++++++++++++++++++ > qapi-schema.json | 19 +- > qemu-char.c | 272 +++++++++++++++++++++++++---- > qemu-options.hx | 25 ++- > stubs/Makefile.objs | 8 + > stubs/bdrv-commit-all.c | 7 + > stubs/chr-msmouse.c | 7 + > stubs/get-next-serial.c | 3 + > stubs/is-daemonized.c | 7 + > stubs/machine-init-done.c | 6 + > stubs/monitor-init.c | 6 + > stubs/notify-event.c | 6 + > stubs/vc-init.c | 7 + > tests/Makefile | 4 + > tests/vhost-user-test.c | 309 +++++++++++++++++++++++++++++++++ > vl.c | 23 ++- > 42 files changed, 1979 insertions(+), 158 deletions(-) > create mode 100644 docs/specs/vhost-user.txt > create mode 100644 hw/virtio/vhost-backend.c > create mode 100644 hw/virtio/vhost-user.c > create mode 100644 include/hw/virtio/vhost-backend.h > create mode 100644 include/net/vhost-user.h > create mode 100644 net/vhost-user.c > create mode 100644 stubs/bdrv-commit-all.c > create mode 100644 stubs/chr-msmouse.c > create mode 100644 stubs/get-next-serial.c > create mode 100644 stubs/is-daemonized.c > create mode 100644 stubs/machine-init-done.c > create mode 100644 stubs/monitor-init.c > create mode 100644 stubs/notify-event.c > create mode 100644 stubs/vc-init.c > create mode 100644 tests/vhost-user-test.c >