From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46057) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aNzt9-00025S-NA for qemu-devel@nongnu.org; Tue, 26 Jan 2016 04:26:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aNzt6-000797-1r for qemu-devel@nongnu.org; Tue, 26 Jan 2016 04:26:15 -0500 Received: from mail-wm0-x22d.google.com ([2a00:1450:400c:c09::22d]:34060) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aNzt5-00078y-NL for qemu-devel@nongnu.org; Tue, 26 Jan 2016 04:26:11 -0500 Received: by mail-wm0-x22d.google.com with SMTP id u188so97885127wmu.1 for ; Tue, 26 Jan 2016 01:26:11 -0800 (PST) Message-ID: <56A73BA2.9070601@6wind.com> Date: Tue, 26 Jan 2016 10:25:54 +0100 From: Didier Pallard MIME-Version: 1.0 References: <1449136399-4158-1-git-send-email-didier.pallard@6wind.com> <20160125112158-mutt-send-email-victork@redhat.com> In-Reply-To: <20160125112158-mutt-send-email-victork@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Linux vhost-user interrupt management fixes List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Victor Kaplansky Cc: thibaut.collet@6wind.com, jmg@6wind.com, qemu-devel@nongnu.org On 01/25/2016 10:22 AM, Victor Kaplansky wrote: > On Thu, Dec 03, 2015 at 10:53:16AM +0100, Didier Pallard wrote: >> Hi, >> >> I recently did some stress tests of a vhost-user interface using an UDP >> traffic generator. Traffic generator was connected to 2 physical ports >> that are in turn connected to 2 virtio ports through a linux bridge, VM >> (running linux) doing routing to forward packets between the 2 virtio ports. >> When traffic reaches high pps rates of small packets, I faced the 2 following >> problems: >> >> - at some time, my qemu socket becomes full, causing qemu to send incomplete >> SET_VRING_CALL messages to vhost-user backend (without proper fd set in >> ancillary data). >> - after some time, some interrupts are lost, causing the VM to stop >> transmitting packets. >> >> Both problems come from the fact that interrupt masking/unmasking of the VM >> is deferred to vhost-user backend through the linux socket. >> First problem comes from the fact that socket buffer gets full; it is corrected >> in the first patch of the serie. >> Second problem is a bit more complex. From what i understand of the code, >> when VM wants to mask/unmask interrupts, qemu traps the command and sends a >> SET_VRING_CALL to the vhost-user to swap interrupt notifications either to >> a dummy descriptor or to an fd that was given to kvm module to route >> interruption to the VM. After sending SET_VRING_CALL message through >> the socket, VM code continues to run, assuming that the interrupts are now >> masked; but due to linux socket, this message may be buffered and not currently >> treated by the vhost-user backend, ie interrupts are not really masked/unmasked >> as they ought to be. >> I think it can be solved in two different ways: >> - by waiting for an acknowledgement of vhost-user backend before exiting >> interrupt masking function, this ensures that interrupt masking is correctly >> taken into account before giving back hand to the VM, but it has a very high >> cost in term of cycles. Moreover, unless specifying a new option, it will >> break current API, since existing vhost-user implementations do not >> expect a return message on a SET_VRING_CALL call. >> - second way could be, in case vhost-user is involved, to restore the >> initial behaviour of interrupt masking (ie masking/unmasking interrupts >> by taking/giving vring_call fd from/to kernel kvm module). >> Patches 2 and 3 of the serie propose an implementation of this second option. >> >> Didier >> > Tested-by: Victor Kaplansky > >> Didier Pallard (3): >> char: fix vhost-user socket full >> virtio-pci: add an option to bypass guest_notifier_mask >> vhost-net: force guest_notifier_mask bypass in vhost-user case >> >> hw/net/vhost_net.c | 19 ++++++++++++++++++- >> hw/virtio/vhost.c | 13 +++++++++++++ >> hw/virtio/virtio-pci.c | 29 +++++++++++++++++++++++------ >> hw/virtio/virtio-pci.h | 6 ++++++ >> qemu-char.c | 10 ++++++++++ >> 5 files changed, 70 insertions(+), 7 deletions(-) >> >> -- >> 2.1.4 >> >> thanks for the tests, victor didier