From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=37155 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Phov2-0006Fi-Ts for qemu-devel@nongnu.org; Tue, 25 Jan 2011 14:51:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Phouz-0005d4-WE for qemu-devel@nongnu.org; Tue, 25 Jan 2011 14:51:12 -0500 Received: from e5.ny.us.ibm.com ([32.97.182.145]:52211) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Phouz-0005ci-T7 for qemu-devel@nongnu.org; Tue, 25 Jan 2011 14:51:09 -0500 Received: from d01dlp02.pok.ibm.com (d01dlp02.pok.ibm.com [9.56.224.85]) by e5.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p0PJRQQY018791 for ; Tue, 25 Jan 2011 14:27:31 -0500 Received: from d01relay01.pok.ibm.com (d01relay01.pok.ibm.com [9.56.227.233]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 315E04DE8040 for ; Tue, 25 Jan 2011 14:47:38 -0500 (EST) Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by d01relay01.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p0PJp7QF369010 for ; Tue, 25 Jan 2011 14:51:07 -0500 Received: from d01av01.pok.ibm.com (loopback [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p0PJp6Ax004183 for ; Tue, 25 Jan 2011 14:51:07 -0500 Message-ID: <4D3F29A8.7040205@linux.vnet.ibm.com> Date: Tue, 25 Jan 2011 13:51:04 -0600 From: Anthony Liguori MIME-Version: 1.0 Subject: Re: [Qemu-devel] [PATCH v5 2/4] virtio-pci: Use ioeventfd for virtqueue notify References: <1292166128-10874-1-git-send-email-stefanha@linux.vnet.ibm.com> <1292166128-10874-3-git-send-email-stefanha@linux.vnet.ibm.com> <4D3DCADC.6010308@redhat.com> <20110124193653.GC29941@redhat.com> <4D3DD775.3060003@redhat.com> <20110124194729.GE29941@redhat.com> <4D3DDB99.8090001@redhat.com> <4D3F2222.8010405@linux.vnet.ibm.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Kevin Wolf , Anthony Liguori , Stefan Hajnoczi , "Michael S. Tsirkin" , qemu-devel@nongnu.org, Avi Kivity On 01/25/2011 01:45 PM, Stefan Hajnoczi wrote: > On Tue, Jan 25, 2011 at 7:18 PM, Anthony Liguori > wrote: > >> On 01/25/2011 03:49 AM, Stefan Hajnoczi wrote: >> >>> On Tue, Jan 25, 2011 at 7:12 AM, Stefan Hajnoczi >>> wrote: >>> >>> >>>> On Mon, Jan 24, 2011 at 8:05 PM, Kevin Wolf wrote: >>>> >>>> >>>>> Am 24.01.2011 20:47, schrieb Michael S. Tsirkin: >>>>> >>>>> >>>>>> On Mon, Jan 24, 2011 at 08:48:05PM +0100, Kevin Wolf wrote: >>>>>> >>>>>> >>>>>>> Am 24.01.2011 20:36, schrieb Michael S. Tsirkin: >>>>>>> >>>>>>> >>>>>>>> On Mon, Jan 24, 2011 at 07:54:20PM +0100, Kevin Wolf wrote: >>>>>>>> >>>>>>>> >>>>>>>>> Am 12.12.2010 16:02, schrieb Stefan Hajnoczi: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Virtqueue notify is currently handled synchronously in userspace >>>>>>>>>> virtio. This >>>>>>>>>> prevents the vcpu from executing guest code while hardware >>>>>>>>>> emulation code >>>>>>>>>> handles the notify. >>>>>>>>>> >>>>>>>>>> On systems that support KVM, the ioeventfd mechanism can be used to >>>>>>>>>> make >>>>>>>>>> virtqueue notify a lightweight exit by deferring hardware emulation >>>>>>>>>> to the >>>>>>>>>> iothread and allowing the VM to continue execution. This model is >>>>>>>>>> similar to >>>>>>>>>> how vhost receives virtqueue notifies. >>>>>>>>>> >>>>>>>>>> The result of this change is improved performance for userspace >>>>>>>>>> virtio devices. >>>>>>>>>> Virtio-blk throughput increases especially for multithreaded >>>>>>>>>> scenarios and >>>>>>>>>> virtio-net transmit throughput increases substantially. >>>>>>>>>> >>>>>>>>>> Some virtio devices are known to have guest drivers which expect a >>>>>>>>>> notify to be >>>>>>>>>> processed synchronously and spin waiting for completion. Only >>>>>>>>>> enable ioeventfd >>>>>>>>>> for virtio-blk and virtio-net for now. >>>>>>>>>> >>>>>>>>>> Care must be taken not to interfere with vhost-net, which uses host >>>>>>>>>> notifiers. If the set_host_notifier() API is used by a device >>>>>>>>>> virtio-pci will disable virtio-ioeventfd and let the device deal >>>>>>>>>> with >>>>>>>>>> host notifiers as it wishes. >>>>>>>>>> >>>>>>>>>> After migration and on VM change state (running/paused) >>>>>>>>>> virtio-ioeventfd >>>>>>>>>> will enable/disable itself. >>>>>>>>>> >>>>>>>>>> * VIRTIO_CONFIG_S_DRIVER_OK -> enable virtio-ioeventfd >>>>>>>>>> * !VIRTIO_CONFIG_S_DRIVER_OK -> disable virtio-ioeventfd >>>>>>>>>> * virtio_pci_set_host_notifier() -> disable virtio-ioeventfd >>>>>>>>>> * vm_change_state(running=0) -> disable virtio-ioeventfd >>>>>>>>>> * vm_change_state(running=1) -> enable virtio-ioeventfd >>>>>>>>>> >>>>>>>>>> Signed-off-by: Stefan Hajnoczi >>>>>>>>>> >>>>>>>>>> >>>>>>>>> On current git master I'm getting hangs when running iozone on a >>>>>>>>> virtio-blk disk. "Hang" means that it's not responsive any more and >>>>>>>>> has >>>>>>>>> 100% CPU consumption. >>>>>>>>> >>>>>>>>> I bisected the problem to this patch. Any ideas? >>>>>>>>> >>>>>>>>> Kevin >>>>>>>>> >>>>>>>>> >>>>>>>> Does it help if you set ioeventfd=off on command line? >>>>>>>> >>>>>>>> >>>>>>> Yes, with ioeventfd=off it seems to work fine. >>>>>>> >>>>>>> Kevin >>>>>>> >>>>>>> >>>>>> Then it's the ioeventfd that is to blame. >>>>>> Is it the io thread that consumes 100% CPU? >>>>>> Or the vcpu thread? >>>>>> >>>>>> >>>>> I was building with the default options, i.e. there is no IO thread. >>>>> >>>>> Now I'm just running the test with IO threads enabled, and so far >>>>> everything looks good. So I can only reproduce the problem with IO >>>>> threads disabled. >>>>> >>>>> >>>> Hrm...aio uses SIGUSR2 to force the vcpu to process aio completions >>>> (relevant when --enable-io-thread is not used). I will take a look at >>>> that again and see why we're spinning without checking for ioeventfd >>>> completion. >>>> >>>> >>> Here's my understanding of --disable-io-thread. Added Anthony on CC, >>> please correct me. >>> >>> When I/O thread is disabled our only thread runs guest code until an >>> exit request is made. There are synchronous exit cases like a halt >>> instruction or single step. There are also asynchronous exit cases >>> when signal handlers use qemu_notify_event(), which does cpu_exit(), >>> to set env->exit_request = 1 and unlink the current tb. >>> >>> >> Correct. >> >> Note that this is a problem today. If you have a tight loop in TCG and you >> have nothing that would generate a signal (no pending disk I/O and no >> periodic timer) then the main loop is starved. >> > Even with KVM we can spin inside the guest and get a softlockup due to > the dynticks race condition shown above. In a CPU bound guest that's > doing no I/O it's possible to go AWOL for extended periods of time. > This is a different race. I need to look more deeply into the code. > I can think of two solutions: > 1. Block SIGALRM during critical regions, not sure if the necessary > atomic signal mask capabilities are there in KVM. Haven't looked at > TCG yet either. > 2. Make a portion of the timer code signal-safe and rearm the timer > from within the SIGLARM handler. > Or, switch to timerfd and stop using a signal based alarm timer. Regards, Anthony Liguori > Stefan >