From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=58870 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PhoQ0-0003G8-5f for qemu-devel@nongnu.org; Tue, 25 Jan 2011 14:19:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhoPy-0004oH-Ex for qemu-devel@nongnu.org; Tue, 25 Jan 2011 14:19:07 -0500 Received: from e1.ny.us.ibm.com ([32.97.182.141]:50074) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhoPy-0004nK-8O for qemu-devel@nongnu.org; Tue, 25 Jan 2011 14:19:06 -0500 Received: from d01dlp02.pok.ibm.com (d01dlp02.pok.ibm.com [9.56.224.85]) by e1.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p0PJ9a4k007985 for ; Tue, 25 Jan 2011 14:09:57 -0500 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 28B444DE8040 for ; Tue, 25 Jan 2011 14:15:30 -0500 (EST) Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p0PJIxtP188616 for ; Tue, 25 Jan 2011 14:18:59 -0500 Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p0PJNUJO014828 for ; Tue, 25 Jan 2011 12:23:31 -0700 Message-ID: <4D3F2222.8010405@linux.vnet.ibm.com> Date: Tue, 25 Jan 2011 13:18:58 -0600 From: Anthony Liguori MIME-Version: 1.0 Subject: Re: [Qemu-devel] [PATCH v5 2/4] virtio-pci: Use ioeventfd for virtqueue notify References: <1292166128-10874-1-git-send-email-stefanha@linux.vnet.ibm.com> <1292166128-10874-3-git-send-email-stefanha@linux.vnet.ibm.com> <4D3DCADC.6010308@redhat.com> <20110124193653.GC29941@redhat.com> <4D3DD775.3060003@redhat.com> <20110124194729.GE29941@redhat.com> <4D3DDB99.8090001@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Kevin Wolf , Anthony Liguori , Stefan Hajnoczi , "Michael S. Tsirkin" , qemu-devel@nongnu.org, Avi Kivity On 01/25/2011 03:49 AM, Stefan Hajnoczi wrote: > On Tue, Jan 25, 2011 at 7:12 AM, Stefan Hajnoczi wrote: > >> On Mon, Jan 24, 2011 at 8:05 PM, Kevin Wolf wrote: >> >>> Am 24.01.2011 20:47, schrieb Michael S. Tsirkin: >>> >>>> On Mon, Jan 24, 2011 at 08:48:05PM +0100, Kevin Wolf wrote: >>>> >>>>> Am 24.01.2011 20:36, schrieb Michael S. Tsirkin: >>>>> >>>>>> On Mon, Jan 24, 2011 at 07:54:20PM +0100, Kevin Wolf wrote: >>>>>> >>>>>>> Am 12.12.2010 16:02, schrieb Stefan Hajnoczi: >>>>>>> >>>>>>>> Virtqueue notify is currently handled synchronously in userspace virtio. This >>>>>>>> prevents the vcpu from executing guest code while hardware emulation code >>>>>>>> handles the notify. >>>>>>>> >>>>>>>> On systems that support KVM, the ioeventfd mechanism can be used to make >>>>>>>> virtqueue notify a lightweight exit by deferring hardware emulation to the >>>>>>>> iothread and allowing the VM to continue execution. This model is similar to >>>>>>>> how vhost receives virtqueue notifies. >>>>>>>> >>>>>>>> The result of this change is improved performance for userspace virtio devices. >>>>>>>> Virtio-blk throughput increases especially for multithreaded scenarios and >>>>>>>> virtio-net transmit throughput increases substantially. >>>>>>>> >>>>>>>> Some virtio devices are known to have guest drivers which expect a notify to be >>>>>>>> processed synchronously and spin waiting for completion. Only enable ioeventfd >>>>>>>> for virtio-blk and virtio-net for now. >>>>>>>> >>>>>>>> Care must be taken not to interfere with vhost-net, which uses host >>>>>>>> notifiers. If the set_host_notifier() API is used by a device >>>>>>>> virtio-pci will disable virtio-ioeventfd and let the device deal with >>>>>>>> host notifiers as it wishes. >>>>>>>> >>>>>>>> After migration and on VM change state (running/paused) virtio-ioeventfd >>>>>>>> will enable/disable itself. >>>>>>>> >>>>>>>> * VIRTIO_CONFIG_S_DRIVER_OK -> enable virtio-ioeventfd >>>>>>>> * !VIRTIO_CONFIG_S_DRIVER_OK -> disable virtio-ioeventfd >>>>>>>> * virtio_pci_set_host_notifier() -> disable virtio-ioeventfd >>>>>>>> * vm_change_state(running=0) -> disable virtio-ioeventfd >>>>>>>> * vm_change_state(running=1) -> enable virtio-ioeventfd >>>>>>>> >>>>>>>> Signed-off-by: Stefan Hajnoczi >>>>>>>> >>>>>>> On current git master I'm getting hangs when running iozone on a >>>>>>> virtio-blk disk. "Hang" means that it's not responsive any more and has >>>>>>> 100% CPU consumption. >>>>>>> >>>>>>> I bisected the problem to this patch. Any ideas? >>>>>>> >>>>>>> Kevin >>>>>>> >>>>>> Does it help if you set ioeventfd=off on command line? >>>>>> >>>>> Yes, with ioeventfd=off it seems to work fine. >>>>> >>>>> Kevin >>>>> >>>> Then it's the ioeventfd that is to blame. >>>> Is it the io thread that consumes 100% CPU? >>>> Or the vcpu thread? >>>> >>> I was building with the default options, i.e. there is no IO thread. >>> >>> Now I'm just running the test with IO threads enabled, and so far >>> everything looks good. So I can only reproduce the problem with IO >>> threads disabled. >>> >> Hrm...aio uses SIGUSR2 to force the vcpu to process aio completions >> (relevant when --enable-io-thread is not used). I will take a look at >> that again and see why we're spinning without checking for ioeventfd >> completion. >> > Here's my understanding of --disable-io-thread. Added Anthony on CC, > please correct me. > > When I/O thread is disabled our only thread runs guest code until an > exit request is made. There are synchronous exit cases like a halt > instruction or single step. There are also asynchronous exit cases > when signal handlers use qemu_notify_event(), which does cpu_exit(), > to set env->exit_request = 1 and unlink the current tb. > Correct. Note that this is a problem today. If you have a tight loop in TCG and you have nothing that would generate a signal (no pending disk I/O and no periodic timer) then the main loop is starved. This is a fundamental flaw in TCG. Regards, Anthony Liguori