From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50966) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vlzcg-000876-4B for qemu-devel@nongnu.org; Thu, 28 Nov 2013 06:19:13 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VlzcY-00052f-RA for qemu-devel@nongnu.org; Thu, 28 Nov 2013 06:19:06 -0500 Received: from mail-bk0-f49.google.com ([209.85.214.49]:39699) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VlzcY-00052X-I2 for qemu-devel@nongnu.org; Thu, 28 Nov 2013 06:18:58 -0500 Received: by mail-bk0-f49.google.com with SMTP id my13so3718575bkb.36 for ; Thu, 28 Nov 2013 03:18:57 -0800 (PST) Message-ID: <5297269E.1050500@cloudius-systems.com> Date: Thu, 28 Nov 2013 13:18:54 +0200 From: Avi Kivity MIME-Version: 1.0 References: <5294B461.5000405@redhat.com> <5294B634.4050801@cloudius-systems.com> <20131126150357.GA20352@redhat.com> <5294BC3B.6070902@redhat.com> <5297050E.6000700@redhat.com> <20131128091903.GA4609@kernel.org> <5297118C.3050104@cloudius-systems.com> <20131128101138.GB5822@minantech.com> <52971727.2030600@cloudius-systems.com> <20131128110250.GC5822@minantech.com> In-Reply-To: <20131128110250.GC5822@minantech.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] create a single workqueue for each vm to update vm irq routing table List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gleb Natapov Cc: "Huangweidong (C)" , KVM , Gleb Natapov , "Michael S. Tsirkin" , "Zhanghaoyu (A)" , Luonengjun , "qemu-devel@nongnu.org" , Zanghongyong , Avi Kivity , Paolo Bonzini , "Jinxin (F)" On 11/28/2013 01:02 PM, Gleb Natapov wrote: > On Thu, Nov 28, 2013 at 12:12:55PM +0200, Avi Kivity wrote: >> On 11/28/2013 12:11 PM, Gleb Natapov wrote: >>> On Thu, Nov 28, 2013 at 11:49:00AM +0200, Avi Kivity wrote: >>>> On 11/28/2013 11:19 AM, Gleb Natapov wrote: >>>>> On Thu, Nov 28, 2013 at 09:55:42AM +0100, Paolo Bonzini wrote: >>>>>> Il 28/11/2013 07:27, Zhanghaoyu (A) ha scritto: >>>>>>>>> Without synchronize_rcu you could have >>>>>>>>> >>>>>>>>> VCPU writes to routing table >>>>>>>>> e = entry from IRQ routing table >>>>>>>>> kvm_irq_routing_update(kvm, new); >>>>>>>>> VCPU resumes execution >>>>>>>>> kvm_set_msi_irq(e, &irq); >>>>>>>>> kvm_irq_delivery_to_apic_fast(); >>>>>>>>> >>>>>>>>> where the entry is stale but the VCPU has already resumed execution. >>>>>>>>> >>>>>>> If we use call_rcu()(Not consider the problem that Gleb pointed out temporarily) instead of synchronize_rcu(), should we still ensure this? >>>>>> The problem is that we should ensure this, so using call_rcu is not >>>>>> possible (even not considering the memory allocation problem). >>>>>> >>>>> Not changing current behaviour is certainly safer, but I am still not 100% >>>>> convinced we have to ensure this. >>>>> >>>>> Suppose guest does: >>>>> >>>>> 1: change msi interrupt by writing to pci register >>>>> 2: read the pci register to flush the write >>>>> 3: zero idt >>>>> >>>>> I am pretty certain that this code can get interrupt after step 2 on real HW, >>>>> but I cannot tell if guest can rely on it to be delivered exactly after >>>>> read instruction or it can be delayed by couple of instructions. Seems to me >>>>> it would be fragile for an OS to depend on this behaviour. AFAIK Linux does not. >>>>> >>>> Linux is safe, it does interrupt migration from within the interrupt >>>> handler. If you do that before the device-specific EOI, you won't >>>> get another interrupt until programming the MSI is complete. >>>> >>>> Is virtio safe? IIRC it can post multiple interrupts without guest acks. >>>> >>>> Using call_rcu() is a better solution than srcu IMO. Less code >>>> changes, consistently faster. >>> Why not fix userspace to use KVM_SIGNAL_MSI instead? >>> >>> >> Shouldn't it work with old userspace too? Maybe I misunderstood your intent. > Zhanghaoyu said that the problem mostly hurts in real-time telecom > environment, so I propose how he can fix the problem in his specific > environment. It will not fix older userspace obviously, but kernel > fix will also require kernel update and usually updating userspace > is easier. > > Isn't the latency due to interrupt migration causing long synchronize_rcu()s? How does KVM_SIGNAL_MSI help? The problem occurs with assigned devices too AFAICT.