From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: [RFC][PATCH 28/45] qemu-kvm: msix: Drop tracking of used vectors Date: Tue, 18 Oct 2011 16:08:46 +0200 Message-ID: <4E9D886E.3090201@siemens.com> References: <20111017154852.GC7876@redhat.com> <4E9C81CC.5040900@web.de> <20111018115813.GF28776@redhat.com> <4E9D6C5B.8050204@siemens.com> <20111018123305.GK28776@redhat.com> <4E9D734C.2060504@siemens.com> <20111018124818.GO28776@redhat.com> <4E9D786D.4060802@siemens.com> <20111018133719.GS28776@redhat.com> <4E9D831E.100@siemens.com> <20111018140156.GA4980@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Alex Williamson , Marcelo Tosatti , Avi Kivity , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20111018140156.GA4980@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org List-Id: kvm.vger.kernel.org On 2011-10-18 16:01, Michael S. Tsirkin wrote: >>>>>>> >>>>>>> I actually would not mind preallocating everything upfront which is much >>>>>>> easier. But with your patch we get a silent failure or a drastic >>>>>>> slowdown which is much more painful IMO. >>>>>> >>>>>> Again: did we already saw that limit? And where does it come from if not >>>>>> from KVM? >>>>> >>>>> It's a hardware limitation of intel APICs. interrupt vector is encoded >>>>> in an 8 bit field in msi address. So you can have at most 256 of these. >>>> >>>> There should be no such limitation with pseudo GSIs we use for MSI >>>> injection. They end up as MSI messages again, so actually 256 (-reserved >>>> vectors) * number-of-cpus (on x86). >>> >>> This limits which CPUs can get the interrupt though. >>> Linux seems to have a global pool as it wants to be able to freely >>> balance vectors between CPUs. Or, consider a guest with a single CPU :) >>> >>> Anyway, why argue - there is a limitation, and it's not coming from KVM, >>> right? >> >> No, our limit we hit with MSI message routing are first of all KVM GSIs, >> and there only pseudo GSIs that do not go to any interrupt controller >> with limited pins. > > I see KVM_MAX_IRQ_ROUTES 1024 > This is > 256 so KVM does not seem to be the problem. We can generate way more different MSI messages than 256. A message may encode the target CPU, so you have this number in the equation e.g. > >> That could easily be lifted in the kernel if we run >> into shortages in practice. > > What I was saying is that resources are limited even without kvm. What other resources related to this particular case are exhausted before GSI numbers? Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:36146) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RGALb-0004Jx-B6 for qemu-devel@nongnu.org; Tue, 18 Oct 2011 10:08:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RGALZ-00041n-KR for qemu-devel@nongnu.org; Tue, 18 Oct 2011 10:08:51 -0400 Received: from goliath.siemens.de ([192.35.17.28]:16154) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RGALZ-00041E-3u for qemu-devel@nongnu.org; Tue, 18 Oct 2011 10:08:49 -0400 Message-ID: <4E9D886E.3090201@siemens.com> Date: Tue, 18 Oct 2011 16:08:46 +0200 From: Jan Kiszka MIME-Version: 1.0 References: <20111017154852.GC7876@redhat.com> <4E9C81CC.5040900@web.de> <20111018115813.GF28776@redhat.com> <4E9D6C5B.8050204@siemens.com> <20111018123305.GK28776@redhat.com> <4E9D734C.2060504@siemens.com> <20111018124818.GO28776@redhat.com> <4E9D786D.4060802@siemens.com> <20111018133719.GS28776@redhat.com> <4E9D831E.100@siemens.com> <20111018140156.GA4980@redhat.com> In-Reply-To: <20111018140156.GA4980@redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC][PATCH 28/45] qemu-kvm: msix: Drop tracking of used vectors List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: Alex Williamson , Marcelo Tosatti , Avi Kivity , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" On 2011-10-18 16:01, Michael S. Tsirkin wrote: >>>>>>> >>>>>>> I actually would not mind preallocating everything upfront which is much >>>>>>> easier. But with your patch we get a silent failure or a drastic >>>>>>> slowdown which is much more painful IMO. >>>>>> >>>>>> Again: did we already saw that limit? And where does it come from if not >>>>>> from KVM? >>>>> >>>>> It's a hardware limitation of intel APICs. interrupt vector is encoded >>>>> in an 8 bit field in msi address. So you can have at most 256 of these. >>>> >>>> There should be no such limitation with pseudo GSIs we use for MSI >>>> injection. They end up as MSI messages again, so actually 256 (-reserved >>>> vectors) * number-of-cpus (on x86). >>> >>> This limits which CPUs can get the interrupt though. >>> Linux seems to have a global pool as it wants to be able to freely >>> balance vectors between CPUs. Or, consider a guest with a single CPU :) >>> >>> Anyway, why argue - there is a limitation, and it's not coming from KVM, >>> right? >> >> No, our limit we hit with MSI message routing are first of all KVM GSIs, >> and there only pseudo GSIs that do not go to any interrupt controller >> with limited pins. > > I see KVM_MAX_IRQ_ROUTES 1024 > This is > 256 so KVM does not seem to be the problem. We can generate way more different MSI messages than 256. A message may encode the target CPU, so you have this number in the equation e.g. > >> That could easily be lifted in the kernel if we run >> into shortages in practice. > > What I was saying is that resources are limited even without kvm. What other resources related to this particular case are exhausted before GSI numbers? Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux