From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yang Zhang Subject: Re: Enable more than 255 VCPU support without irq remapping function in the guest Date: Thu, 28 Apr 2016 09:11:29 +0800 Message-ID: <57216341.80006@gmail.com> References: <571F93CA.40200@intel.com> <571F9487.5090009@siemens.com> <20160426164939.GA18900@potion> <57203B9D.6020402@gmail.com> <57204D28.4070706@siemens.com> <572088D0.7040805@gmail.com> <57208A54.40502@siemens.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "Lan, Tianyu" , pbonzini@redhat.com, kvm@vger.kernel.org, tglx@linutronix.de, gleb@redhat.com, mst@redhat.com, x86@kernel.org, Peter Xu , Igor Mammedov To: Jan Kiszka , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= Return-path: Received: from mail-pf0-f193.google.com ([209.85.192.193]:35118 "EHLO mail-pf0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751257AbcD1BLj (ORCPT ); Wed, 27 Apr 2016 21:11:39 -0400 Received: by mail-pf0-f193.google.com with SMTP id r187so7849249pfr.2 for ; Wed, 27 Apr 2016 18:11:39 -0700 (PDT) In-Reply-To: <57208A54.40502@siemens.com> Sender: kvm-owner@vger.kernel.org List-ID: On 2016/4/27 17:45, Jan Kiszka wrote: > On 2016-04-27 11:39, Yang Zhang wrote: >> On 2016/4/27 13:24, Jan Kiszka wrote: >>> On 2016-04-27 06:10, Yang Zhang wrote: >>>> On 2016/4/27 0:49, Radim Kr=C4=8Dm=C3=A1=C5=99 wrote: >>>>> 2016-04-26 18:17+0200, Jan Kiszka: >>>>>> On 2016-04-26 18:14, Lan, Tianyu wrote: >>>>>>> Hi All: >>>>>>> >>>>>>> Recently I am working on extending max vcpu to more than 256 on= the >>>>>>> both >>>>>>> KVM/Xen. For some HPC cases, it needs many vcpus. The job requi= res to >>>>>>> use X2APIC in the guest which supports 32-bit APIC id. Linux ke= rnel >>>>>>> requires irq remapping function during enabling X2APIC when max >>>>>>> APIC id >>>>>>> is more than 255(More detail please see try_to_enable_x2apic())= =2E >>>>> >>>>> Our of curiosity, how many VCPUs are you aiming at? >>>>> >>>>>>> The irq remapping function helps to deliver irq to cpu 255~. IO= APIC >>>>>>> just >>>>>>> supports 8-bit target APIC id field and only can deliver irq to >>>>>>> cpu 0~255. >>>>>>> >>>>>>> So far both KVM/Xen doesn't enable irq remapping function. If e= nable >>>>>>> the >>>>>>> function, it seems a huge job which need to rework IO-APIC, loc= al >>>>>>> APIC, >>>>>>> MSI parts and add virtual VTD support in the KVM. >>>>>>> >>>>>>> Other quick way to enable more than 256 VCPUs is to eliminate t= he >>>>>>> dependency between irq remapping and X2APIC in the guest linux >>>>>>> kernel. >>>>>>> So far I can boot the guest after removing the dependency. >>>>>>> The side effect I thought is that irq only can deliver to 0~255= vcpus >>>>>>> but 256 vcpus seem enough to balance irq requests in the guest.= In >>>>>>> the >>>>>>> most cases, there are fewer devices in the guest. >>>>>>> >>>>>>> I wonder whether it's feasible. There maybe some other side eff= ects I >>>>>>> didn't think of. Very appreciate for your comments. >>>>>> >>>>>> Radim is working on the KVM side already, Peter is currently >>>>>> driving the >>>>>> VT-d interrupt emulation topic in QEMU. It's in reach, I would s= ay. :) >>>>> >>>>> + Igor extends QEMU to support more than 255 in internal structur= es and >>>>> ACPI. What remains mostly untracked is Seabios/OVMF. >>>> >>>> If we don't want the interrupt from internal device delivers to CP= U >>>>> 255, do we still need the VT-d interrupt remapping emulation? I t= hink >>>> firmware is able to send IPI to wakeup APs even without IR and OS = is >>>> able to do it too. So basically, only KVM and Qemu's support is en= ough. >>> >>> What are "internal devices" for you? And which OS do you know that = would >>> handle such artificial setups without prio massive patching? >> >> Sorry, a typo. I mean the external devices of IOAPIC/MSI/MSIX. Doesn= 't >> current Linux use x2apic without IR in VM? > > If and only if there only need to be 254 CPUs to be addressed. > >> >>> >>> We do need VT-d IR emulation in order to present our guest a well >>> specified and support architecture for running > 255 CPUs. >> >> I mean in Tianyu's case, if he doesn't care about to deliver externa= l >> interrupt to CPU >255, IR is not required. > > What matters is the guest OS. See my other reply on this why this > doesn't work, even for Linux. Since there only few devices in his case, set the irq affinity manually= =20 is enough. --=20 best regards yang