From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: Enable more than 255 VCPU support without irq remapping function in the guest Date: Wed, 27 Apr 2016 07:24:56 +0200 Message-ID: <57204D28.4070706@siemens.com> References: <571F93CA.40200@intel.com> <571F9487.5090009@siemens.com> <20160426164939.GA18900@potion> <57203B9D.6020402@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "Lan, Tianyu" , pbonzini@redhat.com, kvm@vger.kernel.org, tglx@linutronix.de, gleb@redhat.com, mst@redhat.com, x86@kernel.org, Peter Xu , Igor Mammedov To: Yang Zhang , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= Return-path: Received: from thoth.sbs.de ([192.35.17.2]:53530 "EHLO thoth.sbs.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752430AbcD0FZO (ORCPT ); Wed, 27 Apr 2016 01:25:14 -0400 In-Reply-To: <57203B9D.6020402@gmail.com> Sender: kvm-owner@vger.kernel.org List-ID: On 2016-04-27 06:10, Yang Zhang wrote: > On 2016/4/27 0:49, Radim Kr=C4=8Dm=C3=A1=C5=99 wrote: >> 2016-04-26 18:17+0200, Jan Kiszka: >>> On 2016-04-26 18:14, Lan, Tianyu wrote: >>>> Hi All: >>>> >>>> Recently I am working on extending max vcpu to more than 256 on th= e >>>> both >>>> KVM/Xen. For some HPC cases, it needs many vcpus. The job requires= to >>>> use X2APIC in the guest which supports 32-bit APIC id. Linux kerne= l >>>> requires irq remapping function during enabling X2APIC when max AP= IC id >>>> is more than 255(More detail please see try_to_enable_x2apic()). >> >> Our of curiosity, how many VCPUs are you aiming at? >> >>>> The irq remapping function helps to deliver irq to cpu 255~. IOAPI= C >>>> just >>>> supports 8-bit target APIC id field and only can deliver irq to >>>> cpu 0~255. >>>> >>>> So far both KVM/Xen doesn't enable irq remapping function. If enab= le >>>> the >>>> function, it seems a huge job which need to rework IO-APIC, local = APIC, >>>> MSI parts and add virtual VTD support in the KVM. >>>> >>>> Other quick way to enable more than 256 VCPUs is to eliminate the >>>> dependency between irq remapping and X2APIC in the guest linux ker= nel. >>>> So far I can boot the guest after removing the dependency. >>>> The side effect I thought is that irq only can deliver to 0~255 vc= pus >>>> but 256 vcpus seem enough to balance irq requests in the guest. In= the >>>> most cases, there are fewer devices in the guest. >>>> >>>> I wonder whether it's feasible. There maybe some other side effect= s I >>>> didn't think of. Very appreciate for your comments. >>> >>> Radim is working on the KVM side already, Peter is currently drivin= g the >>> VT-d interrupt emulation topic in QEMU. It's in reach, I would say.= :) >> >> + Igor extends QEMU to support more than 255 in internal structures = and >> ACPI. What remains mostly untracked is Seabios/OVMF. >=20 > If we don't want the interrupt from internal device delivers to CPU >>255, do we still need the VT-d interrupt remapping emulation? I think > firmware is able to send IPI to wakeup APs even without IR and OS is > able to do it too. So basically, only KVM and Qemu's support is enoug= h. What are "internal devices" for you? And which OS do you know that woul= d handle such artificial setups without prio massive patching? We do need VT-d IR emulation in order to present our guest a well specified and support architecture for running > 255 CPUs. Jan --=20 Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux