From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yang Zhang Subject: Re: Enable more than 255 VCPU support without irq remapping function in the guest Date: Wed, 27 Apr 2016 12:10:05 +0800 Message-ID: <57203B9D.6020402@gmail.com> References: <571F93CA.40200@intel.com> <571F9487.5090009@siemens.com> <20160426164939.GA18900@potion> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "Lan, Tianyu" , pbonzini@redhat.com, kvm@vger.kernel.org, tglx@linutronix.de, gleb@redhat.com, mst@redhat.com, x86@kernel.org, Peter Xu , Igor Mammedov To: =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , Jan Kiszka Return-path: Received: from mail-pf0-f194.google.com ([209.85.192.194]:34760 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750811AbcD0EKV (ORCPT ); Wed, 27 Apr 2016 00:10:21 -0400 Received: by mail-pf0-f194.google.com with SMTP id 145so3294416pfz.1 for ; Tue, 26 Apr 2016 21:10:20 -0700 (PDT) In-Reply-To: <20160426164939.GA18900@potion> Sender: kvm-owner@vger.kernel.org List-ID: On 2016/4/27 0:49, Radim Kr=C4=8Dm=C3=A1=C5=99 wrote: > 2016-04-26 18:17+0200, Jan Kiszka: >> On 2016-04-26 18:14, Lan, Tianyu wrote: >>> Hi All: >>> >>> Recently I am working on extending max vcpu to more than 256 on the= both >>> KVM/Xen. For some HPC cases, it needs many vcpus. The job requires = to >>> use X2APIC in the guest which supports 32-bit APIC id. Linux kernel >>> requires irq remapping function during enabling X2APIC when max API= C id >>> is more than 255(More detail please see try_to_enable_x2apic()). > > Our of curiosity, how many VCPUs are you aiming at? > >>> The irq remapping function helps to deliver irq to cpu 255~. IOAPIC= just >>> supports 8-bit target APIC id field and only can deliver irq to >>> cpu 0~255. >>> >>> So far both KVM/Xen doesn't enable irq remapping function. If enabl= e the >>> function, it seems a huge job which need to rework IO-APIC, local A= PIC, >>> MSI parts and add virtual VTD support in the KVM. >>> >>> Other quick way to enable more than 256 VCPUs is to eliminate the >>> dependency between irq remapping and X2APIC in the guest linux kern= el. >>> So far I can boot the guest after removing the dependency. >>> The side effect I thought is that irq only can deliver to 0~255 vcp= us >>> but 256 vcpus seem enough to balance irq requests in the guest. In = the >>> most cases, there are fewer devices in the guest. >>> >>> I wonder whether it's feasible. There maybe some other side effects= I >>> didn't think of. Very appreciate for your comments. >> >> Radim is working on the KVM side already, Peter is currently driving= the >> VT-d interrupt emulation topic in QEMU. It's in reach, I would say. = :) > > + Igor extends QEMU to support more than 255 in internal structures a= nd > ACPI. What remains mostly untracked is Seabios/OVMF. If we don't want the interrupt from internal device delivers to CPU=20 >255, do we still need the VT-d interrupt remapping emulation? I think= =20 firmware is able to send IPI to wakeup APs even without IR and OS is=20 able to do it too. So basically, only KVM and Qemu's support is enough. > >> PS: Please no PV mess, at least without good reasons. > > Seconded. > > (If we designed all related devices as virtware, then it would not be > that bad, but slightly modifying and putting hardware drivers into > situations that cannot happen in hardware, not even in the spec, an= d > then juggling the KVM side to make them work, is a road to hell.) > --=20 best regards yang