Re: Enable more than 255 VCPU support without irq remapping function in the guest

[Lan Tianyu <tianyu.lan@intel.com>]

Hi Radim:

On 2016年04月27日 00:49, Radim Krčmář wrote:
> 2016-04-26 18:17+0200, Jan Kiszka:
>> On 2016-04-26 18:14, Lan, Tianyu wrote:
>>> Hi All:
>>>
>>> Recently I am working on extending max vcpu to more than 256 on the both
>>> KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to
>>> use X2APIC in the guest which supports 32-bit APIC id. Linux kernel
>>> requires irq remapping function during enabling X2APIC when max APIC id
>>> is more than 255(More detail please see try_to_enable_x2apic()).
> 
> Our of curiosity, how many VCPUs are you aiming at?

I think it's 1024.

In the short term, we hope hypervisor at least supports 288 vcpus
because Xeon phi chip already supports 288 logical cpus. As hardware
development, there will be more logical cpus and we hope one guest can
totally uses all cpu resources on the chip to meet HPC requirement.

> 
>>> The irq remapping function helps to deliver irq to cpu 255~. IOAPIC just
>>> supports 8-bit target APIC id field and only can deliver irq to
>>> cpu 0~255.
>>>
>>> So far both KVM/Xen doesn't enable irq remapping function. If enable the
>>> function, it seems a huge job which need to rework IO-APIC, local APIC,
>>> MSI parts and add virtual VTD support in the KVM.
>>>
>>> Other quick way to enable more than 256 VCPUs is to eliminate the
>>> dependency between irq remapping and X2APIC in the guest linux kernel.
>>> So far I can boot the guest after removing the dependency.
>>> The side effect I thought is that irq only can deliver to 0~255 vcpus
>>> but 256 vcpus seem enough to balance irq requests in the guest. In the
>>> most cases, there are fewer devices in the guest.
>>>
>>> I wonder whether it's feasible. There maybe some other side effects I
>>> didn't think of. Very appreciate for your comments.
>>
>> Radim is working on the KVM side already, Peter is currently driving the
>> VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
> 
> + Igor extends QEMU to support more than 255 in internal structures and
> ACPI.  What remains mostly untracked is Seabios/OVMF.

Thanks for you information. How about KVM X2APIC part? Do you have patch
to extend KVM X2APIC to support 32-bit APIC ID?



> 
>> PS: Please no PV mess, at least without good reasons.
> 
> Seconded.
> 
> (If we designed all related devices as virtware, then it would not be
>  that bad, but slightly modifying and putting hardware drivers into
>  situations that cannot happen in hardware, not even in the spec, and
>  then juggling the KVM side to make them work, is a road to hell.)
> 


-- 
Best regards
Tianyu Lan