Re: Enable more than 255 VCPU support without irq remapping function in the guest

[Yang Zhang <yang.zhang.wz@gmail.com>]

On 2016/4/27 13:24, Jan Kiszka wrote:
> On 2016-04-27 06:10, Yang Zhang wrote:
>> On 2016/4/27 0:49, Radim Krčmář wrote:
>>> 2016-04-26 18:17+0200, Jan Kiszka:
>>>> On 2016-04-26 18:14, Lan, Tianyu wrote:
>>>>> Hi All:
>>>>>
>>>>> Recently I am working on extending max vcpu to more than 256 on the
>>>>> both
>>>>> KVM/Xen. For some HPC cases, it needs many vcpus. The job requires to
>>>>> use X2APIC in the guest which supports 32-bit APIC id. Linux kernel
>>>>> requires irq remapping function during enabling X2APIC when max APIC id
>>>>> is more than 255(More detail please see try_to_enable_x2apic()).
>>>
>>> Our of curiosity, how many VCPUs are you aiming at?
>>>
>>>>> The irq remapping function helps to deliver irq to cpu 255~. IOAPIC
>>>>> just
>>>>> supports 8-bit target APIC id field and only can deliver irq to
>>>>> cpu 0~255.
>>>>>
>>>>> So far both KVM/Xen doesn't enable irq remapping function. If enable
>>>>> the
>>>>> function, it seems a huge job which need to rework IO-APIC, local APIC,
>>>>> MSI parts and add virtual VTD support in the KVM.
>>>>>
>>>>> Other quick way to enable more than 256 VCPUs is to eliminate the
>>>>> dependency between irq remapping and X2APIC in the guest linux kernel.
>>>>> So far I can boot the guest after removing the dependency.
>>>>> The side effect I thought is that irq only can deliver to 0~255 vcpus
>>>>> but 256 vcpus seem enough to balance irq requests in the guest. In the
>>>>> most cases, there are fewer devices in the guest.
>>>>>
>>>>> I wonder whether it's feasible. There maybe some other side effects I
>>>>> didn't think of. Very appreciate for your comments.
>>>>
>>>> Radim is working on the KVM side already, Peter is currently driving the
>>>> VT-d interrupt emulation topic in QEMU. It's in reach, I would say. :)
>>>
>>> + Igor extends QEMU to support more than 255 in internal structures and
>>> ACPI.  What remains mostly untracked is Seabios/OVMF.
>>
>> If we don't want the interrupt from internal device delivers to CPU
>>> 255, do we still need the VT-d interrupt remapping emulation? I think
>> firmware is able to send IPI to wakeup APs even without IR and OS is
>> able to do it too. So basically, only KVM and Qemu's support is enough.
>
> What are "internal devices" for you? And which OS do you know that would
> handle such artificial setups without prio massive patching?

Sorry, a typo. I mean the external devices of IOAPIC/MSI/MSIX. Doesn't 
current Linux use x2apic without IR in VM?

>
> We do need VT-d IR emulation in order to present our guest a well
> specified and support architecture for running > 255 CPUs.

I mean in Tianyu's case, if he doesn't care about to deliver external 
interrupt to CPU >255, IR is not required.


-- 
best regards
yang