KVM_MAX_VCPU hard limit of 255 on x86

KVM_MAX_VCPU hard limit of 255 on x86

[Hardik H Bagdi]

Hello,

For research, I am experimenting with increasing the number of VCPUs
in the guest OS.
I can increase the number of VCPUs till 255 in the guest OS but more
than that results in the following error-

kvm_init_vcpu failed: Invalid argument

The hardcodede value of 255 can be found here for x86.
./arch/x86/include/asm/kvm_host.h:38:#define KVM_MAX_VCPUS 255

#define a new value is not successful i.e. the same error persists.
I'm using qemu 2.5.0 with host kernel 3.19.0
Any help will be really appreciated.

Thank you,
Hardik Bagdi

Re: KVM_MAX_VCPU hard limit of 255 on x86

[Bandan Das]

Hardik H Bagdi <hbagdi1@binghamton.edu> writes:

> Hello,
>
> For research, I am experimenting with increasing the number of VCPUs
> in the guest OS.
> I can increase the number of VCPUs till 255 in the guest OS but more
> than that results in the following error-
>
> kvm_init_vcpu failed: Invalid argument
>
> The hardcodede value of 255 can be found here for x86.
> ./arch/x86/include/asm/kvm_host.h:38:#define KVM_MAX_VCPUS 255
>
> #define a new value is not successful i.e. the same error persists.
> I'm using qemu 2.5.0 with host kernel 3.19.0
> Any help will be really appreciated.

Radim (Cced) might have more info but the short answer is that you
can't just simply increase the number. For >255, you need x2apic
with interrupt remapping which is still missing. 

> Thank you,
> Hardik Bagdi
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM_MAX_VCPU hard limit of 255 on x86

[Radim Krčmář]

2016-04-01 17:55-0400, Bandan Das:
> Hardik H Bagdi <hbagdi1@binghamton.edu> writes:
>> Hello,
>>
>> For research, I am experimenting with increasing the number of VCPUs
>> in the guest OS.
>> I can increase the number of VCPUs till 255 in the guest OS but more
>> than that results in the following error-
> 
> Radim (Cced) might have more info but the short answer is that you
> can't just simply increase the number. For >255, you need x2apic
> with interrupt remapping which is still missing. 

Yes, interrupt remapping is being developed for the split irqchip.
You could hack around that (we chose not to) so another blocker is that
QEMU cannot handle more than 255 VCPUs, mainly because of its AML
generator.  Igor (Cc'd) knows more about the status of ACPI.

(And firmware doesn't implement x2APIC, and other minor problems ...)

All pieces should be in place within half a year, though.
The next hard limit is ~2^20 or ~2^32.

Re: KVM_MAX_VCPU hard limit of 255 on x86

[Igor Mammedov]

On Mon, 4 Apr 2016 17:17:19 +0200
Radim Krčmář <rkrcmar@redhat.com> wrote:

> 2016-04-01 17:55-0400, Bandan Das:
> > Hardik H Bagdi <hbagdi1@binghamton.edu> writes:
> >> Hello,
> >>
> >> For research, I am experimenting with increasing the number of
> >> VCPUs in the guest OS.
> >> I can increase the number of VCPUs till 255 in the guest OS but
> >> more than that results in the following error-
> > 
> > Radim (Cced) might have more info but the short answer is that you
> > can't just simply increase the number. For >255, you need x2apic
> > with interrupt remapping which is still missing. 
> 
> Yes, interrupt remapping is being developed for the split irqchip.
> You could hack around that (we chose not to) so another blocker is
> that QEMU cannot handle more than 255 VCPUs, mainly because of its AML
> generator.  Igor (Cc'd) knows more about the status of ACPI.
I'm going to post patches that enable more than 255 VCPUs
on ACPI side of QEMU once development window for 2.7 opens.
 
> (And firmware doesn't implement x2APIC, and other minor problems ...)
Firmware probably doesn't need more than 1 VCPU, maybe we can
get rid of if it enumerating VCPUs via broadcast AP wakeup
since it no longer builds ACPI tables anymore.

It should save some boot time due to removing timeout for waiting
APs wakeup.
 
> All pieces should be in place within half a year, though.
> The next hard limit is ~2^20 or ~2^32.
In current version of my ACPI patches it's 2^32 limit so far.

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM_MAX_VCPU hard limit of 255 on x86

[Radim Krčmář]

2016-04-04 21:14+0200, Igor Mammedov:
> On Mon, 4 Apr 2016 17:17:19 +0200
> Radim Krčmář <rkrcmar@redhat.com> wrote:
>>                                                 another blocker is
>> that QEMU cannot handle more than 255 VCPUs, mainly because of its AML
>> generator.  Igor (Cc'd) knows more about the status of ACPI.
> I'm going to post patches that enable more than 255 VCPUs
> on ACPI side of QEMU once development window for 2.7 opens.

Great, thanks!

>> (And firmware doesn't implement x2APIC, and other minor problems ...)
> Firmware probably doesn't need more than 1 VCPU, maybe we can
> get rid of if it enumerating VCPUs via broadcast AP wakeup
> since it no longer builds ACPI tables anymore.
> 
> It should save some boot time due to removing timeout for waiting
> APs wakeup.

Rudimentary x2APIC support is needed, because the OS should get control
with all LAPICs in x2APIC mode when any APIC ID is over 255.
We could code that in QEMU to save some time, but I am reluctant to move
the whole MP initialization before >255 is proven as working. :)

>> All pieces should be in place within half a year, though.
>> The next hard limit is ~2^20 or ~2^32.
> In current version of my ACPI patches it's 2^32 limit so far.

Perfect.  (2^16-1)*16 would be a limit only if some major OS didn't want
understand that logical addressing doesn't work beyond that ...
(Which shouldn't happen.)

Re: KVM_MAX_VCPU hard limit of 255 on x86

[Igor Mammedov]

On Mon, 4 Apr 2016 22:34:08 +0200
Radim Krčmář <rkrcmar@redhat.com> wrote:

> 2016-04-04 21:14+0200, Igor Mammedov:
> >> (And firmware doesn't implement x2APIC, and other minor
> >> problems ...)
> > Firmware probably doesn't need more than 1 VCPU, maybe we can
> > get rid of if it enumerating VCPUs via broadcast AP wakeup
> > since it no longer builds ACPI tables anymore.
> > 
> > It should save some boot time due to removing timeout for waiting
> > APs wakeup.
> 
> Rudimentary x2APIC support is needed, because the OS should get
> control with all LAPICs in x2APIC mode when any APIC ID is over 255.
> We could code that in QEMU to save some time, but I am reluctant to
> move the whole MP initialization before >255 is proven as working. :)
As you said it's up to OS to get control of all LAPICs, i.e. wake them
up. I probably miss something but how would firmware be involved in
this?

Re: KVM_MAX_VCPU hard limit of 255 on x86

[Radim Krčmář]

2016-04-05 13:20+0200, Igor Mammedov:
> On Mon, 4 Apr 2016 22:34:08 +0200
> Radim Krčmář <rkrcmar@redhat.com> wrote:
>> 2016-04-04 21:14+0200, Igor Mammedov:
>> >> (And firmware doesn't implement x2APIC, and other minor
>> >> problems ...)
>> > Firmware probably doesn't need more than 1 VCPU, maybe we can
>> > get rid of if it enumerating VCPUs via broadcast AP wakeup
>> > since it no longer builds ACPI tables anymore.
>> > 
>> > It should save some boot time due to removing timeout for waiting
>> > APs wakeup.
>> 
>> Rudimentary x2APIC support is needed, because the OS should get
>> control with all LAPICs in x2APIC mode when any APIC ID is over 255.
>> We could code that in QEMU to save some time, but I am reluctant to
>> move the whole MP initialization before >255 is proven as working. :)
> As you said it's up to OS to get control of all LAPICs, i.e. wake them
> up. I probably miss something but how would firmware be involved in
> this?

The spec says that putting all LAPICs into x2APIC mode is firmware's job
(10.12.8.1 Consistency of APIC IDs and CPUID, but we can do it in QEMU).

An OS can expect that all LAPICs will be in x2APIC mode, which causes a
problem if the firmware didn't switch to x2APIC, because non-broadcast
INIT/SIPI to to high APIC IDs would fail.

I'm lost now, are we talking about optimizing seabios' smp_setup()?

(I think smp_setup touches APs because of SDM, Dec 2015, 8.4.3 MP
 Initialization Protocol Algorithm for MP Systems, items 6 and 7.)