[PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[Ani Sinha]

Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
enabled in the kernel. QEMU has been tested to correctly boot a linux guest
with 1728 vcpus both with edk2 and seabios firmwares. So bump up the max_cpus
value for q35 machines versions 9 and newer to 1728. Q35 machines versions
8.2 and older continue to support 1024 maximum vcpus as before for
compatibility.

If KVM is not able to support the specified number of vcpus, QEMU would
return the following error messages:

$ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)

Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Julia Suvorova <jusual@redhat.com>
Signed-off-by: Ani Sinha <anisinha@redhat.com>
---
 hw/i386/pc_q35.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index f43d5142b8..bfa627a70b 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
     m->default_nic = "e1000e";
     m->default_kernel_irqchip_split = false;
     m->no_floppy = 1;
-    m->max_cpus = 1024;
+    m->max_cpus = 1728;
     m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
@@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
 {
     pc_q35_9_0_machine_options(m);
     m->alias = NULL;
+    m->max_cpus = 1024;
     compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
     compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
 }
-- 
2.42.0

Re: [PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[Daniel P. Berrangé]

On Tue, Jan 30, 2024 at 10:14:28PM +0530, Ani Sinha wrote:
> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> enabled in the kernel. QEMU has been tested to correctly boot a linux guest
> with 1728 vcpus both with edk2 and seabios firmwares. So bump up the max_cpus
> value for q35 machines versions 9 and newer to 1728. Q35 machines versions
> 8.2 and older continue to support 1024 maximum vcpus as before for
> compatibility.

Where does the 1728 number come from ?

Did something break at 1729, or did the test machine simply not
have sufficient resources to do practical larger tests ?

> 
> If KVM is not able to support the specified number of vcpus, QEMU would
> return the following error messages:
> 
> $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> 
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Julia Suvorova <jusual@redhat.com>
> Signed-off-by: Ani Sinha <anisinha@redhat.com>
> ---
>  hw/i386/pc_q35.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index f43d5142b8..bfa627a70b 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      m->default_nic = "e1000e";
>      m->default_kernel_irqchip_split = false;
>      m->no_floppy = 1;
> -    m->max_cpus = 1024;
> +    m->max_cpus = 1728;
>      m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> @@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
>  {
>      pc_q35_9_0_machine_options(m);
>      m->alias = NULL;
> +    m->max_cpus = 1024;
>      compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
>      compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
>  }
> -- 
> 2.42.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[Ani Sinha]

> On 30-Jan-2024, at 22:17, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Tue, Jan 30, 2024 at 10:14:28PM +0530, Ani Sinha wrote:
>> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
>> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
>> enabled in the kernel. QEMU has been tested to correctly boot a linux guest
>> with 1728 vcpus both with edk2 and seabios firmwares. So bump up the max_cpus
>> value for q35 machines versions 9 and newer to 1728. Q35 machines versions
>> 8.2 and older continue to support 1024 maximum vcpus as before for
>> compatibility.
> 
> Where does the 1728 number come from ?
> 
> Did something break at 1729, or did the test machine simply not
> have sufficient resources to do practical larger tests ?

Actual limit currently is 1856 for EDK2. The HPE folks tested QEMU with edk2 and QEMU fails to boot beyond that limit.
There are RH internal bugs tracking this and Gerd is working on it from RH side [1].

We would ultimately like to go to 8192 vcpus for SAP HANA but 1728 vcpus is our immediate target for now. If you want, I can resend the patch with 1856 since that is currently the tested limit.

1. https://issues.redhat.com/browse/RHEL-22202


> 
>> 
>> If KVM is not able to support the specified number of vcpus, QEMU would
>> return the following error messages:
>> 
>> $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
>> qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
>> qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
>> Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
>> 
>> Cc: Daniel P. Berrangé <berrange@redhat.com>
>> Cc: Igor Mammedov <imammedo@redhat.com>
>> Cc: Michael S. Tsirkin <mst@redhat.com>
>> Cc: Julia Suvorova <jusual@redhat.com>
>> Signed-off-by: Ani Sinha <anisinha@redhat.com>
>> ---
>> hw/i386/pc_q35.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
>> index f43d5142b8..bfa627a70b 100644
>> --- a/hw/i386/pc_q35.c
>> +++ b/hw/i386/pc_q35.c
>> @@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
>>     m->default_nic = "e1000e";
>>     m->default_kernel_irqchip_split = false;
>>     m->no_floppy = 1;
>> -    m->max_cpus = 1024;
>> +    m->max_cpus = 1728;
>>     m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
>>     machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
>>     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>> @@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
>> {
>>     pc_q35_9_0_machine_options(m);
>>     m->alias = NULL;
>> +    m->max_cpus = 1024;
>>     compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
>>     compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
>> }
>> -- 
>> 2.42.0
>> 
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[Daniel P. Berrangé]

On Tue, Jan 30, 2024 at 10:39:51PM +0530, Ani Sinha wrote:
> 
> 
> > On 30-Jan-2024, at 22:17, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Tue, Jan 30, 2024 at 10:14:28PM +0530, Ani Sinha wrote:
> >> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> >> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> >> enabled in the kernel. QEMU has been tested to correctly boot a linux guest
> >> with 1728 vcpus both with edk2 and seabios firmwares. So bump up the max_cpus
> >> value for q35 machines versions 9 and newer to 1728. Q35 machines versions
> >> 8.2 and older continue to support 1024 maximum vcpus as before for
> >> compatibility.
> > 
> > Where does the 1728 number come from ?
> > 
> > Did something break at 1729, or did the test machine simply not
> > have sufficient resources to do practical larger tests ?
> 
> Actual limit currently is 1856 for EDK2. The HPE folks tested QEMU with edk2 and QEMU fails to boot beyond that limit.
> There are RH internal bugs tracking this and Gerd is working on it from RH side [1].
> 
> We would ultimately like to go to 8192 vcpus for SAP HANA but 1728 vcpus is our immediate target for now. If you want, I can resend the patch with 1856 since that is currently the tested limit.

Yes, could you resend with 1856, and include a description of
the blocking problem in the commit message for the historical
record, as this is the kind of thing that people will have
forgotten when re-visiting the patch later and wondering why
this limit was chosen.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[Philippe Mathieu-Daudé]

On 30/1/24 18:09, Ani Sinha wrote:
> 
> 
>> On 30-Jan-2024, at 22:17, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>
>> On Tue, Jan 30, 2024 at 10:14:28PM +0530, Ani Sinha wrote:
>>> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
>>> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
>>> enabled in the kernel. QEMU has been tested to correctly boot a linux guest
>>> with 1728 vcpus both with edk2 and seabios firmwares. So bump up the max_cpus
>>> value for q35 machines versions 9 and newer to 1728. Q35 machines versions
>>> 8.2 and older continue to support 1024 maximum vcpus as before for
>>> compatibility.
>>
>> Where does the 1728 number come from ?
>>
>> Did something break at 1729, or did the test machine simply not
>> have sufficient resources to do practical larger tests ?
> 
> Actual limit currently is 1856 for EDK2. The HPE folks tested QEMU with edk2 and QEMU fails to boot beyond that limit.
> There are RH internal bugs tracking this and Gerd is working on it from RH side [1].
> 
> We would ultimately like to go to 8192 vcpus for SAP HANA but 1728 vcpus is our immediate target for now. If you want, I can resend the patch with 1856 since that is currently the tested limit.
> 
> 1. https://issues.redhat.com/browse/RHEL-22202
Out of curiosity has the limit to be multiple of 64?

Re: [PATCH] pc: q35: Bump max_cpus to 1728 vcpus

[Michael S. Tsirkin]

On Tue, Jan 30, 2024 at 10:39:51PM +0530, Ani Sinha wrote:
> 
> 
> > On 30-Jan-2024, at 22:17, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Tue, Jan 30, 2024 at 10:14:28PM +0530, Ani Sinha wrote:
> >> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> >> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> >> enabled in the kernel. QEMU has been tested to correctly boot a linux guest
> >> with 1728 vcpus both with edk2 and seabios firmwares. So bump up the max_cpus
> >> value for q35 machines versions 9 and newer to 1728. Q35 machines versions
> >> 8.2 and older continue to support 1024 maximum vcpus as before for
> >> compatibility.
> > 
> > Where does the 1728 number come from ?
> > 
> > Did something break at 1729, or did the test machine simply not
> > have sufficient resources to do practical larger tests ?
> 
> Actual limit currently is 1856 for EDK2. The HPE folks tested QEMU with edk2 and QEMU fails to boot beyond that limit.
> There are RH internal bugs tracking this and Gerd is working on it from RH side [1].
> 
> We would ultimately like to go to 8192 vcpus for SAP HANA but 1728 vcpus is our immediate target for now. If you want, I can resend the patch with 1856 since that is currently the tested limit.
> 
> 1. https://issues.redhat.com/browse/RHEL-22202

What is requested here is that you document the source of the number whatever
it is in a code comment and commit log.


> 
> > 
> >> 
> >> If KVM is not able to support the specified number of vcpus, QEMU would
> >> return the following error messages:
> >> 
> >> $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> >> qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> >> qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> >> Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> >> 
> >> Cc: Daniel P. Berrangé <berrange@redhat.com>
> >> Cc: Igor Mammedov <imammedo@redhat.com>
> >> Cc: Michael S. Tsirkin <mst@redhat.com>
> >> Cc: Julia Suvorova <jusual@redhat.com>
> >> Signed-off-by: Ani Sinha <anisinha@redhat.com>
> >> ---
> >> hw/i386/pc_q35.c | 3 ++-
> >> 1 file changed, 2 insertions(+), 1 deletion(-)
> >> 
> >> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> >> index f43d5142b8..bfa627a70b 100644
> >> --- a/hw/i386/pc_q35.c
> >> +++ b/hw/i386/pc_q35.c
> >> @@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
> >>     m->default_nic = "e1000e";
> >>     m->default_kernel_irqchip_split = false;
> >>     m->no_floppy = 1;
> >> -    m->max_cpus = 1024;
> >> +    m->max_cpus = 1728;
> >>     m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
> >>     machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
> >>     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> >> @@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
> >> {
> >>     pc_q35_9_0_machine_options(m);
> >>     m->alias = NULL;
> >> +    m->max_cpus = 1024;
> >>     compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
> >>     compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
> >> }
> >> -- 
> >> 2.42.0
> >> 
> > 
> > With regards,
> > Daniel
> > -- 
> > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>