[PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[Ani Sinha]

Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
enabled in the kernel. At present, QEMU has been tested to correctly boot a
linux guest with 4096 vcpus both with edk2 and seabios firmwares.
So bump up the value max_cpus to 4096 for q35 machines versions 9 and newer.
Q35 machines versions 8.2 and older continue to support 1024 maximum vcpus
as before for compatibility reasons.

If KVM is not able to support the specified number of vcpus, QEMU would
return the following error messages:

$ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)

Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Julia Suvorova <jusual@redhat.com>
Cc: kraxel@redhat.com
Signed-off-by: Ani Sinha <anisinha@redhat.com>
---
 hw/i386/pc_q35.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Changelog:
v3: bump up to 4096 vcpus. It has now been tested to work with edk2.
See RH Jira: https://issues.redhat.com/browse/RHEL-22202

v2: bump up the vcpu number to 1856. Add failure messages from ekd2 in
the commit description.

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index d346fa3b1d..ae60e6b919 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -366,7 +366,7 @@ static void pc_q35_machine_options(MachineClass *m)
     m->default_nic = "e1000e";
     m->default_kernel_irqchip_split = false;
     m->no_floppy = 1;
-    m->max_cpus = 1024;
+    m->max_cpus = 4096;
     m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
@@ -387,6 +387,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
 {
     pc_q35_9_0_machine_options(m);
     m->alias = NULL;
+    m->max_cpus = 1024;
     compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
     compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
 }
-- 
2.42.0

Re: [PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[Daniel P. Berrangé]

On Tue, Feb 20, 2024 at 09:12:04PM +0530, Ani Sinha wrote:
> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> enabled in the kernel. At present, QEMU has been tested to correctly boot a
> linux guest with 4096 vcpus both with edk2 and seabios firmwares.
> So bump up the value max_cpus to 4096 for q35 machines versions 9 and newer.
> Q35 machines versions 8.2 and older continue to support 1024 maximum vcpus
> as before for compatibility reasons.
> 
> If KVM is not able to support the specified number of vcpus, QEMU would
> return the following error messages:
> 
> $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> 
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Julia Suvorova <jusual@redhat.com>
> Cc: kraxel@redhat.com
> Signed-off-by: Ani Sinha <anisinha@redhat.com>
> ---
>  hw/i386/pc_q35.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> Changelog:
> v3: bump up to 4096 vcpus. It has now been tested to work with edk2.
> See RH Jira: https://issues.redhat.com/browse/RHEL-22202

That bug indicates a dependancy on a EDK2 patch

  https://github.com/kraxel/edk2/commit/7a03c17f0f4f4a9003d77db2660c8e087604b2f0

we'll need to rebase the EDK2 ROMs in QEMU to get that included.

Meanwhile, plesae at least call out this EDK2 commit as a
pre-requisite in the commit message, so people know the
EDK2 ROMs in QEMU won't work (yet).

> 
> v2: bump up the vcpu number to 1856. Add failure messages from ekd2 in
> the commit description.
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index d346fa3b1d..ae60e6b919 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -366,7 +366,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      m->default_nic = "e1000e";
>      m->default_kernel_irqchip_split = false;
>      m->no_floppy = 1;
> -    m->max_cpus = 1024;
> +    m->max_cpus = 4096;
>      m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> @@ -387,6 +387,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
>  {
>      pc_q35_9_0_machine_options(m);
>      m->alias = NULL;
> +    m->max_cpus = 1024;
>      compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
>      compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
>  }

Assuming tweaked commit message

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[Gerd Hoffmann]

On Tue, Feb 20, 2024 at 03:50:57PM +0000, Daniel P. Berrangé wrote:
> On Tue, Feb 20, 2024 at 09:12:04PM +0530, Ani Sinha wrote:
> > Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> > Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> > enabled in the kernel. At present, QEMU has been tested to correctly boot a
> > linux guest with 4096 vcpus both with edk2 and seabios firmwares.
> > So bump up the value max_cpus to 4096 for q35 machines versions 9 and newer.
> > Q35 machines versions 8.2 and older continue to support 1024 maximum vcpus
> > as before for compatibility reasons.
> > 
> > If KVM is not able to support the specified number of vcpus, QEMU would
> > return the following error messages:
> > 
> > $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> > qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> > 
> > Cc: Daniel P. Berrangé <berrange@redhat.com>
> > Cc: Igor Mammedov <imammedo@redhat.com>
> > Cc: Michael S. Tsirkin <mst@redhat.com>
> > Cc: Julia Suvorova <jusual@redhat.com>
> > Cc: kraxel@redhat.com
> > Signed-off-by: Ani Sinha <anisinha@redhat.com>
> > ---
> >  hw/i386/pc_q35.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > Changelog:
> > v3: bump up to 4096 vcpus. It has now been tested to work with edk2.
> > See RH Jira: https://issues.redhat.com/browse/RHEL-22202
> 
> That bug indicates a dependancy on a EDK2 patch
> 
>   https://github.com/kraxel/edk2/commit/7a03c17f0f4f4a9003d77db2660c8e087604b2f0
> 
> we'll need to rebase the EDK2 ROMs in QEMU to get that included.

Which will btw take a while.  edk2 is in freeze for the 2024-02 release
right now, I expect the changes land upstream shortly thereafter and
will be part of the 2024-05 release.  So end may / early june would be
the time when rebasing to release, or somewhen in march or april when we
rebase to a git snapshot ...

> Meanwhile, plesae at least call out this EDK2 commit as a
> pre-requisite in the commit message, so people know the
> EDK2 ROMs in QEMU won't work (yet).

That surely makes sense.

Oh, and it's more than just that one commit.  I don't think it makes
sense to compile a list of commits given this is a moving target
(upstream review is in progress right now).

take care,
  Gerd

Re: [PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[Daniel P. Berrangé]

On Tue, Feb 20, 2024 at 05:16:37PM +0100, Gerd Hoffmann wrote:
> On Tue, Feb 20, 2024 at 03:50:57PM +0000, Daniel P. Berrangé wrote:
> > On Tue, Feb 20, 2024 at 09:12:04PM +0530, Ani Sinha wrote:
> > > Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> > > Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> > > enabled in the kernel. At present, QEMU has been tested to correctly boot a
> > > linux guest with 4096 vcpus both with edk2 and seabios firmwares.
> > > So bump up the value max_cpus to 4096 for q35 machines versions 9 and newer.
> > > Q35 machines versions 8.2 and older continue to support 1024 maximum vcpus
> > > as before for compatibility reasons.
> > > 
> > > If KVM is not able to support the specified number of vcpus, QEMU would
> > > return the following error messages:
> > > 
> > > $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> > > qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > > qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > > Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> > > 
> > > Cc: Daniel P. Berrangé <berrange@redhat.com>
> > > Cc: Igor Mammedov <imammedo@redhat.com>
> > > Cc: Michael S. Tsirkin <mst@redhat.com>
> > > Cc: Julia Suvorova <jusual@redhat.com>
> > > Cc: kraxel@redhat.com
> > > Signed-off-by: Ani Sinha <anisinha@redhat.com>
> > > ---
> > >  hw/i386/pc_q35.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > Changelog:
> > > v3: bump up to 4096 vcpus. It has now been tested to work with edk2.
> > > See RH Jira: https://issues.redhat.com/browse/RHEL-22202
> > 
> > That bug indicates a dependancy on a EDK2 patch
> > 
> >   https://github.com/kraxel/edk2/commit/7a03c17f0f4f4a9003d77db2660c8e087604b2f0
> > 
> > we'll need to rebase the EDK2 ROMs in QEMU to get that included.
> 
> Which will btw take a while.  edk2 is in freeze for the 2024-02 release
> right now, I expect the changes land upstream shortly thereafter and
> will be part of the 2024-05 release.  So end may / early june would be
> the time when rebasing to release, or somewhen in march or april when we
> rebase to a git snapshot ...
> 
> > Meanwhile, plesae at least call out this EDK2 commit as a
> > pre-requisite in the commit message, so people know the
> > EDK2 ROMs in QEMU won't work (yet).
> 
> That surely makes sense.
> 
> Oh, and it's more than just that one commit.  I don't think it makes
> sense to compile a list of commits given this is a moving target
> (upstream review is in progress right now).

Ok, lets just mention something hand-wavy like

 "pending various upstream EDK2 fixes which will probably be
  in the 2024-05 release"


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[Ani Sinha]

> On 20-Feb-2024, at 21:46, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> On Tue, Feb 20, 2024 at 03:50:57PM +0000, Daniel P. Berrangé wrote:
>> On Tue, Feb 20, 2024 at 09:12:04PM +0530, Ani Sinha wrote:
>>> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
>>> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
>>> enabled in the kernel. At present, QEMU has been tested to correctly boot a
>>> linux guest with 4096 vcpus both with edk2 and seabios firmwares.
>>> So bump up the value max_cpus to 4096 for q35 machines versions 9 and newer.
>>> Q35 machines versions 8.2 and older continue to support 1024 maximum vcpus
>>> as before for compatibility reasons.
>>> 
>>> If KVM is not able to support the specified number of vcpus, QEMU would
>>> return the following error messages:
>>> 
>>> $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
>>> qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
>>> qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
>>> Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
>>> 
>>> Cc: Daniel P. Berrangé <berrange@redhat.com>
>>> Cc: Igor Mammedov <imammedo@redhat.com>
>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>> Cc: Julia Suvorova <jusual@redhat.com>
>>> Cc: kraxel@redhat.com
>>> Signed-off-by: Ani Sinha <anisinha@redhat.com>
>>> ---
>>> hw/i386/pc_q35.c | 3 ++-
>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>> 
>>> Changelog:
>>> v3: bump up to 4096 vcpus. It has now been tested to work with edk2.
>>> See RH Jira: https://issues.redhat.com/browse/RHEL-22202
>> 
>> That bug indicates a dependancy on a EDK2 patch
>> 
>>  https://github.com/kraxel/edk2/commit/7a03c17f0f4f4a9003d77db2660c8e087604b2f0
>> 
>> we'll need to rebase the EDK2 ROMs in QEMU to get that included.
> 
> Which will btw take a while.  edk2 is in freeze for the 2024-02 release
> right now, I expect the changes land upstream shortly thereafter and
> will be part of the 2024-05 release.  So end may / early june would be
> the time when rebasing to release, or somewhen in march or april when we
> rebase to a git snapshot ...
> 
>> Meanwhile, plesae at least call out this EDK2 commit as a
>> pre-requisite in the commit message, so people know the
>> EDK2 ROMs in QEMU won't work (yet).
> 
> That surely makes sense.
> 
> Oh, and it's more than just that one commit.  I don't think it makes
> sense to compile a list of commits given this is a moving target
> (upstream review is in progress right now).

Looking at the edk2 GH, are these the PR that are waiting for upstream review/merge that relate to vcpu scaling?

https://github.com/tianocore/edk2/pull/5375
https://github.com/tianocore/edk2/pull/5327

Re: [PATCH v3] pc: q35: Bump max_cpus to 4096 vcpus

[Gerd Hoffmann]

  Hi,

> Looking at the edk2 GH, are these the PR that are waiting for upstream review/merge that relate to vcpu scaling?
> 
> https://github.com/tianocore/edk2/pull/5375
> https://github.com/tianocore/edk2/pull/5327

These are draft MRs for running CI.

The current devel branches are:
  https://github.com/kraxel/edk2/tree/devel/many-vcpus
  https://github.com/kraxel/edk2/tree/devel/many-vcpus-mpinitlib

All of them will expire at some point though, so I don't think it is a
good idea to include them in the commit message.  They will point into
nowhere in a year or so.

take care,
  Gerd