[PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Ani Sinha]

Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
enabled in the kernel. At present, QEMU has been tested to correctly boot a
linux guest with 1856 vcpus and no more both with edk2 and seabios firmwares.
If an additional vcpu is added, that is with 1857 vcpus, edk2 currently fails
with the following error messages:

AllocatePages failed: No 0x400 Pages is available.
There is only left 0x2BF pages memory resource to be allocated.
ERROR: Out of aligned pages
ASSERT /builddir/build/BUILD/edk2-ba91d0292e/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c(814): BigPageAddress != 0

This error exists only with edk2. Seabios currently can boot a linux guest
fine with 4096 vcpus. Since the lowest common denominator for a working VM for
both edk2 and seabios is 1856 vcpus, bump up the value max_cpus to 1856 for q35
machines versions 9 and newer. Q35 machines versions 8.2 and older continue
to support 1024 maximum vcpus as before for compatibility reasons.

If KVM is not able to support the specified number of vcpus, QEMU would
return the following error messages:

$ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)

Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Julia Suvorova <jusual@redhat.com>
Cc: kraxel@redhat.com
Signed-off-by: Ani Sinha <anisinha@redhat.com>
---
 hw/i386/pc_q35.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Changelog:
v2: bump up the vcpu number to 1856. Add failure messages from ekd2 in
the commit description.
See also RH Jira https://issues.redhat.com/browse/RHEL-22202

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index f43d5142b8..f9c4b6594d 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
     m->default_nic = "e1000e";
     m->default_kernel_irqchip_split = false;
     m->no_floppy = 1;
-    m->max_cpus = 1024;
+    m->max_cpus = 1856;
     m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
@@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
 {
     pc_q35_9_0_machine_options(m);
     m->alias = NULL;
+    m->max_cpus = 1024;
     compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
     compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
 }
-- 
2.42.0

Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Zhao Liu]

Hi Ani,

On Wed, Jan 31, 2024 at 08:19:06AM +0530, Ani Sinha wrote:
> Date: Wed, 31 Jan 2024 08:19:06 +0530
> From: Ani Sinha <anisinha@redhat.com>
> Subject: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus
> X-Mailer: git-send-email 2.42.0
> 
> Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> enabled in the kernel. At present, QEMU has been tested to correctly boot a
> linux guest with 1856 vcpus and no more both with edk2 and seabios firmwares.

About background, could I ask if there will be Host machines with so
much CPUs? What are the benefits of vCPUs that far exceed the number
of Host CPUs?

Thanks,
Zhao

> If an additional vcpu is added, that is with 1857 vcpus, edk2 currently fails
> with the following error messages:
> 
> AllocatePages failed: No 0x400 Pages is available.
> There is only left 0x2BF pages memory resource to be allocated.
> ERROR: Out of aligned pages
> ASSERT /builddir/build/BUILD/edk2-ba91d0292e/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c(814): BigPageAddress != 0
> 
> This error exists only with edk2. Seabios currently can boot a linux guest
> fine with 4096 vcpus. Since the lowest common denominator for a working VM for
> both edk2 and seabios is 1856 vcpus, bump up the value max_cpus to 1856 for q35
> machines versions 9 and newer. Q35 machines versions 8.2 and older continue
> to support 1024 maximum vcpus as before for compatibility reasons.
> 
> If KVM is not able to support the specified number of vcpus, QEMU would
> return the following error messages:
> 
> $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> 
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Julia Suvorova <jusual@redhat.com>
> Cc: kraxel@redhat.com
> Signed-off-by: Ani Sinha <anisinha@redhat.com>
> ---
>  hw/i386/pc_q35.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> Changelog:
> v2: bump up the vcpu number to 1856. Add failure messages from ekd2 in
> the commit description.
> See also RH Jira https://issues.redhat.com/browse/RHEL-22202
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index f43d5142b8..f9c4b6594d 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      m->default_nic = "e1000e";
>      m->default_kernel_irqchip_split = false;
>      m->no_floppy = 1;
> -    m->max_cpus = 1024;
> +    m->max_cpus = 1856;
>      m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> @@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
>  {
>      pc_q35_9_0_machine_options(m);
>      m->alias = NULL;
> +    m->max_cpus = 1024;
>      compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
>      compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
>  }
> -- 
> 2.42.0
> 
> 

Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Ani Sinha]

On Wed, Jan 31, 2024 at 9:27 AM Zhao Liu <zhao1.liu@intel.com> wrote:
>
> Hi Ani,
>
> On Wed, Jan 31, 2024 at 08:19:06AM +0530, Ani Sinha wrote:
> > Date: Wed, 31 Jan 2024 08:19:06 +0530
> > From: Ani Sinha <anisinha@redhat.com>
> > Subject: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus
> > X-Mailer: git-send-email 2.42.0
> >
> > Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> > Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> > enabled in the kernel. At present, QEMU has been tested to correctly boot a
> > linux guest with 1856 vcpus and no more both with edk2 and seabios firmwares.
>
> About background, could I ask if there will be Host machines with so
> much CPUs? What are the benefits of vCPUs that far exceed the number
> of Host CPUs?

Yes HPE has SAP HANA host machines with large numbers of physical
cores and memory. For example QEMU was tested on a system with 3840
cores.

>
> Thanks,
> Zhao
>
> > If an additional vcpu is added, that is with 1857 vcpus, edk2 currently fails
> > with the following error messages:
> >
> > AllocatePages failed: No 0x400 Pages is available.
> > There is only left 0x2BF pages memory resource to be allocated.
> > ERROR: Out of aligned pages
> > ASSERT /builddir/build/BUILD/edk2-ba91d0292e/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c(814): BigPageAddress != 0
> >
> > This error exists only with edk2. Seabios currently can boot a linux guest
> > fine with 4096 vcpus. Since the lowest common denominator for a working VM for
> > both edk2 and seabios is 1856 vcpus, bump up the value max_cpus to 1856 for q35
> > machines versions 9 and newer. Q35 machines versions 8.2 and older continue
> > to support 1024 maximum vcpus as before for compatibility reasons.
> >
> > If KVM is not able to support the specified number of vcpus, QEMU would
> > return the following error messages:
> >
> > $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728
> > qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> >
> > Cc: Daniel P. Berrangé <berrange@redhat.com>
> > Cc: Igor Mammedov <imammedo@redhat.com>
> > Cc: Michael S. Tsirkin <mst@redhat.com>
> > Cc: Julia Suvorova <jusual@redhat.com>
> > Cc: kraxel@redhat.com
> > Signed-off-by: Ani Sinha <anisinha@redhat.com>
> > ---
> >  hw/i386/pc_q35.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > Changelog:
> > v2: bump up the vcpu number to 1856. Add failure messages from ekd2 in
> > the commit description.
> > See also RH Jira https://issues.redhat.com/browse/RHEL-22202
> >
> > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > index f43d5142b8..f9c4b6594d 100644
> > --- a/hw/i386/pc_q35.c
> > +++ b/hw/i386/pc_q35.c
> > @@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
> >      m->default_nic = "e1000e";
> >      m->default_kernel_irqchip_split = false;
> >      m->no_floppy = 1;
> > -    m->max_cpus = 1024;
> > +    m->max_cpus = 1856;
> >      m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> > @@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
> >  {
> >      pc_q35_9_0_machine_options(m);
> >      m->alias = NULL;
> > +    m->max_cpus = 1024;
> >      compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
> >      compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
> >  }
> > --
> > 2.42.0
> >
> >
>

Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Zhao Liu]

On Wed, Jan 31, 2024 at 10:47:29AM +0530, Ani Sinha wrote:
> Date: Wed, 31 Jan 2024 10:47:29 +0530
> From: Ani Sinha <anisinha@redhat.com>
> Subject: Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus
> 
> On Wed, Jan 31, 2024 at 9:27 AM Zhao Liu <zhao1.liu@intel.com> wrote:
> >
> > Hi Ani,
> >
> > On Wed, Jan 31, 2024 at 08:19:06AM +0530, Ani Sinha wrote:
> > > Date: Wed, 31 Jan 2024 08:19:06 +0530
> > > From: Ani Sinha <anisinha@redhat.com>
> > > Subject: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus
> > > X-Mailer: git-send-email 2.42.0
> > >
> > > Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> > > Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> > > enabled in the kernel. At present, QEMU has been tested to correctly boot a
> > > linux guest with 1856 vcpus and no more both with edk2 and seabios firmwares.
> >
> > About background, could I ask if there will be Host machines with so
> > much CPUs? What are the benefits of vCPUs that far exceed the number
> > of Host CPUs?
> 
> Yes HPE has SAP HANA host machines with large numbers of physical
> cores and memory. For example QEMU was tested on a system with 3840
> cores.

Thanks! For such large system, does the vCPU need the CPU affinity, or
just let them run free on the Host's physical cores?

> 
> >
> > Thanks,
> > Zhao
> >
> > > If an additional vcpu is added, that is with 1857 vcpus, edk2 currently fails
> > > with the following error messages:
> > >
> > > AllocatePages failed: No 0x400 Pages is available.
> > > There is only left 0x2BF pages memory resource to be allocated.
> > > ERROR: Out of aligned pages
> > > ASSERT /builddir/build/BUILD/edk2-ba91d0292e/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c(814): BigPageAddress != 0
> > >
> > > This error exists only with edk2. Seabios currently can boot a linux guest
> > > fine with 4096 vcpus. Since the lowest common denominator for a working VM for
> > > both edk2 and seabios is 1856 vcpus, bump up the value max_cpus to 1856 for q35
> > > machines versions 9 and newer. Q35 machines versions 8.2 and older continue
> > > to support 1024 maximum vcpus as before for compatibility reasons.
> > >
> > > If KVM is not able to support the specified number of vcpus, QEMU would
> > > return the following error messages:
> > >
> > > $ ./qemu-system-x86_64 -cpu host -accel kvm -machine q35 -smp 1728

In practice, do users need to set the socket level topology and NUMA to
be consistent with Host for this large system?

NUMA settings are also related to topology, and it's better if NUMA is
also covered.

Thanks,
Zhao

> > > qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > > qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (1728) exceeds the recommended cpus supported by KVM (12)
> > > Number of SMP cpus requested (1728) exceeds the maximum cpus supported by KVM (1024)
> > >
> > > Cc: Daniel P. Berrangé <berrange@redhat.com>
> > > Cc: Igor Mammedov <imammedo@redhat.com>
> > > Cc: Michael S. Tsirkin <mst@redhat.com>
> > > Cc: Julia Suvorova <jusual@redhat.com>
> > > Cc: kraxel@redhat.com
> > > Signed-off-by: Ani Sinha <anisinha@redhat.com>
> > > ---
> > >  hw/i386/pc_q35.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > Changelog:
> > > v2: bump up the vcpu number to 1856. Add failure messages from ekd2 in
> > > the commit description.
> > > See also RH Jira https://issues.redhat.com/browse/RHEL-22202
> > >
> > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > index f43d5142b8..f9c4b6594d 100644
> > > --- a/hw/i386/pc_q35.c
> > > +++ b/hw/i386/pc_q35.c
> > > @@ -375,7 +375,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > >      m->default_nic = "e1000e";
> > >      m->default_kernel_irqchip_split = false;
> > >      m->no_floppy = 1;
> > > -    m->max_cpus = 1024;
> > > +    m->max_cpus = 1856;
> > >      m->no_parallel = !module_object_class_by_name(TYPE_ISA_PARALLEL);
> > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_AMD_IOMMU_DEVICE);
> > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> > > @@ -396,6 +396,7 @@ static void pc_q35_8_2_machine_options(MachineClass *m)
> > >  {
> > >      pc_q35_9_0_machine_options(m);
> > >      m->alias = NULL;
> > > +    m->max_cpus = 1024;
> > >      compat_props_add(m->compat_props, hw_compat_8_2, hw_compat_8_2_len);
> > >      compat_props_add(m->compat_props, pc_compat_8_2, pc_compat_8_2_len);
> > >  }
> > > --
> > > 2.42.0
> > >
> > >
> >
> 
> 

Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Daniel P. Berrangé]

On Wed, Jan 31, 2024 at 02:54:19PM +0800, Zhao Liu wrote:
> On Wed, Jan 31, 2024 at 10:47:29AM +0530, Ani Sinha wrote:
> > Date: Wed, 31 Jan 2024 10:47:29 +0530
> > From: Ani Sinha <anisinha@redhat.com>
> > Subject: Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus
> > 
> > On Wed, Jan 31, 2024 at 9:27 AM Zhao Liu <zhao1.liu@intel.com> wrote:
> > >
> > > Hi Ani,
> > >
> > > On Wed, Jan 31, 2024 at 08:19:06AM +0530, Ani Sinha wrote:
> > > > Date: Wed, 31 Jan 2024 08:19:06 +0530
> > > > From: Ani Sinha <anisinha@redhat.com>
> > > > Subject: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus
> > > > X-Mailer: git-send-email 2.42.0
> > > >
> > > > Since commit f10a570b093e6 ("KVM: x86: Add CONFIG_KVM_MAX_NR_VCPUS to allow up to 4096 vCPUs")
> > > > Linux kernel can support upto a maximum number of 4096 vCPUS when MAXSMP is
> > > > enabled in the kernel. At present, QEMU has been tested to correctly boot a
> > > > linux guest with 1856 vcpus and no more both with edk2 and seabios firmwares.
> > >
> > > About background, could I ask if there will be Host machines with so
> > > much CPUs? What are the benefits of vCPUs that far exceed the number
> > > of Host CPUs?
> > 
> > Yes HPE has SAP HANA host machines with large numbers of physical
> > cores and memory. For example QEMU was tested on a system with 3840
> > cores.
> 
> Thanks! For such large system, does the vCPU need the CPU affinity, or
> just let them run free on the Host's physical cores?

It depends what you are trying to achieve. The tradeoffs for guest
placement on small systems still pretty much apply on large systems
too. There may be factors which alter the balance for the tradeoffs,
but the theme is still determined by what the guest owner requires
and what the host owner wants to achieve.

Strict host:guest CPU affinity will give the guest a deterministic
amount of host CPU time, and lowest latencies. It is wasteful of
host resources though, because if a guest CPU is idle, a different
guests' CPU can't use that host CPU time. So letting guests run
freely across host cores and overcommiting CPUs will maximize
utilization but give non-deterministic timeslices & latency to
guests.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Gerd Hoffmann]

On Wed, Jan 31, 2024 at 08:19:06AM +0530, Ani Sinha wrote:
 
> This error exists only with edk2. Seabios currently can boot a linux guest
> fine with 4096 vcpus. Since the lowest common denominator for a working VM for
> both edk2 and seabios is 1856 vcpus, bump up the value max_cpus to 1856 for q35
> machines versions 9 and newer. Q35 machines versions 8.2 and older continue
> to support 1024 maximum vcpus as before for compatibility reasons.

There is a high chance this limit will be gone soon.

https://github.com/kraxel/edk2/commit/7a03c17f0f4f4a9003d77db2660c8e087604b2f0

If 4096 works fine with seabios, I'd suggest to go with 4096 as limit.

take care,
  Gerd

Re: [PATCH v2] pc: q35: Bump max_cpus to 1856 vcpus

[Ani Sinha]

> On 31-Jan-2024, at 17:24, Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> On Wed, Jan 31, 2024 at 08:19:06AM +0530, Ani Sinha wrote:
> 
>> This error exists only with edk2. Seabios currently can boot a linux guest
>> fine with 4096 vcpus. Since the lowest common denominator for a working VM for
>> both edk2 and seabios is 1856 vcpus, bump up the value max_cpus to 1856 for q35
>> machines versions 9 and newer. Q35 machines versions 8.2 and older continue
>> to support 1024 maximum vcpus as before for compatibility reasons.
> 
> There is a high chance this limit will be gone soon.
> 
> https://github.com/kraxel/edk2/commit/7a03c17f0f4f4a9003d77db2660c8e087604b2f0
> 
> If 4096 works fine with seabios, I'd suggest to go with 4096 as limit.

Any updates on this? Looking at https://issues.redhat.com/browse/RHEL-22202 , seems there is still some outstanding issues?