[Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Lan Tianyu]

Intel Xeon phi chip will support 352 logical threads. For HPC
usage case, it will create a huge VM with vcpus number as same as host
cpus. This patch is to increase max vcpu number to 352.

Signed-off-by: Lan Tianyu <tianyu.lan@intel.com>
---
 hw/i386/pc_q35.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 169a214..5e93749 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -299,7 +299,7 @@ static void pc_q35_machine_options(MachineClass *m)
     m->default_display = "std";
     m->no_floppy = 1;
     m->has_dynamic_sysbus = true;
-    m->max_cpus = 288;
+    m->max_cpus = 352;
 }
 
 static void pc_q35_2_10_machine_options(MachineClass *m)
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Konrad Rzeszutek Wilk]

On Wed, Aug 09, 2017 at 11:58:20PM -0400, Lan Tianyu wrote:
> Intel Xeon phi chip will support 352 logical threads. For HPC
> usage case, it will create a huge VM with vcpus number as same as host
> cpus. This patch is to increase max vcpu number to 352.

There is this ioctl called KVM_CAP_MAX_VCPUS. Why not use that?

Oh wait, this is generic code. Perhaps there is a way to make
this more abstract? Where this information is provided via
an function to which various accelerators register?

> 
> Signed-off-by: Lan Tianyu <tianyu.lan@intel.com>
> ---
>  hw/i386/pc_q35.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index 169a214..5e93749 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -299,7 +299,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      m->default_display = "std";
>      m->no_floppy = 1;
>      m->has_dynamic_sysbus = true;
> -    m->max_cpus = 288;
> +    m->max_cpus = 352;
>  }
>  
>  static void pc_q35_2_10_machine_options(MachineClass *m)
> -- 
> 1.8.3.1
> 
> 

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Eduardo Habkost]

On Thu, Aug 10, 2017 at 02:08:38PM -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 09, 2017 at 11:58:20PM -0400, Lan Tianyu wrote:
> > Intel Xeon phi chip will support 352 logical threads. For HPC
> > usage case, it will create a huge VM with vcpus number as same as host
> > cpus. This patch is to increase max vcpu number to 352.
> 
> There is this ioctl called KVM_CAP_MAX_VCPUS. Why not use that?
> 
> Oh wait, this is generic code. Perhaps there is a way to make
> this more abstract? Where this information is provided via
> an function to which various accelerators register?

We already check KVM_CAP_MAX_VCPUS, see kvm-all.c:kvm_init().
MachineClass::max_cpus is a limit defined at compile time, based
on the limitations of the existing QEMU machine/device code.

-- 
Eduardo

[Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Lan Tianyu]

Intel Xeon phi chip will support 352 logical threads. For HPC
usage case, it will create a huge VM with vcpus number as same as host
cpus. This patch is to increase max vcpu number to 352.

Signed-off-by: Lan Tianyu <tianyu.lan@intel.com>
---
 hw/i386/pc_q35.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 169a214..5e93749 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -299,7 +299,7 @@ static void pc_q35_machine_options(MachineClass *m)
     m->default_display = "std";
     m->no_floppy = 1;
     m->has_dynamic_sysbus = true;
-    m->max_cpus = 288;
+    m->max_cpus = 352;
 }
 
 static void pc_q35_2_10_machine_options(MachineClass *m)
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Daniel P. Berrange]

On Thu, Aug 10, 2017 at 06:08:07PM +0800, Lan Tianyu wrote:
> Intel Xeon phi chip will support 352 logical threads. For HPC
> usage case, it will create a huge VM with vcpus number as same as host
> cpus. This patch is to increase max vcpu number to 352.

If we pick arbitray limits based on size of physical CPUs that happen
to be shipping today, we'll continue the cat+mouse game forever trailing
latest CPUs that vendors ship.

IMHO we should pick a higher number influenced by technical constraints
of the q35 impl instead. eg can we go straight to something like 512 or
1024  ?

> Signed-off-by: Lan Tianyu <tianyu.lan@intel.com>
> ---
>  hw/i386/pc_q35.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index 169a214..5e93749 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -299,7 +299,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      m->default_display = "std";
>      m->no_floppy = 1;
>      m->has_dynamic_sysbus = true;
> -    m->max_cpus = 288;
> +    m->max_cpus = 352;
>  }

You'll need to introduce machine type back compat support so that we
avoid changing the 2.10 q35 machine type - only the 2.11 machine
type should have the new limit.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Lan Tianyu]

On 2017年08月10日 18:26, Daniel P. Berrange wrote:
> On Thu, Aug 10, 2017 at 06:08:07PM +0800, Lan Tianyu wrote:
>> Intel Xeon phi chip will support 352 logical threads. For HPC
>> usage case, it will create a huge VM with vcpus number as same as host
>> cpus. This patch is to increase max vcpu number to 352.
> 
> If we pick arbitray limits based on size of physical CPUs that happen
> to be shipping today, we'll continue the cat+mouse game forever trailing
> latest CPUs that vendors ship.
> 
> IMHO we should pick a higher number influenced by technical constraints
> of the q35 impl instead. eg can we go straight to something like 512 or
> 1024  ?

Sure. 512 should be enough and some arrays is defined according to max
vcpu number.

> 
>> Signed-off-by: Lan Tianyu <tianyu.lan@intel.com>
>> ---
>>  hw/i386/pc_q35.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
>> index 169a214..5e93749 100644
>> --- a/hw/i386/pc_q35.c
>> +++ b/hw/i386/pc_q35.c
>> @@ -299,7 +299,7 @@ static void pc_q35_machine_options(MachineClass *m)
>>      m->default_display = "std";
>>      m->no_floppy = 1;
>>      m->has_dynamic_sysbus = true;
>> -    m->max_cpus = 288;
>> +    m->max_cpus = 352;
>>  }
> 
> You'll need to introduce machine type back compat support so that we
> avoid changing the 2.10 q35 machine type - only the 2.11 machine
> type should have the new limit.

How about the following change?

-static void pc_q35_2_10_machine_options(MachineClass *m)
+static void pc_q35_2_11_machine_options(MachineClass *m)
 {
      pc_q35_machine_options(m);
      m->alias = "q35";
+}
+
+static void pc_q35_2_10_machine_options(MachineClass *m)
+{
+    pc_q35_2_11_options(m);
+    m->alias = "q35";
+    m->max_cpus = 288;
     m->numa_auto_assign_ram = numa_legacy_auto_assign_ram;
 }



> 
> Regards,
> Daniel
> 


-- 
Best regards
Tianyu Lan

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Radim Krčmář]

2017-08-10 19:02+0800, Lan Tianyu:
> On 2017年08月10日 18:26, Daniel P. Berrange wrote:
>> On Thu, Aug 10, 2017 at 06:08:07PM +0800, Lan Tianyu wrote:
>>> Intel Xeon phi chip will support 352 logical threads. For HPC
>>> usage case, it will create a huge VM with vcpus number as same as host
>>> cpus. This patch is to increase max vcpu number to 352.
>> 
>> If we pick arbitray limits based on size of physical CPUs that happen
>> to be shipping today, we'll continue the cat+mouse game forever trailing
>> latest CPUs that vendors ship.
>> 
>> IMHO we should pick a higher number influenced by technical constraints
>> of the q35 impl instead. eg can we go straight to something like 512 or
>> 1024  ?
> 
> Sure. 512 should be enough and some arrays is defined according to max
> vcpu number.

Hm, which arrays are that?  I was thinking it is safe to bump it to
INT_MAX as the number is only used when setting global max_cpus.

Thanks.

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Eduardo Habkost]

On Thu, Aug 10, 2017 at 02:41:03PM +0200, Radim Krčmář wrote:
> 2017-08-10 19:02+0800, Lan Tianyu:
> > On 2017年08月10日 18:26, Daniel P. Berrange wrote:
> >> On Thu, Aug 10, 2017 at 06:08:07PM +0800, Lan Tianyu wrote:
> >>> Intel Xeon phi chip will support 352 logical threads. For HPC
> >>> usage case, it will create a huge VM with vcpus number as same as host
> >>> cpus. This patch is to increase max vcpu number to 352.
> >> 
> >> If we pick arbitray limits based on size of physical CPUs that happen
> >> to be shipping today, we'll continue the cat+mouse game forever trailing
> >> latest CPUs that vendors ship.
> >> 
> >> IMHO we should pick a higher number influenced by technical constraints
> >> of the q35 impl instead. eg can we go straight to something like 512 or
> >> 1024  ?
> > 
> > Sure. 512 should be enough and some arrays is defined according to max
> > vcpu number.
> 
> Hm, which arrays are that?  I was thinking it is safe to bump it to
> INT_MAX as the number is only used when setting global max_cpus.

We had a MAX_CPUMASK_BITS macro, and bitmaps whose sizes were
defined at compile time based on it.  But commit
cdda2018e3b9ce0c18938767dfdb1e05a05b67ca removed it.  Probably
those arrays all use max_cpus, by now (and the default for
max_cpus is smp_cpus, not MachineClass::max_cpus).

Anyway, if we set it to INT_MAX, there are some cases where more
appropriate error checking/reporting could be required because
they won't handle overflow very well:
* pcms->apic_id_limit initialization at pc_cpus_init()
* ACPI code that assumes possible_cpus->cpus[i].arch_id fits
  in a 32-bit integer
* Other x86_cpu_apic_id_from_index() calls in PC code
  (especially the initialization of possible_cpus->cpus[i].arch_id).
  Note that x86_cpu_apic_id_from_index(cpu_index) might not fit
  in 32 bits even if cpu_index <= UINT32_MAX.

-- 
Eduardo

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Radim Krčmář]

2017-08-10 15:16-0300, Eduardo Habkost:
> On Thu, Aug 10, 2017 at 02:41:03PM +0200, Radim Krčmář wrote:
> > 2017-08-10 19:02+0800, Lan Tianyu:
> > > On 2017年08月10日 18:26, Daniel P. Berrange wrote:
> > >> On Thu, Aug 10, 2017 at 06:08:07PM +0800, Lan Tianyu wrote:
> > >>> Intel Xeon phi chip will support 352 logical threads. For HPC
> > >>> usage case, it will create a huge VM with vcpus number as same as host
> > >>> cpus. This patch is to increase max vcpu number to 352.
> > >> 
> > >> If we pick arbitray limits based on size of physical CPUs that happen
> > >> to be shipping today, we'll continue the cat+mouse game forever trailing
> > >> latest CPUs that vendors ship.
> > >> 
> > >> IMHO we should pick a higher number influenced by technical constraints
> > >> of the q35 impl instead. eg can we go straight to something like 512 or
> > >> 1024  ?
> > > 
> > > Sure. 512 should be enough and some arrays is defined according to max
> > > vcpu number.
> > 
> > Hm, which arrays are that?  I was thinking it is safe to bump it to
> > INT_MAX as the number is only used when setting global max_cpus.
> 
> We had a MAX_CPUMASK_BITS macro, and bitmaps whose sizes were
> defined at compile time based on it.  But commit
> cdda2018e3b9ce0c18938767dfdb1e05a05b67ca removed it.  Probably
> those arrays all use max_cpus, by now (and the default for
> max_cpus is smp_cpus, not MachineClass::max_cpus).

Ah, thanks.

> Anyway, if we set it to INT_MAX, there are some cases where more
> appropriate error checking/reporting could be required because
> they won't handle overflow very well:
> * pcms->apic_id_limit initialization at pc_cpus_init()
> * ACPI code that assumes possible_cpus->cpus[i].arch_id fits
>   in a 32-bit integer
> * Other x86_cpu_apic_id_from_index() calls in PC code
>   (especially the initialization of possible_cpus->cpus[i].arch_id).
>   Note that x86_cpu_apic_id_from_index(cpu_index) might not fit
>   in 32 bits even if cpu_index <= UINT32_MAX.

Good point, looks like it all comes to x86_cpu_apic_id_from_index().
Each level of the topology has at most one underutilized bit, so
2^(32 - 3) would be safe.

It is still needlessly large for the foreseeable future, but 512 is
going to be surpassed pretty soon, so I think that jumping at least to
8k would be better.
(8k the current default maximum for Linux and the resulting overcommit
 of ~20 is bearable for smoke testing on current hardware.)

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Lan Tianyu]

On 2017年08月11日 03:22, Radim Krčmář wrote:
> 2017-08-10 15:16-0300, Eduardo Habkost:
>> On Thu, Aug 10, 2017 at 02:41:03PM +0200, Radim Krčmář wrote:
>>> 2017-08-10 19:02+0800, Lan Tianyu:
>>>> On 2017年08月10日 18:26, Daniel P. Berrange wrote:
>>>>> On Thu, Aug 10, 2017 at 06:08:07PM +0800, Lan Tianyu wrote:
>>>>>> Intel Xeon phi chip will support 352 logical threads. For HPC
>>>>>> usage case, it will create a huge VM with vcpus number as same as host
>>>>>> cpus. This patch is to increase max vcpu number to 352.
>>>>>
>>>>> If we pick arbitray limits based on size of physical CPUs that happen
>>>>> to be shipping today, we'll continue the cat+mouse game forever trailing
>>>>> latest CPUs that vendors ship.
>>>>>
>>>>> IMHO we should pick a higher number influenced by technical constraints
>>>>> of the q35 impl instead. eg can we go straight to something like 512 or
>>>>> 1024  ?
>>>>
>>>> Sure. 512 should be enough and some arrays is defined according to max
>>>> vcpu number.
>>>
>>> Hm, which arrays are that?  I was thinking it is safe to bump it to
>>> INT_MAX as the number is only used when setting global max_cpus.
>>
>> We had a MAX_CPUMASK_BITS macro, and bitmaps whose sizes were
>> defined at compile time based on it.  But commit
>> cdda2018e3b9ce0c18938767dfdb1e05a05b67ca removed it.  Probably
>> those arrays all use max_cpus, by now (and the default for
>> max_cpus is smp_cpus, not MachineClass::max_cpus).
> 
> Ah, thanks.
> 
>> Anyway, if we set it to INT_MAX, there are some cases where more
>> appropriate error checking/reporting could be required because
>> they won't handle overflow very well:
>> * pcms->apic_id_limit initialization at pc_cpus_init()
>> * ACPI code that assumes possible_cpus->cpus[i].arch_id fits
>>   in a 32-bit integer
>> * Other x86_cpu_apic_id_from_index() calls in PC code
>>   (especially the initialization of possible_cpus->cpus[i].arch_id).
>>   Note that x86_cpu_apic_id_from_index(cpu_index) might not fit
>>   in 32 bits even if cpu_index <= UINT32_MAX.
> 
> Good point, looks like it all comes to x86_cpu_apic_id_from_index().
> Each level of the topology has at most one underutilized bit, so
> 2^(32 - 3) would be safe.
> 
> It is still needlessly large for the foreseeable future, but 512 is
> going to be surpassed pretty soon, so I think that jumping at least to
> 8k would be better.
> (8k the current default maximum for Linux and the resulting overcommit
>  of ~20 is bearable for smoke testing on current hardware.)
> 

Hi All:
	Thanks for your input. I tried Qemu with 8192 as max_vcpu and it works
normally. I will update patches.

-- 
Best regards
Tianyu Lan

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Paolo Bonzini]

On 10/08/2017 12:08, Lan Tianyu wrote:
> Intel Xeon phi chip will support 352 logical threads. For HPC
> usage case, it will create a huge VM with vcpus number as same as host
> cpus. This patch is to increase max vcpu number to 352.

Even without Xeon Phi, a 24-core, 8-socket system (E7-8890 v4 for
example) will have 384 physical CPUs.  288 was the biggest we could test
at the time, but now that the 256 vCPU barrier has been broken I agree
with going straight to 512.

Paolo

Re: [Qemu-devel] [PATCH] x86: Increase max vcpu number to 352

[Daniel P. Berrange]

On Thu, Aug 10, 2017 at 02:13:30PM +0200, Paolo Bonzini wrote:
> On 10/08/2017 12:08, Lan Tianyu wrote:
> > Intel Xeon phi chip will support 352 logical threads. For HPC
> > usage case, it will create a huge VM with vcpus number as same as host
> > cpus. This patch is to increase max vcpu number to 352.
> 
> Even without Xeon Phi, a 24-core, 8-socket system (E7-8890 v4 for
> example) will have 384 physical CPUs.  288 was the biggest we could test
> at the time, but now that the 256 vCPU barrier has been broken I agree
> with going straight to 512.

FWIW, linux allows  8192 CPUs on x86_64 already, due to alledged
existance of SGI machines with 6096 CPUs....

  commit b53b5eda8194214928c8243d711a75dbf51809fc
  Author: Josh Boyer <jwboyer@redhat.com>
  Date:   Tue Nov 5 09:38:16 2013 -0500

    x86/cpu: Increase max CPU count to 8192
    
    The MAXSMP option is intended to enable silly large numbers of
    CPUs for testing purposes.  The current value of 4096 isn't very
    silly any longer as there are actual SGI machines that approach
    6096 CPUs when taking HT into account.
    
    Increase the value to a nice round 8192 to account for this and
    allow for short term future increases.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|