Re: [Qemu-devel] [PATCH 0/8] [PATCH RFC v3] s390 cpu hotplug

[Christian Borntraeger <borntraeger@de.ibm.com>]

On 04/09/13 14:45, Andreas Färber wrote:
> Hello,
> 
> Am 01.08.2013 16:12, schrieb Jason J. Herne:
>> From: "Jason J. Herne" <jjherne@us.ibm.com>
>>
>> Latest code for cpu Hotplug on S390 architecture.   This one is vastly simpler
>> than v2 as we have decided to avoid the command line specification 
>> of -device s390-cpu.
>>
>> The last version can be found here:
>> http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg01183.html
>>
>> There is also a patch in this series to add cpu-add to the Qemu monitor
>> interface.
>>
>> Hotplugged cpus are created in the configured state and can be used by the
>> guest after the guest onlines the cpu by: 
>> "echo 1 > /sys/bus/cpu/devices/cpuN/online"
>>
>> Hot unplugging is currently not implemented by this code. 
> 
> We have been having several off-list discussions since then that I'll
> try to briefly summarize here, please correct or extend as needed:
> 
> 1) CPU topology for QOM
> 
> Physically a System z machine may have an MCM with, e.g., 6 CPUs with 6
> cores each. But unlike x86, there is PR/SM, LPAR and possibly z/VM in
> between Linux and hardware, so we do actually want to be able to
> hot-plug in quantities of 1 and not by 6 on s390x for the foreseeable
> future. We seem willing to set a QOM ABI in stone based on that assumption.

Just stepping in, Jason is on vacation this week.

To summarize my understanding:
You were thinking if CPU model needs topology (e.g. -device mcm,id=m1, -device cpu,mcm=m1)
and s390 was the only arch left, that you were not sure about if topology is needed? 
All other platforms dont need topology for cpu hotplug?

Yes, we want to be able to hotplug single cores (not chips, not MCMs). 
It is pretty hard to pin the vCPUs to a given real topology for KVM. You need to
pin on LPAR and KVM. Libvirt could  do some pinning of guest vCPUs to host CPUs and
LPAR can have dedicated CPUs. But pinning a full chip (6cores) would only make
sense in very rare cases.


> 
> => s390-cpu (or future subtypes) to be used with device_add.
> => Flat /machine/cpu[n] list in composition tree a possibility.
> 
> 1a) CPU topology for guests
> 
> STSI instruction topology support not implemented yet.

Right not implemented yet, but we certainly want to be able to define the guest
visible topology at some point in time (grouping of cores basically). 
But I guess this does not mean that we have to go away from the flat list of CPUs.


> 
> => Guest unaware of any emulated topology today.

An additional problem is, that for the normal case (linux scheduler, no pinning, also
no gang scheduling) the topology would change too fast. The guest would be busy rebuilding
the scheduler domains all the time.

> 
> hyptop tool requires hypfs implementation for KVM.
> 
> => Guest unaware of sibling VMs today, unlike z/VM and LPAR.

Right. But we will look into hyptop at some point in time. 

> 
> 2) CPU hot-unplug
> 
> Hotplug will always use a unique linear CPU address, even if hot-unplug
> leads to a sparse address space.
> 
> => cpu_num != cpu_index
> 
> 
> With all that in mind, I'll now need to review the s390 patches again.
> 
> For the HMP patch I am waiting on feedback from Igor once he returns
> from his vacation and, if there are no objections, would like to see
> that patch go through Luiz' queue since unrelated to s390x.

Yes, would be cool if the HMP patch could go upstream via that path. That would 
reduce the rework/patch burden of Jason.

> 
> Regards,
> Andreas
> 
>>
>> Jason J. Herne (8):
>>   s390-qemu: cpu hotplug - Define New SCLP Codes
>>   s390-qemu: cpu hotplug - SCLP CPU Info
>>   s390-qemu: cpu hotplug - SCLP Event integration
>>   s390-qemu: cpu hotplug - Storage key global access
>>   s390-qemu: cpu hotplug - ipi_states enhancements
>>   s390-qemu: cpu hotplug - s390 cpu init improvements for hotplug
>>   s390-qemu: cpu hotplug - Implement hot_add_cpu hook
>>   qemu-monitor: HMP cpu-add wrapper
>>
>>  hmp-commands.hx                   |   13 ++++
>>  hmp.c                             |   10 ++++
>>  hmp.h                             |    1 +
>>  hw/s390x/Makefile.objs            |    2 +-
>>  hw/s390x/event-facility.c         |    7 +++
>>  hw/s390x/s390-virtio-ccw.c        |    8 ++-
>>  hw/s390x/s390-virtio.c            |   47 +++++++++------
>>  hw/s390x/s390-virtio.h            |    2 +-
>>  hw/s390x/sclp.c                   |   53 +++++++++++++++-
>>  hw/s390x/sclpcpu.c                |  120 +++++++++++++++++++++++++++++++++++++
>>  include/hw/s390x/event-facility.h |    3 +
>>  include/hw/s390x/sclp.h           |   41 +++++++++++++
>>  target-s390x/cpu.c                |   36 ++++++++++-
>>  target-s390x/cpu.h                |    7 +++
>>  target-s390x/helper.c             |   12 ++++
>>  15 files changed, 336 insertions(+), 26 deletions(-)
>>  create mode 100644 hw/s390x/sclpcpu.c
>>
> 
>