Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Jason J. Herne]

I've done some investigating into using the device_add hmp/qmp command 
to support hot-plugging cpus on S390.  The alternative suggestion was to 
simply use a new cpu_add hmp/qmp command.

device_add accepts all of the same options as the -device command line 
parameter takes.  This would imply that to hot-plug cpu's using device 
add we would need to allow command line arguments of type "-device cpu". 
  All of the implications of this are not currently clear to me.  How 
would this interact with the -smp option, for example, how many cpus are 
created in this case:
     qemu -smp 2 -device cpu,id=cpu0 -device cpu,id=cpu1, -device
cpu,id=cpu2
Is -smp invalid when cpu devices are specified?  We would have to fill 
the smp_cpus variable after all (cpu) devices have been parsed.

Since device_add requires a QOM object name (driver parameter) we
seem to have
two choices.
     1. device_add cpu
     2. device_add s390-cpu
But "cpu" is actually an abstract QOM class and cannot be instantiated 
by object_new("cpu") as is done in device_add processing.  So we need to 
use "s390-cpu".  This adds an architecture specific flavor to cpu
hotplug. I would think we'd want to avoid that somehow.  perhaps we 
simply "translate" that parameter during early device_add processing?

Another issue is that the s390-cpu QOM object class is a child of 
"main-system-bus".  This bus does not support hotplug: 
sysbus->allow_hotplug=0. In order to implement cpu hotplug we would need 
to either switch sysbus->allow_hotplug to 1, or the s390-cpu QOM object 
class would need to move to a bus that supports hotplug.  I'm not sure 
what the implications of either choice would be.

I'm interested in thoughts and comments.  Thanks!

-- 
-- Jason J. Herne (jjherne@linux.vnet.ibm.com)

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Igor Mammedov]

On Fri, 03 May 2013 09:50:50 -0400
"Jason J. Herne" <jjherne@linux.vnet.ibm.com> wrote:

> I've done some investigating into using the device_add hmp/qmp command 
> to support hot-plugging cpus on S390.  The alternative suggestion was to 
> simply use a new cpu_add hmp/qmp command.
> 
> device_add accepts all of the same options as the -device command line 
> parameter takes.  This would imply that to hot-plug cpu's using device 
> add we would need to allow command line arguments of type "-device cpu". 
>   All of the implications of this are not currently clear to me.  How 
> would this interact with the -smp option, for example, how many cpus are 
> created in this case:
>      qemu -smp 2 -device cpu,id=cpu0 -device cpu,id=cpu1, -device
> cpu,id=cpu2
> Is -smp invalid when cpu devices are specified?  We would have to fill 
> the smp_cpus variable after all (cpu) devices have been parsed.
I haven't looked using -device in case of x86 it would require some extra
work besides planned properties+classes to make it usable.
However device_add for your case should be usable and there is no need to use
-device with it. 
-smp x,maxcpus=y will tell board how many CPUs 'x' should be created and
'y' - 'x' how many extra CPUs could be added later.

> 
> Since device_add requires a QOM object name (driver parameter) we
> seem to have
> two choices.
>      1. device_add cpu
>      2. device_add s390-cpu
> But "cpu" is actually an abstract QOM class and cannot be instantiated 
> by object_new("cpu") as is done in device_add processing.  So we need to 
> use "s390-cpu".  This adds an architecture specific flavor to cpu
> hotplug. I would think we'd want to avoid that somehow.  perhaps we 
> simply "translate" that parameter during early device_add processing?
that is exactly intention to use architecture specific types there so that
one could specify what kind/flavor of CPU needs to be added using type names.

> Another issue is that the s390-cpu QOM object class is a child of 
> "main-system-bus".  This bus does not support hotplug: 
> sysbus->allow_hotplug=0. In order to implement cpu hotplug we would need 
> to either switch sysbus->allow_hotplug to 1, or the s390-cpu QOM object 
> class would need to move to a bus that supports hotplug.  I'm not sure 
> what the implications of either choice would be.
For x86 CPU was moved to ICC bus that existed in real hardware, probably there
is similar bus for s390. Alternatively you can consider if s390-cpu could be
bus-less, since commit 2f7bd829db it's possible device_add bus-less devices.

> I'm interested in thoughts and comments.  Thanks!
> 
> -- 
> -- Jason J. Herne (jjherne@linux.vnet.ibm.com)
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Andreas Färber]

Hi,

Am 03.05.2013 15:50, schrieb Jason J. Herne:
> I've done some investigating into using the device_add hmp/qmp command
> to support hot-plugging cpus on S390.  The alternative suggestion was to
> simply use a new cpu_add hmp/qmp command.

A cpu-add QMP command has been merged by now. Using it with
qemu-system-s390x machines will return a QMP error at the moment.

> device_add accepts all of the same options as the -device command line
> parameter takes.  This would imply that to hot-plug cpu's using device
> add we would need to allow command line arguments of type "-device cpu".

In theory we do, ever since making the CPU a device, it just didn't
fully work yet. For all QOM'ified CPUs (i.e., not x86) it should work
crash-free now, but it's untested whether a particular machine copes
with it or not.

>  All of the implications of this are not currently clear to me.  How
> would this interact with the -smp option, for example, how many cpus are
> created in this case:
>     qemu -smp 2 -device cpu,id=cpu0 -device cpu,id=cpu1, -device
> cpu,id=cpu2

Four, if the correct driver is supplied (error for the above).
The -smp option indicates how many CPUs the *machine* instantiates.
In addition you are trying to create two further devices, just like
other machines create a PCI host bridge and a user might try to add
another one.

> Is -smp invalid when cpu devices are specified?  We would have to fill
> the smp_cpus variable after all (cpu) devices have been parsed.

Would we? If so, doing some check of -smp maxcpus and/or updating
whatever variable in CPU's realizefn feels more natural to me than some
post-whatever hook.

> Since device_add requires a QOM object name (driver parameter) we
> seem to have
> two choices.
>     1. device_add cpu
>     2. device_add s390-cpu
> But "cpu" is actually an abstract QOM class and cannot be instantiated
> by object_new("cpu") as is done in device_add processing.  So we need to
> use "s390-cpu".  This adds an architecture specific flavor to cpu
> hotplug. I would think we'd want to avoid that somehow.  perhaps we
> simply "translate" that parameter during early device_add processing?

You are saying that based on the current s390 code. Actually it was
discussed that s390-cpu should be abstract as well and the type should
indicate the actual model - host-s390-cpu, z9-s390-cpu, etc. There were
two KVM calls that covered future structure of CPU modelling (socket ->
core -> thread) and roadmap towards vCPU hotplug - see the minutes on
the list.

The current approach of cpu-add for 1.5 was chosen because the
refactoring of CPUArchState is rather cumbersome and taking too long.

> Another issue is that the s390-cpu QOM object class is a child of
> "main-system-bus". [...]

That's not true, it is not on any bus at all - I have attempted to fix
device_add for this use case and Igor has just sent a patch for unplug.

For x86 we have chosen to introduce the ICC bus to handle hot-adding
APIC devices (which were on SysBus before) alongside the CPU. With
proper CPU modelling that would not be necessary, but for now it has the
advantage of giving us a canonical QOM path to the CPUs for free.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

[Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Jens Freimann]

Hi all,

this is what our approach to CPU hotplug looks like.
With respect to Igor's CPU hotplug series, how should we proceed? 
Should we change the interface to 
qemu_system_cpu_add_notifier/qemu_system_cpu_hotplug_request/cpu-add etc?

Feedback regarding the non-API part is also highly welcome!

regards
Jens

Thang Pham (1):
  s390: implement CPU hotplug

 hmp-commands.hx           |  14 +++++
 hw/s390x/Makefile.objs    |   2 +-
 hw/s390x/event-facility.c |   9 ++++
 hw/s390x/event-facility.h |   3 ++
 hw/s390x/s390-virtio.c    |  34 +++++++++---
 hw/s390x/sclp.c           | 134 +++++++++++++++++++++++++++++++++++++++++++++-
 hw/s390x/sclp.h           |  47 +++++++++++++++-
 include/sysemu/sysemu.h   |   1 +
 monitor.c                 |  31 +++++++++++
 target-s390x/cpu.c        |  59 ++++++++++++++++++++
 target-s390x/cpu.h        |   7 +++
 target-s390x/helper.c     |  43 +++++++++++++++
 vl.c                      |   6 +++
 13 files changed, 379 insertions(+), 11 deletions(-)

-- 
1.7.12.4

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Andreas Färber]

Hi Jens,

Am 03.04.2013 08:42, schrieb Jens Freimann:
> this is what our approach to CPU hotplug looks like.
> With respect to Igor's CPU hotplug series, how should we proceed? 
> Should we change the interface to 
> qemu_system_cpu_add_notifier/qemu_system_cpu_hotplug_request/cpu-add etc?

I am wondering if my recent qdev/device_add fixes would allow to
implement CPU hot-add via device_add for s390x?

Background is that for x86 we currently have a flat CPU core/thread
namespace but would need to deal with sockets, cores and threads to get
topologies right. I assume there are no such issues on s390x, so that
the vCPU to CPUState mapping could stay 1:1?

> Feedback regarding the non-API part is also highly welcome!

I did spot some QOM'ish nitpicks in the qdev_create() but I am wondering
if you and Alex see an urgent need to get this into 1.5 during the Soft
Freeze or whether we can just align it to x86 work for now and deal with
it after the release?

Regards,
Andreas

> Thang Pham (1):
>   s390: implement CPU hotplug
> 
>  hmp-commands.hx           |  14 +++++
>  hw/s390x/Makefile.objs    |   2 +-
>  hw/s390x/event-facility.c |   9 ++++
>  hw/s390x/event-facility.h |   3 ++
>  hw/s390x/s390-virtio.c    |  34 +++++++++---
>  hw/s390x/sclp.c           | 134 +++++++++++++++++++++++++++++++++++++++++++++-
>  hw/s390x/sclp.h           |  47 +++++++++++++++-
>  include/sysemu/sysemu.h   |   1 +
>  monitor.c                 |  31 +++++++++++
>  target-s390x/cpu.c        |  59 ++++++++++++++++++++
>  target-s390x/cpu.h        |   7 +++
>  target-s390x/helper.c     |  43 +++++++++++++++
>  vl.c                      |   6 +++
>  13 files changed, 379 insertions(+), 11 deletions(-)
> 


-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Eduardo Habkost]

On Wed, Apr 17, 2013 at 08:06:37PM +0200, Andreas Färber wrote:
> Hi Jens,
> 
> Am 03.04.2013 08:42, schrieb Jens Freimann:
> > this is what our approach to CPU hotplug looks like.
> > With respect to Igor's CPU hotplug series, how should we proceed? 
> > Should we change the interface to 
> > qemu_system_cpu_add_notifier/qemu_system_cpu_hotplug_request/cpu-add etc?
> 
> I am wondering if my recent qdev/device_add fixes would allow to
> implement CPU hot-add via device_add for s390x?
> 
> Background is that for x86 we currently have a flat CPU core/thread
> namespace but would need to deal with sockets, cores and threads to get
> topologies right. I assume there are no such issues on s390x, so that
> the vCPU to CPUState mapping could stay 1:1?

Also: x86 has a huge number of CPU models we want to convert to separate
classes, and a huge number of configurable CPU features we want to
expose as device properties.

If we don't need any of that on s390x, I agree that it may be reasonable
to try to make device_add work.

-- 
Eduardo

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Jens Freimann]

On Wed, Apr 17, 2013 at 08:06:37PM +0200, Andreas Färber wrote:
> Hi Jens,
> 
> Am 03.04.2013 08:42, schrieb Jens Freimann:
> > this is what our approach to CPU hotplug looks like.
> > With respect to Igor's CPU hotplug series, how should we proceed? 
> > Should we change the interface to 
> > qemu_system_cpu_add_notifier/qemu_system_cpu_hotplug_request/cpu-add etc?
> 
> I am wondering if my recent qdev/device_add fixes would allow to
> implement CPU hot-add via device_add for s390x?

>From what I've seen so far it could be possible, but...
 
> Background is that for x86 we currently have a flat CPU core/thread
> namespace but would need to deal with sockets, cores and threads to get
> topologies right. I assume there are no such issues on s390x, so that
> the vCPU to CPUState mapping could stay 1:1?

s390 hardware today already has a topology and CPU features.  We are not 
modelling it in QEMU right now, but want to go there in the future so that 
there would be no simple 1:1 mapping anymore.
 
> > Feedback regarding the non-API part is also highly welcome!
> 
> I did spot some QOM'ish nitpicks in the qdev_create() but I am wondering
> if you and Alex see an urgent need to get this into 1.5 during the Soft
> Freeze or whether we can just align it to x86 work for now and deal with
> it after the release?

We'll try to have it ready as soon as possible but I think it's unlikely
for 1.5. Getting started with Igor's review feedback right now and looking
at qdev_create() as well. 


Regards,
Jens
 
> Regards,
> Andreas
> 
> > Thang Pham (1):
> >   s390: implement CPU hotplug
> > 
> >  hmp-commands.hx           |  14 +++++
> >  hw/s390x/Makefile.objs    |   2 +-
> >  hw/s390x/event-facility.c |   9 ++++
> >  hw/s390x/event-facility.h |   3 ++
> >  hw/s390x/s390-virtio.c    |  34 +++++++++---
> >  hw/s390x/sclp.c           | 134 +++++++++++++++++++++++++++++++++++++++++++++-
> >  hw/s390x/sclp.h           |  47 +++++++++++++++-
> >  include/sysemu/sysemu.h   |   1 +
> >  monitor.c                 |  31 +++++++++++
> >  target-s390x/cpu.c        |  59 ++++++++++++++++++++
> >  target-s390x/cpu.h        |   7 +++
> >  target-s390x/helper.c     |  43 +++++++++++++++
> >  vl.c                      |   6 +++
> >  13 files changed, 379 insertions(+), 11 deletions(-)
> > 
> 
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
> 

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Andreas Färber]

Am 19.04.2013 09:51, schrieb Jens Freimann:
> On Wed, Apr 17, 2013 at 08:06:37PM +0200, Andreas Färber wrote:
>> Hi Jens,
>>
>> Am 03.04.2013 08:42, schrieb Jens Freimann:
>>> this is what our approach to CPU hotplug looks like.
>>> With respect to Igor's CPU hotplug series, how should we proceed? 
>>> Should we change the interface to 
>>> qemu_system_cpu_add_notifier/qemu_system_cpu_hotplug_request/cpu-add etc?
>>
>> I am wondering if my recent qdev/device_add fixes would allow to
>> implement CPU hot-add via device_add for s390x?
> 
> From what I've seen so far it could be possible, but...

Hm, so probably not a good idea? Just looking for a guinea pig for
infrastructure testing...

>> Background is that for x86 we currently have a flat CPU core/thread
>> namespace but would need to deal with sockets, cores and threads to get
>> topologies right. I assume there are no such issues on s390x, so that
>> the vCPU to CPUState mapping could stay 1:1?
> 
> s390 hardware today already has a topology and CPU features.  We are not 
> modelling it in QEMU right now, but want to go there in the future so that 
> there would be no simple 1:1 mapping anymore.

In that case please enlighten us how the topology model should/could
look like in the future, so that we can align this with the x86
remodelling and APIC/ID discussion.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Igor Mammedov]

On Fri, 19 Apr 2013 15:16:36 +0200
Andreas Färber <afaerber@suse.de> wrote:

> Am 19.04.2013 09:51, schrieb Jens Freimann:
> > On Wed, Apr 17, 2013 at 08:06:37PM +0200, Andreas Färber wrote:
> >> Hi Jens,
> >>
> >> Am 03.04.2013 08:42, schrieb Jens Freimann:
> >>> this is what our approach to CPU hotplug looks like.
> >>> With respect to Igor's CPU hotplug series, how should we proceed? 
> >>> Should we change the interface to 
> >>> qemu_system_cpu_add_notifier/qemu_system_cpu_hotplug_request/cpu-add
> >>> etc?
> >>
> >> I am wondering if my recent qdev/device_add fixes would allow to
> >> implement CPU hot-add via device_add for s390x?
> > 
> > From what I've seen so far it could be possible, but...
> 
> Hm, so probably not a good idea? Just looking for a guinea pig for
> infrastructure testing...
Well it looks like good candidate for this, from what I saw though my
cursory review was that so far s390:
 1. it has only one CPU model, and 
 2. cpu_model string isn't at all
 3. no features are set externally (at least explicitly during CPU creation)

Once patches 1-8/16 from v4 cpu-add are in + Andreas patch for bussless
devices in device_add, generic infrastructure for device_add will be in place.
So new respin might be device_add based.

Perhaps there are missing pieces yet, but it would be easier to spot them
once trying implement stuff this way.

> 
> >> Background is that for x86 we currently have a flat CPU core/thread
> >> namespace but would need to deal with sockets, cores and threads to get
> >> topologies right. I assume there are no such issues on s390x, so that
> >> the vCPU to CPUState mapping could stay 1:1?
> > 
> > s390 hardware today already has a topology and CPU features.  We are not 
> > modelling it in QEMU right now, but want to go there in the future so
> > that there would be no simple 1:1 mapping anymore.
> 
> In that case please enlighten us how the topology model should/could
> look like in the future, so that we can align this with the x86
> remodelling and APIC/ID discussion.

Let's discuss this not only between me and Eduardo.
Is short we propose to have generic numa topology interface that will be
accessible via QOM tree. Currently idea is to have tree that looks like:

/machine / node[0..x]/ socket[0..y] / core [0..z] / thread [0..n]
                                    |
                                    / thread [0..n]

where total amount of threads == max_cpus and the rest of nodes are calculated
using respective -smp options.

> 
> Regards,
> Andreas
> 

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Christian Borntraeger]

On 19/04/13 15:16, Andreas Färber wrote:
[...]
>>> Background is that for x86 we currently have a flat CPU core/thread
>>> namespace but would need to deal with sockets, cores and threads to get
>>> topologies right. I assume there are no such issues on s390x, so that
>>> the vCPU to CPUState mapping could stay 1:1?
>>
>> s390 hardware today already has a topology and CPU features.  We are not 
>> modelling it in QEMU right now, but want to go there in the future so that 
>> there would be no simple 1:1 mapping anymore.
> 
> In that case please enlighten us how the topology model should/could
> look like in the future, so that we can align this with the x86
> remodelling and APIC/ID discussion.

Current models provide topology information via the STSI instruction
(http://publibfi.boulder.ibm.com/epubs/pdf/dz9zr009.pdf
see page 10-134 and following) function code 15. The system basically informs about
cpus and containers, which boils down to how the hardware is organized:

e.g. fr zec12 we have up to 4 Nodes(aka books). A book contains one MCM
with 6 processors. Each processor then has 6 cores.
(see http://publibfi.boulder.ibm.com/epubs/pdf/dz9zr009.pdf)

Future systems might look different, I dont know if and how the topology will
change. Currently there is no information about memory locality, only processor
grouping.

As Jens said, as of now we do not provide any topology information to the
guest, but we might to change that in the future.

I havent followed the x86 discussion, so please let me know if that
doesnt answer your question.


Christian

Re: [Qemu-devel] [RFC/PATCH 0/1] cpu hotplug for s390

[Eduardo Habkost]

On Fri, Apr 19, 2013 at 09:13:32PM +0200, Christian Borntraeger wrote:
> On 19/04/13 15:16, Andreas Färber wrote:
> [...]
> >>> Background is that for x86 we currently have a flat CPU core/thread
> >>> namespace but would need to deal with sockets, cores and threads to get
> >>> topologies right. I assume there are no such issues on s390x, so that
> >>> the vCPU to CPUState mapping could stay 1:1?
> >>
> >> s390 hardware today already has a topology and CPU features.  We are not 
> >> modelling it in QEMU right now, but want to go there in the future so that 
> >> there would be no simple 1:1 mapping anymore.
> > 
> > In that case please enlighten us how the topology model should/could
> > look like in the future, so that we can align this with the x86
> > remodelling and APIC/ID discussion.
> 
> Current models provide topology information via the STSI instruction
> (http://publibfi.boulder.ibm.com/epubs/pdf/dz9zr009.pdf
> see page 10-134 and following) function code 15. The system basically informs about
> cpus and containers, which boils down to how the hardware is organized:
> 
> e.g. fr zec12 we have up to 4 Nodes(aka books). A book contains one MCM
> with 6 processors. Each processor then has 6 cores.
> (see http://publibfi.boulder.ibm.com/epubs/pdf/dz9zr009.pdf)
> 
> Future systems might look different, I dont know if and how the topology will
> change. Currently there is no information about memory locality, only processor
> grouping.
> 
> As Jens said, as of now we do not provide any topology information to the
> guest, but we might to change that in the future.
> 
> I havent followed the x86 discussion, so please let me know if that
> doesnt answer your question.

The x86 solution involves adding a "cpu-add" QMP command by now because
we don't have all that's necessary to make device_add work on x86 but we
want to have CPU hotplug in QEMU 1.5.

But in the case of s390 it may be feasible to make device_add work at
the lowest level (CPU core?) (with no topology setting features, because
you still don't have it), and later you can consider adding objects to
represent books, MCM, processors, cores, etc. That's the long-term plan
for x86: having new classes and objects for CPU sockets/cores/threads.

You could even translate cpu-add to the equivalent of a device_add
command, if you want to make the cpu-add command work on s390 too.

Later we may try to agree on a common target-independent device tree
layout so we can have a target-independent topology-aware CPU hotplug
interface. Or we could simply allow each architecture to use a different
device tree layout to represent CPU topology.

-- 
Eduardo