[Qemu-devel] [RFC PATCH v1 00/10] Core based CPU hotplug for PowerPC sPAPR

[Qemu-devel] [RFC PATCH v1 00/10] Core based CPU hotplug for PowerPC sPAPR

[Bharata B Rao]

Hi,

This is the next version of "Core based CPU hotplug for PowerPC sPAPR" that
was posted at
https://lists.gnu.org/archive/html/qemu-ppc/2016-02/msg00286.html

Here is a quick summary on how this approach is different from the
previous approaches that I have been pursuing with the last one being
v7 that posted here:
https://lists.gnu.org/archive/html/qemu-ppc/2016-01/msg00477.html

- Earlier approaches used an independent PowerPC specific core device while
  this approach uses an sPAPR specific core device that is derived from
  generic core device.
- The earlier approach didn't have the notion of where the boot time as
  well as hot-plugged cores would sit. In this approach QOM links are
  created from machine object to all possible core objects. While the
  links are set for boot time cores during machine init, the same is done
  for hotplugged cores at hotplug time. The name of this link property
  is standardized as "core[N]" since these links link the machine object
  with core devices. The link name ("core[N]") is used with core device's
  "slot" property to identify the QOM link to set for this core.
  (qemu) device_add spapr-cpu-core,id=core2,nr_threads=8,cpu_model=host,slot=core[2]
- The ealier approach created threads from instance_init of the core object
  using a modified cpu_generic_init() routine. It used smp_threads and
  MachineState->cpu_model globals. In the current approach, nr_threads and
  cpu_model are obtained as properties and threads are created from
  property setters. The thread objects are allocated as part of core
  device and object_initialize() is used to initialize them.

Some open questions remain:

- Does this device_add semantics look ok ?
  (qemu) device_add spapr-cpu-core,id=core2,nr_threads=8,cpu_model=host,slot=core[2]
- Is there need for machine to core object links ? If not, what would
  determine the slot/location of the core device ?
- How to fit this with CPUSlotProperties HMP interface that Igor is
  working on ?

This version has the following changes:

- Dropped QMP and HMP enumeration patches for now since it isn't clear
  if the approach I took would be preferrable by all archs. Will wait
  and see how Igor's patches evolve here.
- Added the following pre-req patches to support CPU removal:
  exec: Remove cpu from cpus list during cpu_exec_exit()
  exec: Do vmstate unregistration from cpu_exec_exit()
  cpu: Reclaim vCPU objects
  cpu: Add a sync version of cpu_remove()
- Added sPAPR CPU hot removal support
  xics,xics_kvm: Handle CPU unplug correctly
  spapr: CPU hot unplug support
- Moved up the "slot" property into abstract cpu-core device.
- Recover when thread creation fails (spapr_cpu_core_create_threads())
- cpu_model property in spapr-cpu-core deivce is now tracked using
  ObjectClass pointer instead of string.
- Fail topologies with incomplete cores from within sPAPR's machine init.
- Fixes in spapr-cpu-core object creation from machine init to create
  only boottime CPUs.
- Moved board specific wiring code for CPU threads (spapr_cpu_init)
  into core's realize method to be called after each thread's realization.
- No specific action under TYPE_CPU in ->plug() handler either for hotplug
  or hot removal.
- Moved all core related CPU hotplug routines into spapr_cpu_core.c where
  it truly belongs.
- Setting of NUMA node for hotplugged CPUs moved to spapr_cpu_init()).
- Compared to previous implementation of hot removal that was part of
  different series earlier, this implementation moves all core removal
  logic into spapr_cpu_core.c.
- Some minor cleanups like use of g_new0 instead of g_malloc0 etc.

This patchset is present at:
https://github.com/bharata/qemu/commits/spapr-cpu-core

Bharata B Rao (9):
  exec: Remove cpu from cpus list during cpu_exec_exit()
  exec: Do vmstate unregistration from cpu_exec_exit()
  cpu: Add a sync version of cpu_remove()
  cpu: Abstract CPU core type
  spapr: CPU core device
  spapr: Represent boot CPUs as spapr-cpu-core devices
  spapr: CPU hotplug support
  xics,xics_kvm: Handle CPU unplug correctly
  spapr: CPU hot unplug support

Gu Zheng (1):
  cpu: Reclaim vCPU objects

 cpus.c                          |  50 ++++++
 exec.c                          |  44 ++++-
 hw/cpu/Makefile.objs            |   1 +
 hw/cpu/core.c                   |  44 +++++
 hw/intc/xics.c                  |  14 ++
 hw/intc/xics_kvm.c              |   8 +-
 hw/ppc/Makefile.objs            |   1 +
 hw/ppc/spapr.c                  | 174 +++++++++++++++++--
 hw/ppc/spapr_cpu_core.c         | 361 ++++++++++++++++++++++++++++++++++++++++
 hw/ppc/spapr_events.c           |   3 +
 hw/ppc/spapr_rtas.c             |  24 +++
 include/hw/cpu/core.h           |  30 ++++
 include/hw/ppc/spapr.h          |   9 +
 include/hw/ppc/spapr_cpu_core.h |  42 +++++
 include/hw/ppc/xics.h           |   1 +
 include/qom/cpu.h               |  18 ++
 include/sysemu/kvm.h            |   1 +
 kvm-all.c                       |  57 ++++++-
 kvm-stub.c                      |   5 +
 19 files changed, 862 insertions(+), 25 deletions(-)
 create mode 100644 hw/cpu/core.c
 create mode 100644 hw/ppc/spapr_cpu_core.c
 create mode 100644 include/hw/cpu/core.h
 create mode 100644 include/hw/ppc/spapr_cpu_core.h

-- 
2.1.0

[Qemu-devel] [RFC PATCH v1 01/10] exec: Remove cpu from cpus list during cpu_exec_exit()

[Bharata B Rao]

CPUState *cpu gets added to the cpus list during cpu_exec_init(). It
should be removed from cpu_exec_exit().

cpu_exec_init() is called from generic CPU::instance_finalize and some
archs like PowerPC call it from CPU unrealizefn. So ensure that we
dequeue the cpu only once.

Now -1 value for cpu->cpu_index indicates that we have already dequeued
the cpu for CONFIG_USER_ONLY case also.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 exec.c | 32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)

diff --git a/exec.c b/exec.c
index c62c439..7c3f747 100644
--- a/exec.c
+++ b/exec.c
@@ -588,15 +588,9 @@ static int cpu_get_free_index(Error **errp)
     return cpu;
 }
 
-void cpu_exec_exit(CPUState *cpu)
+static void cpu_release_index(CPUState *cpu)
 {
-    if (cpu->cpu_index == -1) {
-        /* cpu_index was never allocated by this @cpu or was already freed. */
-        return;
-    }
-
     bitmap_clear(cpu_index_map, cpu->cpu_index, 1);
-    cpu->cpu_index = -1;
 }
 #else
 
@@ -611,11 +605,33 @@ static int cpu_get_free_index(Error **errp)
     return cpu_index;
 }
 
-void cpu_exec_exit(CPUState *cpu)
+static void cpu_release_index(CPUState *cpu)
 {
+    return;
 }
 #endif
 
+void cpu_exec_exit(CPUState *cpu)
+{
+#if defined(CONFIG_USER_ONLY)
+    cpu_list_lock();
+#endif
+    if (cpu->cpu_index == -1) {
+        /* cpu_index was never allocated by this @cpu or was already freed. */
+#if defined(CONFIG_USER_ONLY)
+        cpu_list_unlock();
+#endif
+        return;
+    }
+
+    QTAILQ_REMOVE(&cpus, cpu, node);
+    cpu_release_index(cpu);
+    cpu->cpu_index = -1;
+#if defined(CONFIG_USER_ONLY)
+    cpu_list_unlock();
+#endif
+}
+
 void cpu_exec_init(CPUState *cpu, Error **errp)
 {
     CPUClass *cc = CPU_GET_CLASS(cpu);
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 01/10] exec: Remove cpu from cpus list during cpu_exec_exit()

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On Fri, Mar 04, 2016 at 12:24:12PM +0530, Bharata B Rao wrote:
> CPUState *cpu gets added to the cpus list during cpu_exec_init(). It
> should be removed from cpu_exec_exit().
> 
> cpu_exec_init() is called from generic CPU::instance_finalize and some
> archs like PowerPC call it from CPU unrealizefn. So ensure that we
> dequeue the cpu only once.
> 
> Now -1 value for cpu->cpu_index indicates that we have already dequeued
> the cpu for CONFIG_USER_ONLY case also.
> 
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 01/10] exec: Remove cpu from cpus list during cpu_exec_exit()

[Thomas Huth]

On 04.03.2016 07:54, Bharata B Rao wrote:
> CPUState *cpu gets added to the cpus list during cpu_exec_init(). It
> should be removed from cpu_exec_exit().
> 
> cpu_exec_init() is called from generic CPU::instance_finalize and some

s/cpu_exec_init/cpu_exec_exit/

> archs like PowerPC call it from CPU unrealizefn. So ensure that we
> dequeue the cpu only once.
> 
> Now -1 value for cpu->cpu_index indicates that we have already dequeued
> the cpu for CONFIG_USER_ONLY case also.
> 
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> ---
>  exec.c | 32 ++++++++++++++++++++++++--------
>  1 file changed, 24 insertions(+), 8 deletions(-)
> 
> diff --git a/exec.c b/exec.c
> index c62c439..7c3f747 100644
> --- a/exec.c
> +++ b/exec.c
> @@ -588,15 +588,9 @@ static int cpu_get_free_index(Error **errp)
>      return cpu;
>  }
>  
> -void cpu_exec_exit(CPUState *cpu)
> +static void cpu_release_index(CPUState *cpu)
>  {
> -    if (cpu->cpu_index == -1) {
> -        /* cpu_index was never allocated by this @cpu or was already freed. */
> -        return;
> -    }
> -
>      bitmap_clear(cpu_index_map, cpu->cpu_index, 1);
> -    cpu->cpu_index = -1;
>  }
>  #else
>  
> @@ -611,11 +605,33 @@ static int cpu_get_free_index(Error **errp)
>      return cpu_index;
>  }
>  
> -void cpu_exec_exit(CPUState *cpu)
> +static void cpu_release_index(CPUState *cpu)
>  {
> +    return;

You could also simply leave that return statement away, I think.

>  }
>  #endif
>  
> +void cpu_exec_exit(CPUState *cpu)
> +{
> +#if defined(CONFIG_USER_ONLY)
> +    cpu_list_lock();
> +#endif
> +    if (cpu->cpu_index == -1) {
> +        /* cpu_index was never allocated by this @cpu or was already freed. */
> +#if defined(CONFIG_USER_ONLY)
> +        cpu_list_unlock();
> +#endif
> +        return;
> +    }
> +
> +    QTAILQ_REMOVE(&cpus, cpu, node);
> +    cpu_release_index(cpu);
> +    cpu->cpu_index = -1;
> +#if defined(CONFIG_USER_ONLY)
> +    cpu_list_unlock();
> +#endif
> +}

Since there are a couple of these

#if defined(CONFIG_USER_ONLY)
    cpu_list_[un]lock();
#endif

in exec.c already, it might be somewhat nices to declare them at the
beginning of the file as empty functions, somewhat like:

#if !defined(CONFIG_USER_ONLY)
static inline void cpu_list_lock(void)
{
}
static inline void cpu_list_unlock(void)
{
}
#endif

What do you think about that?

Apart from that, the patch looks fine to me.

 Thomas

Re: [Qemu-devel] [RFC PATCH v1 01/10] exec: Remove cpu from cpus list during cpu_exec_exit()

[Bharata B Rao]

On Mon, Mar 07, 2016 at 05:23:48PM +0100, Thomas Huth wrote:
> On 04.03.2016 07:54, Bharata B Rao wrote:
> > CPUState *cpu gets added to the cpus list during cpu_exec_init(). It
> > should be removed from cpu_exec_exit().
> > 
> > cpu_exec_init() is called from generic CPU::instance_finalize and some
> 
> s/cpu_exec_init/cpu_exec_exit/
> 
> > archs like PowerPC call it from CPU unrealizefn. So ensure that we
> > dequeue the cpu only once.
> > 
> > Now -1 value for cpu->cpu_index indicates that we have already dequeued
> > the cpu for CONFIG_USER_ONLY case also.
> > 
> > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > ---
> >  exec.c | 32 ++++++++++++++++++++++++--------
> >  1 file changed, 24 insertions(+), 8 deletions(-)
> > 
> > diff --git a/exec.c b/exec.c
> > index c62c439..7c3f747 100644
> > --- a/exec.c
> > +++ b/exec.c
> > @@ -588,15 +588,9 @@ static int cpu_get_free_index(Error **errp)
> >      return cpu;
> >  }
> >  
> > -void cpu_exec_exit(CPUState *cpu)
> > +static void cpu_release_index(CPUState *cpu)
> >  {
> > -    if (cpu->cpu_index == -1) {
> > -        /* cpu_index was never allocated by this @cpu or was already freed. */
> > -        return;
> > -    }
> > -
> >      bitmap_clear(cpu_index_map, cpu->cpu_index, 1);
> > -    cpu->cpu_index = -1;
> >  }
> >  #else
> >  
> > @@ -611,11 +605,33 @@ static int cpu_get_free_index(Error **errp)
> >      return cpu_index;
> >  }
> >  
> > -void cpu_exec_exit(CPUState *cpu)
> > +static void cpu_release_index(CPUState *cpu)
> >  {
> > +    return;
> 
> You could also simply leave that return statement away, I think.
> 
> >  }
> >  #endif
> >  
> > +void cpu_exec_exit(CPUState *cpu)
> > +{
> > +#if defined(CONFIG_USER_ONLY)
> > +    cpu_list_lock();
> > +#endif
> > +    if (cpu->cpu_index == -1) {
> > +        /* cpu_index was never allocated by this @cpu or was already freed. */
> > +#if defined(CONFIG_USER_ONLY)
> > +        cpu_list_unlock();
> > +#endif
> > +        return;
> > +    }
> > +
> > +    QTAILQ_REMOVE(&cpus, cpu, node);
> > +    cpu_release_index(cpu);
> > +    cpu->cpu_index = -1;
> > +#if defined(CONFIG_USER_ONLY)
> > +    cpu_list_unlock();
> > +#endif
> > +}
> 
> Since there are a couple of these
> 
> #if defined(CONFIG_USER_ONLY)
>     cpu_list_[un]lock();
> #endif
> 
> in exec.c already, it might be somewhat nices to declare them at the
> beginning of the file as empty functions, somewhat like:
> 
> #if !defined(CONFIG_USER_ONLY)
> static inline void cpu_list_lock(void)
> {
> }
> static inline void cpu_list_unlock(void)
> {
> }
> #endif
> 
> What do you think about that?

If you and/or the maintainer insist/prefer, I can make the change.

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v1 01/10] exec: Remove cpu from cpus list during cpu_exec_exit()

[Thomas Huth]

On 09.03.2016 08:57, Bharata B Rao wrote:
> On Mon, Mar 07, 2016 at 05:23:48PM +0100, Thomas Huth wrote:
>> On 04.03.2016 07:54, Bharata B Rao wrote:
>>> CPUState *cpu gets added to the cpus list during cpu_exec_init(). It
>>> should be removed from cpu_exec_exit().
>>>
>>> cpu_exec_init() is called from generic CPU::instance_finalize and some
>>
>> s/cpu_exec_init/cpu_exec_exit/
>>
>>> archs like PowerPC call it from CPU unrealizefn. So ensure that we
>>> dequeue the cpu only once.
>>>
>>> Now -1 value for cpu->cpu_index indicates that we have already dequeued
>>> the cpu for CONFIG_USER_ONLY case also.
>>>
>>> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
>>> ---
>>>  exec.c | 32 ++++++++++++++++++++++++--------
>>>  1 file changed, 24 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/exec.c b/exec.c
>>> index c62c439..7c3f747 100644
>>> --- a/exec.c
>>> +++ b/exec.c
>>> @@ -588,15 +588,9 @@ static int cpu_get_free_index(Error **errp)
>>>      return cpu;
>>>  }
>>>  
>>> -void cpu_exec_exit(CPUState *cpu)
>>> +static void cpu_release_index(CPUState *cpu)
>>>  {
>>> -    if (cpu->cpu_index == -1) {
>>> -        /* cpu_index was never allocated by this @cpu or was already freed. */
>>> -        return;
>>> -    }
>>> -
>>>      bitmap_clear(cpu_index_map, cpu->cpu_index, 1);
>>> -    cpu->cpu_index = -1;
>>>  }
>>>  #else
>>>  
>>> @@ -611,11 +605,33 @@ static int cpu_get_free_index(Error **errp)
>>>      return cpu_index;
>>>  }
>>>  
>>> -void cpu_exec_exit(CPUState *cpu)
>>> +static void cpu_release_index(CPUState *cpu)
>>>  {
>>> +    return;
>>
>> You could also simply leave that return statement away, I think.
>>
>>>  }
>>>  #endif
>>>  
>>> +void cpu_exec_exit(CPUState *cpu)
>>> +{
>>> +#if defined(CONFIG_USER_ONLY)
>>> +    cpu_list_lock();
>>> +#endif
>>> +    if (cpu->cpu_index == -1) {
>>> +        /* cpu_index was never allocated by this @cpu or was already freed. */
>>> +#if defined(CONFIG_USER_ONLY)
>>> +        cpu_list_unlock();
>>> +#endif
>>> +        return;
>>> +    }
>>> +
>>> +    QTAILQ_REMOVE(&cpus, cpu, node);
>>> +    cpu_release_index(cpu);
>>> +    cpu->cpu_index = -1;
>>> +#if defined(CONFIG_USER_ONLY)
>>> +    cpu_list_unlock();
>>> +#endif
>>> +}
>>
>> Since there are a couple of these
>>
>> #if defined(CONFIG_USER_ONLY)
>>     cpu_list_[un]lock();
>> #endif
>>
>> in exec.c already, it might be somewhat nices to declare them at the
>> beginning of the file as empty functions, somewhat like:
>>
>> #if !defined(CONFIG_USER_ONLY)
>> static inline void cpu_list_lock(void)
>> {
>> }
>> static inline void cpu_list_unlock(void)
>> {
>> }
>> #endif
>>
>> What do you think about that?
> 
> If you and/or the maintainer insist/prefer, I can make the change.

It would be a nice way to get rid of some #if statements in the code,
but I don't insist on that change ... so feel free to keep the current
state, if you prefer it.

 Thomas

[Qemu-devel] [RFC PATCH v1 02/10] exec: Do vmstate unregistration from cpu_exec_exit()

[Bharata B Rao]

cpu_exec_init() does vmstate_register and register_savevm for the CPU device.
These need to be undone from cpu_exec_exit(). These changes are needed to
support CPU hot removal.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 exec.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/exec.c b/exec.c
index 7c3f747..b8eeb54 100644
--- a/exec.c
+++ b/exec.c
@@ -613,6 +613,8 @@ static void cpu_release_index(CPUState *cpu)
 
 void cpu_exec_exit(CPUState *cpu)
 {
+    CPUClass *cc = CPU_GET_CLASS(cpu);
+
 #if defined(CONFIG_USER_ONLY)
     cpu_list_lock();
 #endif
@@ -630,6 +632,16 @@ void cpu_exec_exit(CPUState *cpu)
 #if defined(CONFIG_USER_ONLY)
     cpu_list_unlock();
 #endif
+
+    if (cc->vmsd != NULL) {
+        vmstate_unregister(NULL, cc->vmsd, cpu);
+    }
+#if defined(CPU_SAVE_VERSION) && !defined(CONFIG_USER_ONLY)
+    unregister_savevm(NULL, "cpu", cpu->env_ptr);
+#endif
+    if (qdev_get_vmsd(DEVICE(cpu)) == NULL) {
+        vmstate_unregister(NULL, &vmstate_cpu_common, cpu);
+    }
 }
 
 void cpu_exec_init(CPUState *cpu, Error **errp)
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 02/10] exec: Do vmstate unregistration from cpu_exec_exit()

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

On Fri, Mar 04, 2016 at 12:24:13PM +0530, Bharata B Rao wrote:
> cpu_exec_init() does vmstate_register and register_savevm for the CPU device.
> These need to be undone from cpu_exec_exit(). These changes are needed to
> support CPU hot removal.
> 
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> ---
>  exec.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/exec.c b/exec.c
> index 7c3f747..b8eeb54 100644
> --- a/exec.c
> +++ b/exec.c
> @@ -613,6 +613,8 @@ static void cpu_release_index(CPUState *cpu)
>  
>  void cpu_exec_exit(CPUState *cpu)
>  {
> +    CPUClass *cc = CPU_GET_CLASS(cpu);
> +
>  #if defined(CONFIG_USER_ONLY)
>      cpu_list_lock();
>  #endif
> @@ -630,6 +632,16 @@ void cpu_exec_exit(CPUState *cpu)
>  #if defined(CONFIG_USER_ONLY)
>      cpu_list_unlock();
>  #endif
> +
> +    if (cc->vmsd != NULL) {
> +        vmstate_unregister(NULL, cc->vmsd, cpu);
> +    }
> +#if defined(CPU_SAVE_VERSION) && !defined(CONFIG_USER_ONLY)
> +    unregister_savevm(NULL, "cpu", cpu->env_ptr);
> +#endif
> +    if (qdev_get_vmsd(DEVICE(cpu)) == NULL) {
> +        vmstate_unregister(NULL, &vmstate_cpu_common, cpu);
> +    }
>  }
>  
>  void cpu_exec_init(CPUState *cpu, Error **errp)

Looking at the current cpu_exec_init() I'm seeing the
vmstate_register() calls which are balacned here, but not a
register_savevm().  Is that due to a change in cpu_exec_init() since
you first wrote this patch?

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 02/10] exec: Do vmstate unregistration from cpu_exec_exit()

[Bharata B Rao]

On Mon, Mar 07, 2016 at 01:51:43PM +1100, David Gibson wrote:
> On Fri, Mar 04, 2016 at 12:24:13PM +0530, Bharata B Rao wrote:
> > cpu_exec_init() does vmstate_register and register_savevm for the CPU device.
> > These need to be undone from cpu_exec_exit(). These changes are needed to
> > support CPU hot removal.
> > 
> > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > ---
> >  exec.c | 12 ++++++++++++
> >  1 file changed, 12 insertions(+)
> > 
> > diff --git a/exec.c b/exec.c
> > index 7c3f747..b8eeb54 100644
> > --- a/exec.c
> > +++ b/exec.c
> > @@ -613,6 +613,8 @@ static void cpu_release_index(CPUState *cpu)
> >  
> >  void cpu_exec_exit(CPUState *cpu)
> >  {
> > +    CPUClass *cc = CPU_GET_CLASS(cpu);
> > +
> >  #if defined(CONFIG_USER_ONLY)
> >      cpu_list_lock();
> >  #endif
> > @@ -630,6 +632,16 @@ void cpu_exec_exit(CPUState *cpu)
> >  #if defined(CONFIG_USER_ONLY)
> >      cpu_list_unlock();
> >  #endif
> > +
> > +    if (cc->vmsd != NULL) {
> > +        vmstate_unregister(NULL, cc->vmsd, cpu);
> > +    }
> > +#if defined(CPU_SAVE_VERSION) && !defined(CONFIG_USER_ONLY)
> > +    unregister_savevm(NULL, "cpu", cpu->env_ptr);
> > +#endif
> > +    if (qdev_get_vmsd(DEVICE(cpu)) == NULL) {
> > +        vmstate_unregister(NULL, &vmstate_cpu_common, cpu);
> > +    }
> >  }
> >  
> >  void cpu_exec_init(CPUState *cpu, Error **errp)
> 
> Looking at the current cpu_exec_init() I'm seeing the
> vmstate_register() calls which are balacned here, but not a
> register_savevm().  Is that due to a change in cpu_exec_init() since
> you first wrote this patch?

Yes register_savevm() was removed by 945123a554c3. Will fix this patch
in the next iteration.

Regards,
Bharata.

[Qemu-devel] [RFC PATCH v1 03/10] cpu: Reclaim vCPU objects

[Bharata B Rao]

From: Gu Zheng <guz.fnst@cn.fujitsu.com>

In order to deal well with the kvm vcpus (which can not be removed without any
protection), we do not close KVM vcpu fd, just record and mark it as stopped
into a list, so that we can reuse it for the appending cpu hot-add request if
possible. It is also the approach that kvm guys suggested:
https://www.mail-archive.com/kvm@vger.kernel.org/msg102839.html

Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
               [- Explicit CPU_REMOVE() from qemu_kvm/tcg_destroy_vcpu()
                  isn't needed as it is done from cpu_exec_exit()
                - Use iothread mutex instead of global mutex during
                  destroy
                - Don't cleanup vCPU object from vCPU thread context
                  but leave it to the callers (device_add/device_del)]
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 cpus.c               | 38 +++++++++++++++++++++++++++++++++++
 include/qom/cpu.h    | 10 +++++++++
 include/sysemu/kvm.h |  1 +
 kvm-all.c            | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 kvm-stub.c           |  5 +++++
 5 files changed, 110 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index 9592163..07cc054 100644
--- a/cpus.c
+++ b/cpus.c
@@ -953,6 +953,18 @@ void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
     qemu_cpu_kick(cpu);
 }
 
+static void qemu_kvm_destroy_vcpu(CPUState *cpu)
+{
+    if (kvm_destroy_vcpu(cpu) < 0) {
+        error_report("kvm_destroy_vcpu failed");
+        exit(EXIT_FAILURE);
+    }
+}
+
+static void qemu_tcg_destroy_vcpu(CPUState *cpu)
+{
+}
+
 static void flush_queued_work(CPUState *cpu)
 {
     struct qemu_work_item *wi;
@@ -1053,6 +1065,11 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
             }
         }
         qemu_kvm_wait_io_event(cpu);
+        if (cpu->exit && !cpu_can_run(cpu)) {
+            qemu_kvm_destroy_vcpu(cpu);
+            qemu_mutex_unlock_iothread();
+            return NULL;
+        }
     }
 
     return NULL;
@@ -1108,6 +1125,7 @@ static void tcg_exec_all(void);
 static void *qemu_tcg_cpu_thread_fn(void *arg)
 {
     CPUState *cpu = arg;
+    CPUState *remove_cpu = NULL;
 
     rcu_register_thread();
 
@@ -1145,6 +1163,16 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
             }
         }
         qemu_tcg_wait_io_event(QTAILQ_FIRST(&cpus));
+        CPU_FOREACH(cpu) {
+            if (cpu->exit && !cpu_can_run(cpu)) {
+                remove_cpu = cpu;
+                break;
+            }
+        }
+        if (remove_cpu) {
+            qemu_tcg_destroy_vcpu(remove_cpu);
+            remove_cpu = NULL;
+        }
     }
 
     return NULL;
@@ -1301,6 +1329,13 @@ void resume_all_vcpus(void)
     }
 }
 
+void cpu_remove(CPUState *cpu)
+{
+    cpu->stop = true;
+    cpu->exit = true;
+    qemu_cpu_kick(cpu);
+}
+
 /* For temporary buffers for forming a name */
 #define VCPU_THREAD_NAME_SIZE 16
 
@@ -1517,6 +1552,9 @@ static void tcg_exec_all(void)
                 break;
             }
         } else if (cpu->stop || cpu->stopped) {
+            if (cpu->exit) {
+                next_cpu = CPU_NEXT(cpu);
+            }
             break;
         }
     }
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 7052eee..6e5171b 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -237,6 +237,7 @@ struct kvm_run;
  * @halted: Nonzero if the CPU is in suspended state.
  * @stop: Indicates a pending stop request.
  * @stopped: Indicates the CPU has been artificially stopped.
+ * @exit: Indicates the CPU has exited due to an unplug operation.
  * @crash_occurred: Indicates the OS reported a crash (panic) for this CPU
  * @tcg_exit_req: Set to force TCG to stop executing linked TBs for this
  *           CPU and return to its top level loop.
@@ -289,6 +290,7 @@ struct CPUState {
     bool created;
     bool stop;
     bool stopped;
+    bool exit;
     bool crash_occurred;
     bool exit_request;
     uint32_t interrupt_request;
@@ -756,6 +758,14 @@ void cpu_exit(CPUState *cpu);
 void cpu_resume(CPUState *cpu);
 
 /**
+ * cpu_remove:
+ * @cpu: The CPU to remove.
+ *
+ * Requests the CPU to be removed.
+ */
+void cpu_remove(CPUState *cpu);
+
+/**
  * qemu_init_vcpu:
  * @cpu: The vCPU to initialize.
  *
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 6695fa7..5d5b602 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -216,6 +216,7 @@ int kvm_has_intx_set_mask(void);
 
 int kvm_init_vcpu(CPUState *cpu);
 int kvm_cpu_exec(CPUState *cpu);
+int kvm_destroy_vcpu(CPUState *cpu);
 
 #ifdef NEED_CPU_H
 
diff --git a/kvm-all.c b/kvm-all.c
index a65e73f..b41ff46 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -61,6 +61,12 @@
 
 #define KVM_MSI_HASHTAB_SIZE    256
 
+struct KVMParkedVcpu {
+    unsigned long vcpu_id;
+    int kvm_fd;
+    QLIST_ENTRY(KVMParkedVcpu) node;
+};
+
 struct KVMState
 {
     AccelState parent_obj;
@@ -94,6 +100,7 @@ struct KVMState
     QTAILQ_HEAD(msi_hashtab, KVMMSIRoute) msi_hashtab[KVM_MSI_HASHTAB_SIZE];
 #endif
     KVMMemoryListener memory_listener;
+    QLIST_HEAD(, KVMParkedVcpu) kvm_parked_vcpus;
 };
 
 KVMState *kvm_state;
@@ -237,6 +244,53 @@ static int kvm_set_user_memory_region(KVMMemoryListener *kml, KVMSlot *slot)
     return kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
 }
 
+int kvm_destroy_vcpu(CPUState *cpu)
+{
+    KVMState *s = kvm_state;
+    long mmap_size;
+    struct KVMParkedVcpu *vcpu = NULL;
+    int ret = 0;
+
+    DPRINTF("kvm_destroy_vcpu\n");
+
+    mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
+    if (mmap_size < 0) {
+        ret = mmap_size;
+        DPRINTF("KVM_GET_VCPU_MMAP_SIZE failed\n");
+        goto err;
+    }
+
+    ret = munmap(cpu->kvm_run, mmap_size);
+    if (ret < 0) {
+        goto err;
+    }
+
+    vcpu = g_malloc0(sizeof(*vcpu));
+    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
+    vcpu->kvm_fd = cpu->kvm_fd;
+    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
+err:
+    return ret;
+}
+
+static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id)
+{
+    struct KVMParkedVcpu *cpu;
+
+    QLIST_FOREACH(cpu, &s->kvm_parked_vcpus, node) {
+        if (cpu->vcpu_id == vcpu_id) {
+            int kvm_fd;
+
+            QLIST_REMOVE(cpu, node);
+            kvm_fd = cpu->kvm_fd;
+            g_free(cpu);
+            return kvm_fd;
+        }
+    }
+
+    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
+}
+
 int kvm_init_vcpu(CPUState *cpu)
 {
     KVMState *s = kvm_state;
@@ -245,7 +299,7 @@ int kvm_init_vcpu(CPUState *cpu)
 
     DPRINTF("kvm_init_vcpu\n");
 
-    ret = kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)kvm_arch_vcpu_id(cpu));
+    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
     if (ret < 0) {
         DPRINTF("kvm_create_vcpu failed\n");
         goto err;
@@ -1508,6 +1562,7 @@ static int kvm_init(MachineState *ms)
 #ifdef KVM_CAP_SET_GUEST_DEBUG
     QTAILQ_INIT(&s->kvm_sw_breakpoints);
 #endif
+    QLIST_INIT(&s->kvm_parked_vcpus);
     s->vmfd = -1;
     s->fd = qemu_open("/dev/kvm", O_RDWR);
     if (s->fd == -1) {
diff --git a/kvm-stub.c b/kvm-stub.c
index b962b24..61f9d5c 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -33,6 +33,11 @@ bool kvm_allowed;
 bool kvm_readonly_mem_allowed;
 bool kvm_ioeventfd_any_length_allowed;
 
+int kvm_destroy_vcpu(CPUState *cpu)
+{
+    return -ENOSYS;
+}
+
 int kvm_init_vcpu(CPUState *cpu)
 {
     return -ENOSYS;
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 03/10] cpu: Reclaim vCPU objects

[Thomas Huth]

On 04.03.2016 07:54, Bharata B Rao wrote:
> From: Gu Zheng <guz.fnst@cn.fujitsu.com>
> 
> In order to deal well with the kvm vcpus (which can not be removed without any
> protection), we do not close KVM vcpu fd, just record and mark it as stopped
> into a list, so that we can reuse it for the appending cpu hot-add request if
> possible. It is also the approach that kvm guys suggested:
> https://www.mail-archive.com/kvm@vger.kernel.org/msg102839.html
> 
> Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
> Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
> Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
>                [- Explicit CPU_REMOVE() from qemu_kvm/tcg_destroy_vcpu()
>                   isn't needed as it is done from cpu_exec_exit()
>                 - Use iothread mutex instead of global mutex during
>                   destroy
>                 - Don't cleanup vCPU object from vCPU thread context
>                   but leave it to the callers (device_add/device_del)]
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  cpus.c               | 38 +++++++++++++++++++++++++++++++++++
>  include/qom/cpu.h    | 10 +++++++++
>  include/sysemu/kvm.h |  1 +
>  kvm-all.c            | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++-
>  kvm-stub.c           |  5 +++++
>  5 files changed, 110 insertions(+), 1 deletion(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 9592163..07cc054 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -953,6 +953,18 @@ void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
>      qemu_cpu_kick(cpu);
>  }
>  
> +static void qemu_kvm_destroy_vcpu(CPUState *cpu)
> +{
> +    if (kvm_destroy_vcpu(cpu) < 0) {
> +        error_report("kvm_destroy_vcpu failed");
> +        exit(EXIT_FAILURE);
> +    }
> +}
> +
> +static void qemu_tcg_destroy_vcpu(CPUState *cpu)
> +{
> +}
> +
>  static void flush_queued_work(CPUState *cpu)
>  {
>      struct qemu_work_item *wi;
> @@ -1053,6 +1065,11 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
>              }
>          }
>          qemu_kvm_wait_io_event(cpu);
> +        if (cpu->exit && !cpu_can_run(cpu)) {
> +            qemu_kvm_destroy_vcpu(cpu);
> +            qemu_mutex_unlock_iothread();
> +            return NULL;
> +        }

My comment from last time still applies:

You could increase readability of the code by changing the condition of
the loop instead - currently it is a "while (1)" ... you could turn that
into a "do { ... } while (!cpu->exit || cpu_can_run(cpu))" and then
destroy the cpu after the loop.

>      }
>  
>      return NULL;
> @@ -1108,6 +1125,7 @@ static void tcg_exec_all(void);
>  static void *qemu_tcg_cpu_thread_fn(void *arg)
>  {
>      CPUState *cpu = arg;
> +    CPUState *remove_cpu = NULL;
>  
>      rcu_register_thread();
>  
> @@ -1145,6 +1163,16 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
>              }
>          }
>          qemu_tcg_wait_io_event(QTAILQ_FIRST(&cpus));
> +        CPU_FOREACH(cpu) {
> +            if (cpu->exit && !cpu_can_run(cpu)) {
> +                remove_cpu = cpu;
> +                break;
> +            }
> +        }
> +        if (remove_cpu) {
> +            qemu_tcg_destroy_vcpu(remove_cpu);
> +            remove_cpu = NULL;
> +        }
>      }
>  
>      return NULL;
> @@ -1301,6 +1329,13 @@ void resume_all_vcpus(void)
>      }
>  }
>  
> +void cpu_remove(CPUState *cpu)
> +{
> +    cpu->stop = true;
> +    cpu->exit = true;
> +    qemu_cpu_kick(cpu);
> +}
> +
>  /* For temporary buffers for forming a name */
>  #define VCPU_THREAD_NAME_SIZE 16
>  
> @@ -1517,6 +1552,9 @@ static void tcg_exec_all(void)
>                  break;
>              }
>          } else if (cpu->stop || cpu->stopped) {
> +            if (cpu->exit) {
> +                next_cpu = CPU_NEXT(cpu);
> +            }
>              break;
>          }
>      }
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 7052eee..6e5171b 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -237,6 +237,7 @@ struct kvm_run;
>   * @halted: Nonzero if the CPU is in suspended state.
>   * @stop: Indicates a pending stop request.
>   * @stopped: Indicates the CPU has been artificially stopped.
> + * @exit: Indicates the CPU has exited due to an unplug operation.
>   * @crash_occurred: Indicates the OS reported a crash (panic) for this CPU
>   * @tcg_exit_req: Set to force TCG to stop executing linked TBs for this
>   *           CPU and return to its top level loop.
> @@ -289,6 +290,7 @@ struct CPUState {
>      bool created;
>      bool stop;
>      bool stopped;
> +    bool exit;

Another comment from last review:

There is also a "exit_request" member in this struct already ... maybe
you could name the new variable differently to avoid confusion?
Something like "remove_request" or "unplug_request" ?

>      bool crash_occurred;
>      bool exit_request;
>      uint32_t interrupt_request;
...

Apart from that, the patch looks fine to me.

 Thomas

Re: [Qemu-devel] [RFC PATCH v1 03/10] cpu: Reclaim vCPU objects

[Bharata B Rao]

On Mon, Mar 07, 2016 at 08:05:58PM +0100, Thomas Huth wrote:
> On 04.03.2016 07:54, Bharata B Rao wrote:
> > From: Gu Zheng <guz.fnst@cn.fujitsu.com>
> > 
> > In order to deal well with the kvm vcpus (which can not be removed without any
> > protection), we do not close KVM vcpu fd, just record and mark it as stopped
> > into a list, so that we can reuse it for the appending cpu hot-add request if
> > possible. It is also the approach that kvm guys suggested:
> > https://www.mail-archive.com/kvm@vger.kernel.org/msg102839.html
> > 
> > Signed-off-by: Chen Fan <chen.fan.fnst@cn.fujitsu.com>
> > Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
> > Signed-off-by: Zhu Guihua <zhugh.fnst@cn.fujitsu.com>
> > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> >                [- Explicit CPU_REMOVE() from qemu_kvm/tcg_destroy_vcpu()
> >                   isn't needed as it is done from cpu_exec_exit()
> >                 - Use iothread mutex instead of global mutex during
> >                   destroy
> >                 - Don't cleanup vCPU object from vCPU thread context
> >                   but leave it to the callers (device_add/device_del)]
> > Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> > ---
> >  cpus.c               | 38 +++++++++++++++++++++++++++++++++++
> >  include/qom/cpu.h    | 10 +++++++++
> >  include/sysemu/kvm.h |  1 +
> >  kvm-all.c            | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++-
> >  kvm-stub.c           |  5 +++++
> >  5 files changed, 110 insertions(+), 1 deletion(-)
> > 
> > diff --git a/cpus.c b/cpus.c
> > index 9592163..07cc054 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -953,6 +953,18 @@ void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
> >      qemu_cpu_kick(cpu);
> >  }
> >  
> > +static void qemu_kvm_destroy_vcpu(CPUState *cpu)
> > +{
> > +    if (kvm_destroy_vcpu(cpu) < 0) {
> > +        error_report("kvm_destroy_vcpu failed");
> > +        exit(EXIT_FAILURE);
> > +    }
> > +}
> > +
> > +static void qemu_tcg_destroy_vcpu(CPUState *cpu)
> > +{
> > +}
> > +
> >  static void flush_queued_work(CPUState *cpu)
> >  {
> >      struct qemu_work_item *wi;
> > @@ -1053,6 +1065,11 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
> >              }
> >          }
> >          qemu_kvm_wait_io_event(cpu);
> > +        if (cpu->exit && !cpu_can_run(cpu)) {
> > +            qemu_kvm_destroy_vcpu(cpu);
> > +            qemu_mutex_unlock_iothread();
> > +            return NULL;
> > +        }
> 
> My comment from last time still applies:
> 
> You could increase readability of the code by changing the condition of
> the loop instead - currently it is a "while (1)" ... you could turn that
> into a "do { ... } while (!cpu->exit || cpu_can_run(cpu))" and then
> destroy the cpu after the loop.

Sorry for missing this, will take of this and the other comment in this
thread in the next version.

Regards,
Bharata.

[Qemu-devel] [RFC PATCH v1 04/10] cpu: Add a sync version of cpu_remove()

[Bharata B Rao]

This sync API will be used by the CPU hotplug code to wait for the CPU to
completely get removed before flagging the failure to the device_add
command.

Sync version of this call is needed to correctly recover from CPU
realization failures when ->plug() handler fails.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 cpus.c            | 12 ++++++++++++
 include/qom/cpu.h |  8 ++++++++
 2 files changed, 20 insertions(+)

diff --git a/cpus.c b/cpus.c
index 07cc054..4268334 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1067,6 +1067,8 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
         qemu_kvm_wait_io_event(cpu);
         if (cpu->exit && !cpu_can_run(cpu)) {
             qemu_kvm_destroy_vcpu(cpu);
+            cpu->created = false;
+            qemu_cond_signal(&qemu_cpu_cond);
             qemu_mutex_unlock_iothread();
             return NULL;
         }
@@ -1171,6 +1173,8 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
         }
         if (remove_cpu) {
             qemu_tcg_destroy_vcpu(remove_cpu);
+            cpu->created = false;
+            qemu_cond_signal(&qemu_cpu_cond);
             remove_cpu = NULL;
         }
     }
@@ -1336,6 +1340,14 @@ void cpu_remove(CPUState *cpu)
     qemu_cpu_kick(cpu);
 }
 
+void cpu_remove_sync(CPUState *cpu)
+{
+    cpu_remove(cpu);
+    while (cpu->created) {
+        qemu_cond_wait(&qemu_cpu_cond, &qemu_global_mutex);
+    }
+}
+
 /* For temporary buffers for forming a name */
 #define VCPU_THREAD_NAME_SIZE 16
 
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 6e5171b..de8600d 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -765,6 +765,14 @@ void cpu_resume(CPUState *cpu);
  */
 void cpu_remove(CPUState *cpu);
 
+ /**
+ * cpu_remove_sync:
+ * @cpu: The CPU to remove.
+ *
+ * Requests the CPU to be removed and waits till it is removed.
+ */
+void cpu_remove_sync(CPUState *cpu);
+
 /**
  * qemu_init_vcpu:
  * @cpu: The vCPU to initialize.
-- 
2.1.0

[Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Bharata B Rao]

Add an abstract CPU core type that could be used by machines that want
to define and hotplug CPUs in core granularity.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 hw/cpu/Makefile.objs  |  1 +
 hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+)
 create mode 100644 hw/cpu/core.c
 create mode 100644 include/hw/cpu/core.h

diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
index 0954a18..942a4bb 100644
--- a/hw/cpu/Makefile.objs
+++ b/hw/cpu/Makefile.objs
@@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
 obj-$(CONFIG_REALVIEW) += realview_mpcore.o
 obj-$(CONFIG_A9MPCORE) += a9mpcore.o
 obj-$(CONFIG_A15MPCORE) += a15mpcore.o
+obj-y += core.o
 
diff --git a/hw/cpu/core.c b/hw/cpu/core.c
new file mode 100644
index 0000000..d8caf37
--- /dev/null
+++ b/hw/cpu/core.c
@@ -0,0 +1,44 @@
+/*
+ * CPU core abstract device
+ *
+ * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include "hw/cpu/core.h"
+
+static char *core_prop_get_slot(Object *obj, Error **errp)
+{
+    CPUCore *core = CPU_CORE(obj);
+
+    return g_strdup(core->slot);
+}
+
+static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
+{
+    CPUCore *core = CPU_CORE(obj);
+
+    core->slot = g_strdup(val);
+}
+
+static void cpu_core_instance_init(Object *obj)
+{
+    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
+                            NULL);
+}
+
+static const TypeInfo cpu_core_type_info = {
+    .name = TYPE_CPU_CORE,
+    .parent = TYPE_DEVICE,
+    .abstract = true,
+    .instance_size = sizeof(CPUCore),
+    .instance_init = cpu_core_instance_init,
+};
+
+static void cpu_core_register_types(void)
+{
+    type_register_static(&cpu_core_type_info);
+}
+
+type_init(cpu_core_register_types)
diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
new file mode 100644
index 0000000..2daa724
--- /dev/null
+++ b/include/hw/cpu/core.h
@@ -0,0 +1,30 @@
+/*
+ * CPU core abstract device
+ *
+ * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#ifndef HW_CPU_CORE_H
+#define HW_CPU_CORE_H
+
+#include "qemu/osdep.h"
+#include "hw/qdev.h"
+
+#define TYPE_CPU_CORE "cpu-core"
+
+#define CPU_CORE(obj) \
+    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
+
+typedef struct CPUCore {
+    /*< private >*/
+    DeviceState parent_obj;
+
+    /*< public >*/
+    char *slot;
+} CPUCore;
+
+#define CPU_CORE_SLOT_PROP "slot"
+
+#endif
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Fri,  4 Mar 2016 12:24:16 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> Add an abstract CPU core type that could be used by machines that want
> to define and hotplug CPUs in core granularity.
> 
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> ---
>  hw/cpu/Makefile.objs  |  1 +
>  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
>  3 files changed, 75 insertions(+)
>  create mode 100644 hw/cpu/core.c
>  create mode 100644 include/hw/cpu/core.h
> 
> diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> index 0954a18..942a4bb 100644
> --- a/hw/cpu/Makefile.objs
> +++ b/hw/cpu/Makefile.objs
> @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
>  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
>  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
>  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> +obj-y += core.o
>  
> diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> new file mode 100644
> index 0000000..d8caf37
> --- /dev/null
> +++ b/hw/cpu/core.c
> @@ -0,0 +1,44 @@
> +/*
> + * CPU core abstract device
> + *
> + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +#include "hw/cpu/core.h"
> +
> +static char *core_prop_get_slot(Object *obj, Error **errp)
> +{
> +    CPUCore *core = CPU_CORE(obj);
> +
> +    return g_strdup(core->slot);
> +}
> +
> +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> +{
> +    CPUCore *core = CPU_CORE(obj);
> +
> +    core->slot = g_strdup(val);
> +}
> +
> +static void cpu_core_instance_init(Object *obj)
> +{
> +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> +                            NULL);
> +}
> +
> +static const TypeInfo cpu_core_type_info = {
> +    .name = TYPE_CPU_CORE,
> +    .parent = TYPE_DEVICE,
> +    .abstract = true,
> +    .instance_size = sizeof(CPUCore),
> +    .instance_init = cpu_core_instance_init,
> +};
> +
> +static void cpu_core_register_types(void)
> +{
> +    type_register_static(&cpu_core_type_info);
> +}
> +
> +type_init(cpu_core_register_types)
> diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> new file mode 100644
> index 0000000..2daa724
> --- /dev/null
> +++ b/include/hw/cpu/core.h
> @@ -0,0 +1,30 @@
> +/*
> + * CPU core abstract device
> + *
> + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +#ifndef HW_CPU_CORE_H
> +#define HW_CPU_CORE_H
> +
> +#include "qemu/osdep.h"
> +#include "hw/qdev.h"
> +
> +#define TYPE_CPU_CORE "cpu-core"
> +
> +#define CPU_CORE(obj) \
> +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> +
> +typedef struct CPUCore {
> +    /*< private >*/
> +    DeviceState parent_obj;
> +
> +    /*< public >*/
> +    char *slot;
> +} CPUCore;
> +
> +#define CPU_CORE_SLOT_PROP "slot"
as it's generic property I'd rename to 'core' so it would fit all users


on top of that I'd add numeric 'threads' property to base class so
all derived cores would inherit it.

Then as easy integration with -smp threads=x, a machine could push
a global variable 'cpu-core.threads=[smp_threads]' which would
make every created cpu-core object to have threads set
at instance_init() time (device_init).

That way user won't have to specify 'threads=y' for every
  device_add spapr-core,core=x
as it will be taken from global property 'cpu-core.threads'
but if user wishes he/she still could override global by explicitly
providing thread property at device_add time:
  device_add spapr-core,core=x,threads=y

wrt this series it would mean, instead of creating threads in property
setter, delaying threads creation to core.realize() time,
but since realize is allowed to fail it should be fine do so.

> +
> +#endif

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Bharata B Rao]

On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:
> On Fri,  4 Mar 2016 12:24:16 +0530
> Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> 
> > Add an abstract CPU core type that could be used by machines that want
> > to define and hotplug CPUs in core granularity.
> > 
> > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > ---
> >  hw/cpu/Makefile.objs  |  1 +
> >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> >  3 files changed, 75 insertions(+)
> >  create mode 100644 hw/cpu/core.c
> >  create mode 100644 include/hw/cpu/core.h
> > 
> > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > index 0954a18..942a4bb 100644
> > --- a/hw/cpu/Makefile.objs
> > +++ b/hw/cpu/Makefile.objs
> > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > +obj-y += core.o
> >  
> > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > new file mode 100644
> > index 0000000..d8caf37
> > --- /dev/null
> > +++ b/hw/cpu/core.c
> > @@ -0,0 +1,44 @@
> > +/*
> > + * CPU core abstract device
> > + *
> > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +#include "hw/cpu/core.h"
> > +
> > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > +{
> > +    CPUCore *core = CPU_CORE(obj);
> > +
> > +    return g_strdup(core->slot);
> > +}
> > +
> > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > +{
> > +    CPUCore *core = CPU_CORE(obj);
> > +
> > +    core->slot = g_strdup(val);
> > +}
> > +
> > +static void cpu_core_instance_init(Object *obj)
> > +{
> > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > +                            NULL);
> > +}
> > +
> > +static const TypeInfo cpu_core_type_info = {
> > +    .name = TYPE_CPU_CORE,
> > +    .parent = TYPE_DEVICE,
> > +    .abstract = true,
> > +    .instance_size = sizeof(CPUCore),
> > +    .instance_init = cpu_core_instance_init,
> > +};
> > +
> > +static void cpu_core_register_types(void)
> > +{
> > +    type_register_static(&cpu_core_type_info);
> > +}
> > +
> > +type_init(cpu_core_register_types)
> > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > new file mode 100644
> > index 0000000..2daa724
> > --- /dev/null
> > +++ b/include/hw/cpu/core.h
> > @@ -0,0 +1,30 @@
> > +/*
> > + * CPU core abstract device
> > + *
> > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +#ifndef HW_CPU_CORE_H
> > +#define HW_CPU_CORE_H
> > +
> > +#include "qemu/osdep.h"
> > +#include "hw/qdev.h"
> > +
> > +#define TYPE_CPU_CORE "cpu-core"
> > +
> > +#define CPU_CORE(obj) \
> > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > +
> > +typedef struct CPUCore {
> > +    /*< private >*/
> > +    DeviceState parent_obj;
> > +
> > +    /*< public >*/
> > +    char *slot;
> > +} CPUCore;
> > +
> > +#define CPU_CORE_SLOT_PROP "slot"
> as it's generic property I'd rename to 'core' so it would fit all users

Ok. Also note that this is a string property which is associated with the
link name (string) that we created from machine object to this core. I think
it would be ideal if this becomes an interger  property in which case it
becomes easier to feed the core location into your CPUSlotProperties.core.

> on top of that I'd add numeric 'threads' property to base class so
> all derived cores would inherit it.
> 
> Then as easy integration with -smp threads=x, a machine could push
> a global variable 'cpu-core.threads=[smp_threads]' which would
> make every created cpu-core object to have threads set
> at instance_init() time (device_init).
> 
> That way user won't have to specify 'threads=y' for every
>   device_add spapr-core,core=x
> as it will be taken from global property 'cpu-core.threads'
> but if user wishes he/she still could override global by explicitly
> providing thread property at device_add time:
>   device_add spapr-core,core=x,threads=y
> 
> wrt this series it would mean, instead of creating threads in property
> setter, delaying threads creation to core.realize() time,
> but since realize is allowed to fail it should be fine do so.

Ok that would suit us as there are two properties on which thread creation
is dependent upon: nr_threads and cpu_model. If thread objects can be
created at core realize time, then we don't have to resort to the ugliness
of creating the threads from either of the property setters. I always
assumed that we shouldn't be creating objects from realize, but if that
is fine, it is good.

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Fri, 4 Mar 2016 16:32:53 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:
> > On Fri,  4 Mar 2016 12:24:16 +0530
> > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> >   
> > > Add an abstract CPU core type that could be used by machines that want
> > > to define and hotplug CPUs in core granularity.
> > > 
> > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > ---
> > >  hw/cpu/Makefile.objs  |  1 +
> > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > >  3 files changed, 75 insertions(+)
> > >  create mode 100644 hw/cpu/core.c
> > >  create mode 100644 include/hw/cpu/core.h
> > > 
> > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > index 0954a18..942a4bb 100644
> > > --- a/hw/cpu/Makefile.objs
> > > +++ b/hw/cpu/Makefile.objs
> > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > +obj-y += core.o
> > >  
> > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > new file mode 100644
> > > index 0000000..d8caf37
> > > --- /dev/null
> > > +++ b/hw/cpu/core.c
> > > @@ -0,0 +1,44 @@
> > > +/*
> > > + * CPU core abstract device
> > > + *
> > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > + * See the COPYING file in the top-level directory.
> > > + */
> > > +#include "hw/cpu/core.h"
> > > +
> > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > +{
> > > +    CPUCore *core = CPU_CORE(obj);
> > > +
> > > +    return g_strdup(core->slot);
> > > +}
> > > +
> > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > +{
> > > +    CPUCore *core = CPU_CORE(obj);
> > > +
> > > +    core->slot = g_strdup(val);
> > > +}
> > > +
> > > +static void cpu_core_instance_init(Object *obj)
> > > +{
> > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > +                            NULL);
> > > +}
> > > +
> > > +static const TypeInfo cpu_core_type_info = {
> > > +    .name = TYPE_CPU_CORE,
> > > +    .parent = TYPE_DEVICE,
> > > +    .abstract = true,
> > > +    .instance_size = sizeof(CPUCore),
> > > +    .instance_init = cpu_core_instance_init,
> > > +};
> > > +
> > > +static void cpu_core_register_types(void)
> > > +{
> > > +    type_register_static(&cpu_core_type_info);
> > > +}
> > > +
> > > +type_init(cpu_core_register_types)
> > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > new file mode 100644
> > > index 0000000..2daa724
> > > --- /dev/null
> > > +++ b/include/hw/cpu/core.h
> > > @@ -0,0 +1,30 @@
> > > +/*
> > > + * CPU core abstract device
> > > + *
> > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > + * See the COPYING file in the top-level directory.
> > > + */
> > > +#ifndef HW_CPU_CORE_H
> > > +#define HW_CPU_CORE_H
> > > +
> > > +#include "qemu/osdep.h"
> > > +#include "hw/qdev.h"
> > > +
> > > +#define TYPE_CPU_CORE "cpu-core"
> > > +
> > > +#define CPU_CORE(obj) \
> > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > +
> > > +typedef struct CPUCore {
> > > +    /*< private >*/
> > > +    DeviceState parent_obj;
> > > +
> > > +    /*< public >*/
> > > +    char *slot;
> > > +} CPUCore;
> > > +
> > > +#define CPU_CORE_SLOT_PROP "slot"  
> > as it's generic property I'd rename to 'core' so it would fit all users  
> 
> Ok. Also note that this is a string property which is associated with the
> link name (string) that we created from machine object to this core. I think
> it would be ideal if this becomes an interger  property in which case it
> becomes easier to feed the core location into your CPUSlotProperties.core.
agreed, it should be core number.


> > on top of that I'd add numeric 'threads' property to base class so
> > all derived cores would inherit it.
> > 
> > Then as easy integration with -smp threads=x, a machine could push
> > a global variable 'cpu-core.threads=[smp_threads]' which would
> > make every created cpu-core object to have threads set
> > at instance_init() time (device_init).
> > 
> > That way user won't have to specify 'threads=y' for every
> >   device_add spapr-core,core=x
> > as it will be taken from global property 'cpu-core.threads'
> > but if user wishes he/she still could override global by explicitly
> > providing thread property at device_add time:
> >   device_add spapr-core,core=x,threads=y
> > 
> > wrt this series it would mean, instead of creating threads in property
> > setter, delaying threads creation to core.realize() time,
> > but since realize is allowed to fail it should be fine do so.  
> 
> Ok that would suit us as there are two properties on which thread creation
> is dependent upon: nr_threads and cpu_model. If thread objects can be
> created at core realize time, then we don't have to resort to the ugliness
> of creating the threads from either of the property setters. I always
> assumed that we shouldn't be creating objects from realize, but if that
> is fine, it is good.
since realize is allowed to fail, it should be safe from hotplug pov
to create internal objects there, as far as proper cleanups are done
for failure path.

> 
> Regards,
> Bharata.
> 

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 7323 bytes --]

On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:
> On Fri, 4 Mar 2016 16:32:53 +0530
> Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> 
> > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:
> > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > >   
> > > > Add an abstract CPU core type that could be used by machines that want
> > > > to define and hotplug CPUs in core granularity.
> > > > 
> > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > ---
> > > >  hw/cpu/Makefile.objs  |  1 +
> > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > >  3 files changed, 75 insertions(+)
> > > >  create mode 100644 hw/cpu/core.c
> > > >  create mode 100644 include/hw/cpu/core.h
> > > > 
> > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > index 0954a18..942a4bb 100644
> > > > --- a/hw/cpu/Makefile.objs
> > > > +++ b/hw/cpu/Makefile.objs
> > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > +obj-y += core.o
> > > >  
> > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > new file mode 100644
> > > > index 0000000..d8caf37
> > > > --- /dev/null
> > > > +++ b/hw/cpu/core.c
> > > > @@ -0,0 +1,44 @@
> > > > +/*
> > > > + * CPU core abstract device
> > > > + *
> > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > + *
> > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > + * See the COPYING file in the top-level directory.
> > > > + */
> > > > +#include "hw/cpu/core.h"
> > > > +
> > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > +{
> > > > +    CPUCore *core = CPU_CORE(obj);
> > > > +
> > > > +    return g_strdup(core->slot);
> > > > +}
> > > > +
> > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > +{
> > > > +    CPUCore *core = CPU_CORE(obj);
> > > > +
> > > > +    core->slot = g_strdup(val);
> > > > +}
> > > > +
> > > > +static void cpu_core_instance_init(Object *obj)
> > > > +{
> > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > +                            NULL);
> > > > +}
> > > > +
> > > > +static const TypeInfo cpu_core_type_info = {
> > > > +    .name = TYPE_CPU_CORE,
> > > > +    .parent = TYPE_DEVICE,
> > > > +    .abstract = true,
> > > > +    .instance_size = sizeof(CPUCore),
> > > > +    .instance_init = cpu_core_instance_init,
> > > > +};
> > > > +
> > > > +static void cpu_core_register_types(void)
> > > > +{
> > > > +    type_register_static(&cpu_core_type_info);
> > > > +}
> > > > +
> > > > +type_init(cpu_core_register_types)
> > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > new file mode 100644
> > > > index 0000000..2daa724
> > > > --- /dev/null
> > > > +++ b/include/hw/cpu/core.h
> > > > @@ -0,0 +1,30 @@
> > > > +/*
> > > > + * CPU core abstract device
> > > > + *
> > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > + *
> > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > + * See the COPYING file in the top-level directory.
> > > > + */
> > > > +#ifndef HW_CPU_CORE_H
> > > > +#define HW_CPU_CORE_H
> > > > +
> > > > +#include "qemu/osdep.h"
> > > > +#include "hw/qdev.h"
> > > > +
> > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > +
> > > > +#define CPU_CORE(obj) \
> > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > +
> > > > +typedef struct CPUCore {
> > > > +    /*< private >*/
> > > > +    DeviceState parent_obj;
> > > > +
> > > > +    /*< public >*/
> > > > +    char *slot;
> > > > +} CPUCore;
> > > > +
> > > > +#define CPU_CORE_SLOT_PROP "slot"  
> > > as it's generic property I'd rename to 'core' so it would fit all users  
> > 
> > Ok. Also note that this is a string property which is associated with the
> > link name (string) that we created from machine object to this core. I think
> > it would be ideal if this becomes an interger  property in which case it
> > becomes easier to feed the core location into your CPUSlotProperties.core.
> agreed, it should be core number.

The slot stuff is continuing to confuse me a bit.  I see that we need
some kind of "address" value, but how best to do it is not clear to
me.

Changing this to an integer sounds like it's probably a good idea.
I'm a bit wary of just calling it "core" though.  Do all platforms
even necessarily have a core id?

I'm wondering if the addressing is something that needs to move the
the platform specific subtypes, while some other stuff can move to the
generic base type.

> > > on top of that I'd add numeric 'threads' property to base class so
> > > all derived cores would inherit it.
> > > 
> > > Then as easy integration with -smp threads=x, a machine could push
> > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > make every created cpu-core object to have threads set
> > > at instance_init() time (device_init).
> > > 
> > > That way user won't have to specify 'threads=y' for every
> > >   device_add spapr-core,core=x
> > > as it will be taken from global property 'cpu-core.threads'
> > > but if user wishes he/she still could override global by explicitly
> > > providing thread property at device_add time:
> > >   device_add spapr-core,core=x,threads=y
> > > 
> > > wrt this series it would mean, instead of creating threads in property
> > > setter, delaying threads creation to core.realize() time,
> > > but since realize is allowed to fail it should be fine do so.  
> > 
> > Ok that would suit us as there are two properties on which thread creation
> > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > created at core realize time, then we don't have to resort to the ugliness
> > of creating the threads from either of the property setters. I always
> > assumed that we shouldn't be creating objects from realize, but if that
> > is fine, it is good.
> since realize is allowed to fail, it should be safe from hotplug pov
> to create internal objects there, as far as proper cleanups are done
> for failure path.

Right, moving the "nr_threads" property to the base type seems like a
good idea to me.  Even for subtypes which have a fixed number of
threads we should be able to handle by just verifying and rejecting in
the subtype's realize.  And even for those core types, being able to read
the number of threads in a generic way is a good thing.

I'm not clear from the above if you're also intending to move at least
the adding of the threads as child properties is supposed to go into
the base type, but that also sounds like a good idea, again for
consistency.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Bharata B Rao]

On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:
> On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:
> > On Fri, 4 Mar 2016 16:32:53 +0530
> > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > 
> > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:
> > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > >   
> > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > to define and hotplug CPUs in core granularity.
> > > > > 
> > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > ---
> > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > >  3 files changed, 75 insertions(+)
> > > > >  create mode 100644 hw/cpu/core.c
> > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > 
> > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > index 0954a18..942a4bb 100644
> > > > > --- a/hw/cpu/Makefile.objs
> > > > > +++ b/hw/cpu/Makefile.objs
> > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > +obj-y += core.o
> > > > >  
> > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > new file mode 100644
> > > > > index 0000000..d8caf37
> > > > > --- /dev/null
> > > > > +++ b/hw/cpu/core.c
> > > > > @@ -0,0 +1,44 @@
> > > > > +/*
> > > > > + * CPU core abstract device
> > > > > + *
> > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > + * See the COPYING file in the top-level directory.
> > > > > + */
> > > > > +#include "hw/cpu/core.h"
> > > > > +
> > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > +{
> > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > +
> > > > > +    return g_strdup(core->slot);
> > > > > +}
> > > > > +
> > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > +{
> > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > +
> > > > > +    core->slot = g_strdup(val);
> > > > > +}
> > > > > +
> > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > +{
> > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > +                            NULL);
> > > > > +}
> > > > > +
> > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > +    .name = TYPE_CPU_CORE,
> > > > > +    .parent = TYPE_DEVICE,
> > > > > +    .abstract = true,
> > > > > +    .instance_size = sizeof(CPUCore),
> > > > > +    .instance_init = cpu_core_instance_init,
> > > > > +};
> > > > > +
> > > > > +static void cpu_core_register_types(void)
> > > > > +{
> > > > > +    type_register_static(&cpu_core_type_info);
> > > > > +}
> > > > > +
> > > > > +type_init(cpu_core_register_types)
> > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > new file mode 100644
> > > > > index 0000000..2daa724
> > > > > --- /dev/null
> > > > > +++ b/include/hw/cpu/core.h
> > > > > @@ -0,0 +1,30 @@
> > > > > +/*
> > > > > + * CPU core abstract device
> > > > > + *
> > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > + * See the COPYING file in the top-level directory.
> > > > > + */
> > > > > +#ifndef HW_CPU_CORE_H
> > > > > +#define HW_CPU_CORE_H
> > > > > +
> > > > > +#include "qemu/osdep.h"
> > > > > +#include "hw/qdev.h"
> > > > > +
> > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > +
> > > > > +#define CPU_CORE(obj) \
> > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > +
> > > > > +typedef struct CPUCore {
> > > > > +    /*< private >*/
> > > > > +    DeviceState parent_obj;
> > > > > +
> > > > > +    /*< public >*/
> > > > > +    char *slot;
> > > > > +} CPUCore;
> > > > > +
> > > > > +#define CPU_CORE_SLOT_PROP "slot"  
> > > > as it's generic property I'd rename to 'core' so it would fit all users  
> > > 
> > > Ok. Also note that this is a string property which is associated with the
> > > link name (string) that we created from machine object to this core. I think
> > > it would be ideal if this becomes an interger  property in which case it
> > > becomes easier to feed the core location into your CPUSlotProperties.core.
> > agreed, it should be core number.
> 
> The slot stuff is continuing to confuse me a bit.  I see that we need
> some kind of "address" value, but how best to do it is not clear to
> me.
> 
> Changing this to an integer sounds like it's probably a good idea.
> I'm a bit wary of just calling it "core" though.  Do all platforms
> even necessarily have a core id?
> 
> I'm wondering if the addressing is something that needs to move the
> the platform specific subtypes, while some other stuff can move to the
> generic base type.
> 
> > > > on top of that I'd add numeric 'threads' property to base class so
> > > > all derived cores would inherit it.
> > > > 
> > > > Then as easy integration with -smp threads=x, a machine could push
> > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > make every created cpu-core object to have threads set
> > > > at instance_init() time (device_init).
> > > > 
> > > > That way user won't have to specify 'threads=y' for every
> > > >   device_add spapr-core,core=x
> > > > as it will be taken from global property 'cpu-core.threads'
> > > > but if user wishes he/she still could override global by explicitly
> > > > providing thread property at device_add time:
> > > >   device_add spapr-core,core=x,threads=y
> > > > 
> > > > wrt this series it would mean, instead of creating threads in property
> > > > setter, delaying threads creation to core.realize() time,
> > > > but since realize is allowed to fail it should be fine do so.  
> > > 
> > > Ok that would suit us as there are two properties on which thread creation
> > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > created at core realize time, then we don't have to resort to the ugliness
> > > of creating the threads from either of the property setters. I always
> > > assumed that we shouldn't be creating objects from realize, but if that
> > > is fine, it is good.
> > since realize is allowed to fail, it should be safe from hotplug pov
> > to create internal objects there, as far as proper cleanups are done
> > for failure path.
> 
> Right, moving the "nr_threads" property to the base type seems like a
> good idea to me.

And we will also move the cpu_model property (now being tracked by
an ObjectClass pointer) to the base type ?

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Mon, 7 Mar 2016 14:01:55 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:
> > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:  
> > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > >   
> > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:  
> > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > >     
> > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > to define and hotplug CPUs in core granularity.
> > > > > > 
> > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > ---
> > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > >  3 files changed, 75 insertions(+)
> > > > > >  create mode 100644 hw/cpu/core.c
> > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > 
> > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > index 0954a18..942a4bb 100644
> > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > +obj-y += core.o
> > > > > >  
> > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > new file mode 100644
> > > > > > index 0000000..d8caf37
> > > > > > --- /dev/null
> > > > > > +++ b/hw/cpu/core.c
> > > > > > @@ -0,0 +1,44 @@
> > > > > > +/*
> > > > > > + * CPU core abstract device
> > > > > > + *
> > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > + *
> > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > + * See the COPYING file in the top-level directory.
> > > > > > + */
> > > > > > +#include "hw/cpu/core.h"
> > > > > > +
> > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > +{
> > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > +
> > > > > > +    return g_strdup(core->slot);
> > > > > > +}
> > > > > > +
> > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > +{
> > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > +
> > > > > > +    core->slot = g_strdup(val);
> > > > > > +}
> > > > > > +
> > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > +{
> > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > +                            NULL);
> > > > > > +}
> > > > > > +
> > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > +    .parent = TYPE_DEVICE,
> > > > > > +    .abstract = true,
> > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > +};
> > > > > > +
> > > > > > +static void cpu_core_register_types(void)
> > > > > > +{
> > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > +}
> > > > > > +
> > > > > > +type_init(cpu_core_register_types)
> > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > new file mode 100644
> > > > > > index 0000000..2daa724
> > > > > > --- /dev/null
> > > > > > +++ b/include/hw/cpu/core.h
> > > > > > @@ -0,0 +1,30 @@
> > > > > > +/*
> > > > > > + * CPU core abstract device
> > > > > > + *
> > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > + *
> > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > + * See the COPYING file in the top-level directory.
> > > > > > + */
> > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > +#define HW_CPU_CORE_H
> > > > > > +
> > > > > > +#include "qemu/osdep.h"
> > > > > > +#include "hw/qdev.h"
> > > > > > +
> > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > +
> > > > > > +#define CPU_CORE(obj) \
> > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > +
> > > > > > +typedef struct CPUCore {
> > > > > > +    /*< private >*/
> > > > > > +    DeviceState parent_obj;
> > > > > > +
> > > > > > +    /*< public >*/
> > > > > > +    char *slot;
> > > > > > +} CPUCore;
> > > > > > +
> > > > > > +#define CPU_CORE_SLOT_PROP "slot"    
> > > > > as it's generic property I'd rename to 'core' so it would fit all users    
> > > > 
> > > > Ok. Also note that this is a string property which is associated with the
> > > > link name (string) that we created from machine object to this core. I think
> > > > it would be ideal if this becomes an interger  property in which case it
> > > > becomes easier to feed the core location into your CPUSlotProperties.core.  
> > > agreed, it should be core number.  
> > 
> > The slot stuff is continuing to confuse me a bit.  I see that we need
> > some kind of "address" value, but how best to do it is not clear to
> > me.
> > 
> > Changing this to an integer sounds like it's probably a good idea.
> > I'm a bit wary of just calling it "core" though.  Do all platforms
> > even necessarily have a core id?
> > 
> > I'm wondering if the addressing is something that needs to move the
> > the platform specific subtypes, while some other stuff can move to the
> > generic base type.
> >   
> > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > all derived cores would inherit it.
> > > > > 
> > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > make every created cpu-core object to have threads set
> > > > > at instance_init() time (device_init).
> > > > > 
> > > > > That way user won't have to specify 'threads=y' for every
> > > > >   device_add spapr-core,core=x
> > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > but if user wishes he/she still could override global by explicitly
> > > > > providing thread property at device_add time:
> > > > >   device_add spapr-core,core=x,threads=y
> > > > > 
> > > > > wrt this series it would mean, instead of creating threads in property
> > > > > setter, delaying threads creation to core.realize() time,
> > > > > but since realize is allowed to fail it should be fine do so.    
> > > > 
> > > > Ok that would suit us as there are two properties on which thread creation
> > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > created at core realize time, then we don't have to resort to the ugliness
> > > > of creating the threads from either of the property setters. I always
> > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > is fine, it is good.  
> > > since realize is allowed to fail, it should be safe from hotplug pov
> > > to create internal objects there, as far as proper cleanups are done
> > > for failure path.  
> > 
> > Right, moving the "nr_threads" property to the base type seems like a
> > good idea to me.  
> 
> And we will also move the cpu_model property (now being tracked by
> an ObjectClass pointer) to the base type ?
I'm not sure that moving cpu_model to the base class is the right thing,
I'd keep it local to platform for now.

Could you have several spapr core types? One per CPU model?
That way you won't need to track cpu_model when using device_add.

> 
> Regards,
> Bharata.
> 

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 9093 bytes --]

On Mon, Mar 07, 2016 at 11:40:11AM +0100, Igor Mammedov wrote:
> On Mon, 7 Mar 2016 14:01:55 +0530
> Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> 
> > On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:
> > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:  
> > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > >   
> > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:  
> > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > >     
> > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > 
> > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > ---
> > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > >  3 files changed, 75 insertions(+)
> > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > 
> > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > index 0954a18..942a4bb 100644
> > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > +obj-y += core.o
> > > > > > >  
> > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > new file mode 100644
> > > > > > > index 0000000..d8caf37
> > > > > > > --- /dev/null
> > > > > > > +++ b/hw/cpu/core.c
> > > > > > > @@ -0,0 +1,44 @@
> > > > > > > +/*
> > > > > > > + * CPU core abstract device
> > > > > > > + *
> > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > + *
> > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > + */
> > > > > > > +#include "hw/cpu/core.h"
> > > > > > > +
> > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > +{
> > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > +
> > > > > > > +    return g_strdup(core->slot);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > +{
> > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > +
> > > > > > > +    core->slot = g_strdup(val);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > +{
> > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > +                            NULL);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > +    .abstract = true,
> > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > +};
> > > > > > > +
> > > > > > > +static void cpu_core_register_types(void)
> > > > > > > +{
> > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > +}
> > > > > > > +
> > > > > > > +type_init(cpu_core_register_types)
> > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > new file mode 100644
> > > > > > > index 0000000..2daa724
> > > > > > > --- /dev/null
> > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > @@ -0,0 +1,30 @@
> > > > > > > +/*
> > > > > > > + * CPU core abstract device
> > > > > > > + *
> > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > + *
> > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > + */
> > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > +#define HW_CPU_CORE_H
> > > > > > > +
> > > > > > > +#include "qemu/osdep.h"
> > > > > > > +#include "hw/qdev.h"
> > > > > > > +
> > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > +
> > > > > > > +#define CPU_CORE(obj) \
> > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > +
> > > > > > > +typedef struct CPUCore {
> > > > > > > +    /*< private >*/
> > > > > > > +    DeviceState parent_obj;
> > > > > > > +
> > > > > > > +    /*< public >*/
> > > > > > > +    char *slot;
> > > > > > > +} CPUCore;
> > > > > > > +
> > > > > > > +#define CPU_CORE_SLOT_PROP "slot"    
> > > > > > as it's generic property I'd rename to 'core' so it would fit all users    
> > > > > 
> > > > > Ok. Also note that this is a string property which is associated with the
> > > > > link name (string) that we created from machine object to this core. I think
> > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > becomes easier to feed the core location into your CPUSlotProperties.core.  
> > > > agreed, it should be core number.  
> > > 
> > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > some kind of "address" value, but how best to do it is not clear to
> > > me.
> > > 
> > > Changing this to an integer sounds like it's probably a good idea.
> > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > even necessarily have a core id?
> > > 
> > > I'm wondering if the addressing is something that needs to move the
> > > the platform specific subtypes, while some other stuff can move to the
> > > generic base type.
> > >   
> > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > all derived cores would inherit it.
> > > > > > 
> > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > make every created cpu-core object to have threads set
> > > > > > at instance_init() time (device_init).
> > > > > > 
> > > > > > That way user won't have to specify 'threads=y' for every
> > > > > >   device_add spapr-core,core=x
> > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > providing thread property at device_add time:
> > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > 
> > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > but since realize is allowed to fail it should be fine do so.    
> > > > > 
> > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > of creating the threads from either of the property setters. I always
> > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > is fine, it is good.  
> > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > to create internal objects there, as far as proper cleanups are done
> > > > for failure path.  
> > > 
> > > Right, moving the "nr_threads" property to the base type seems like a
> > > good idea to me.  
> > 
> > And we will also move the cpu_model property (now being tracked by
> > an ObjectClass pointer) to the base type ?
> I'm not sure that moving cpu_model to the base class is the right thing,
> I'd keep it local to platform for now.

I tend to agree, although I'm not sure that I could really explain why
:/

> Could you have several spapr core types? One per CPU model?
> That way you won't need to track cpu_model when using device_add.

We could in theory, but it would be pretty inconvenient.  Because this
is a paravirt platform, there really can't be any core-level
difference between them, and it would mean creating a fair batch of
core types for the various minor POWER7 and POWER8 variants - and
needing to update this whenever IBM makes a new version.  I suspect it
would also introduce more wrinkles in order to have a correct
"spapr-core-host" type matching the "HOST" cpu thread type.  Since KVM
(HV) only supports the HOST thread type, that's a fairly big issue.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Tue, 8 Mar 2016 14:57:10 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Mon, Mar 07, 2016 at 11:40:11AM +0100, Igor Mammedov wrote:
> > On Mon, 7 Mar 2016 14:01:55 +0530
> > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> >   
> > > On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:  
> > > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:    
> > > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > >     
> > > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:    
> > > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > >       
> > > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > > 
> > > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > ---
> > > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > > >  3 files changed, 75 insertions(+)
> > > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > > 
> > > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > > index 0954a18..942a4bb 100644
> > > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > > +obj-y += core.o
> > > > > > > >  
> > > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000..d8caf37
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/hw/cpu/core.c
> > > > > > > > @@ -0,0 +1,44 @@
> > > > > > > > +/*
> > > > > > > > + * CPU core abstract device
> > > > > > > > + *
> > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > + *
> > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > + */
> > > > > > > > +#include "hw/cpu/core.h"
> > > > > > > > +
> > > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > > +{
> > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > +
> > > > > > > > +    return g_strdup(core->slot);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > > +{
> > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > +
> > > > > > > > +    core->slot = g_strdup(val);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > > +{
> > > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > > +                            NULL);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > > +    .abstract = true,
> > > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +static void cpu_core_register_types(void)
> > > > > > > > +{
> > > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +type_init(cpu_core_register_types)
> > > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000..2daa724
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > > @@ -0,0 +1,30 @@
> > > > > > > > +/*
> > > > > > > > + * CPU core abstract device
> > > > > > > > + *
> > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > + *
> > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > + */
> > > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > > +#define HW_CPU_CORE_H
> > > > > > > > +
> > > > > > > > +#include "qemu/osdep.h"
> > > > > > > > +#include "hw/qdev.h"
> > > > > > > > +
> > > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > > +
> > > > > > > > +#define CPU_CORE(obj) \
> > > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > > +
> > > > > > > > +typedef struct CPUCore {
> > > > > > > > +    /*< private >*/
> > > > > > > > +    DeviceState parent_obj;
> > > > > > > > +
> > > > > > > > +    /*< public >*/
> > > > > > > > +    char *slot;
> > > > > > > > +} CPUCore;
> > > > > > > > +
> > > > > > > > +#define CPU_CORE_SLOT_PROP "slot"      
> > > > > > > as it's generic property I'd rename to 'core' so it would fit all users      
> > > > > > 
> > > > > > Ok. Also note that this is a string property which is associated with the
> > > > > > link name (string) that we created from machine object to this core. I think
> > > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > > becomes easier to feed the core location into your CPUSlotProperties.core.    
> > > > > agreed, it should be core number.    
> > > > 
> > > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > > some kind of "address" value, but how best to do it is not clear to
> > > > me.
> > > > 
> > > > Changing this to an integer sounds like it's probably a good idea.
> > > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > > even necessarily have a core id?
> > > > 
> > > > I'm wondering if the addressing is something that needs to move the
> > > > the platform specific subtypes, while some other stuff can move to the
> > > > generic base type.
> > > >     
> > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > all derived cores would inherit it.
> > > > > > > 
> > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > make every created cpu-core object to have threads set
> > > > > > > at instance_init() time (device_init).
> > > > > > > 
> > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > >   device_add spapr-core,core=x
> > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > providing thread property at device_add time:
> > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > 
> > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > but since realize is allowed to fail it should be fine do so.      
> > > > > > 
> > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > of creating the threads from either of the property setters. I always
> > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > is fine, it is good.    
> > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > to create internal objects there, as far as proper cleanups are done
> > > > > for failure path.    
> > > > 
> > > > Right, moving the "nr_threads" property to the base type seems like a
> > > > good idea to me.    
> > > 
> > > And we will also move the cpu_model property (now being tracked by
> > > an ObjectClass pointer) to the base type ?  
> > I'm not sure that moving cpu_model to the base class is the right thing,
> > I'd keep it local to platform for now.  
> 
> I tend to agree, although I'm not sure that I could really explain why
> :/
> 
> > Could you have several spapr core types? One per CPU model?
> > That way you won't need to track cpu_model when using device_add.  
> 
> We could in theory, but it would be pretty inconvenient.  Because this
> is a paravirt platform, there really can't be any core-level
> difference between them, and it would mean creating a fair batch of
> core types for the various minor POWER7 and POWER8 variants - and
> needing to update this whenever IBM makes a new version.  I suspect it
> would also introduce more wrinkles in order to have a correct
> "spapr-core-host" type matching the "HOST" cpu thread type.  Since KVM
> (HV) only supports the HOST thread type, that's a fairly big issue.
Welcome to x86 world, that's roughly what we have there.

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 10350 bytes --]

On Tue, Mar 08, 2016 at 10:11:17AM +0100, Igor Mammedov wrote:
> On Tue, 8 Mar 2016 14:57:10 +1100
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Mon, Mar 07, 2016 at 11:40:11AM +0100, Igor Mammedov wrote:
> > > On Mon, 7 Mar 2016 14:01:55 +0530
> > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > >   
> > > > On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:  
> > > > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:    
> > > > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > >     
> > > > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:    
> > > > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > > >       
> > > > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > ---
> > > > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > > > >  3 files changed, 75 insertions(+)
> > > > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > > > 
> > > > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > > > index 0954a18..942a4bb 100644
> > > > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > > > +obj-y += core.o
> > > > > > > > >  
> > > > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000..d8caf37
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/hw/cpu/core.c
> > > > > > > > > @@ -0,0 +1,44 @@
> > > > > > > > > +/*
> > > > > > > > > + * CPU core abstract device
> > > > > > > > > + *
> > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > + *
> > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > + */
> > > > > > > > > +#include "hw/cpu/core.h"
> > > > > > > > > +
> > > > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > > > +{
> > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > +
> > > > > > > > > +    return g_strdup(core->slot);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > > > +{
> > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > +
> > > > > > > > > +    core->slot = g_strdup(val);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > > > +{
> > > > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > > > +                            NULL);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > > > +    .abstract = true,
> > > > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > > +static void cpu_core_register_types(void)
> > > > > > > > > +{
> > > > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +type_init(cpu_core_register_types)
> > > > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000..2daa724
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > > > @@ -0,0 +1,30 @@
> > > > > > > > > +/*
> > > > > > > > > + * CPU core abstract device
> > > > > > > > > + *
> > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > + *
> > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > + */
> > > > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > > > +#define HW_CPU_CORE_H
> > > > > > > > > +
> > > > > > > > > +#include "qemu/osdep.h"
> > > > > > > > > +#include "hw/qdev.h"
> > > > > > > > > +
> > > > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > > > +
> > > > > > > > > +#define CPU_CORE(obj) \
> > > > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > > > +
> > > > > > > > > +typedef struct CPUCore {
> > > > > > > > > +    /*< private >*/
> > > > > > > > > +    DeviceState parent_obj;
> > > > > > > > > +
> > > > > > > > > +    /*< public >*/
> > > > > > > > > +    char *slot;
> > > > > > > > > +} CPUCore;
> > > > > > > > > +
> > > > > > > > > +#define CPU_CORE_SLOT_PROP "slot"      
> > > > > > > > as it's generic property I'd rename to 'core' so it would fit all users      
> > > > > > > 
> > > > > > > Ok. Also note that this is a string property which is associated with the
> > > > > > > link name (string) that we created from machine object to this core. I think
> > > > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > > > becomes easier to feed the core location into your CPUSlotProperties.core.    
> > > > > > agreed, it should be core number.    
> > > > > 
> > > > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > > > some kind of "address" value, but how best to do it is not clear to
> > > > > me.
> > > > > 
> > > > > Changing this to an integer sounds like it's probably a good idea.
> > > > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > > > even necessarily have a core id?
> > > > > 
> > > > > I'm wondering if the addressing is something that needs to move the
> > > > > the platform specific subtypes, while some other stuff can move to the
> > > > > generic base type.
> > > > >     
> > > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > > all derived cores would inherit it.
> > > > > > > > 
> > > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > > make every created cpu-core object to have threads set
> > > > > > > > at instance_init() time (device_init).
> > > > > > > > 
> > > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > > >   device_add spapr-core,core=x
> > > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > > providing thread property at device_add time:
> > > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > > 
> > > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > > but since realize is allowed to fail it should be fine do so.      
> > > > > > > 
> > > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > > of creating the threads from either of the property setters. I always
> > > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > > is fine, it is good.    
> > > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > > to create internal objects there, as far as proper cleanups are done
> > > > > > for failure path.    
> > > > > 
> > > > > Right, moving the "nr_threads" property to the base type seems like a
> > > > > good idea to me.    
> > > > 
> > > > And we will also move the cpu_model property (now being tracked by
> > > > an ObjectClass pointer) to the base type ?  
> > > I'm not sure that moving cpu_model to the base class is the right thing,
> > > I'd keep it local to platform for now.  
> > 
> > I tend to agree, although I'm not sure that I could really explain why
> > :/
> > 
> > > Could you have several spapr core types? One per CPU model?
> > > That way you won't need to track cpu_model when using device_add.  
> > 
> > We could in theory, but it would be pretty inconvenient.  Because this
> > is a paravirt platform, there really can't be any core-level
> > difference between them, and it would mean creating a fair batch of
> > core types for the various minor POWER7 and POWER8 variants - and
> > needing to update this whenever IBM makes a new version.  I suspect it
> > would also introduce more wrinkles in order to have a correct
> > "spapr-core-host" type matching the "HOST" cpu thread type.  Since KVM
> > (HV) only supports the HOST thread type, that's a fairly big issue.
> Welcome to x86 world, that's roughly what we have there.

I don't really follow you.  x86 doesn't have core devices at all for
the moment.

What I'm saying here is that using different core subtypes for every
cpu subtype on power would mean 2 types for each minor variant, rather
than just 1.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Wed, 9 Mar 2016 13:55:51 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Tue, Mar 08, 2016 at 10:11:17AM +0100, Igor Mammedov wrote:
> > On Tue, 8 Mar 2016 14:57:10 +1100
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > On Mon, Mar 07, 2016 at 11:40:11AM +0100, Igor Mammedov wrote:  
> > > > On Mon, 7 Mar 2016 14:01:55 +0530
> > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > >     
> > > > > On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:    
> > > > > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:      
> > > > > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > >       
> > > > > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:      
> > > > > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > > > >         
> > > > > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > ---
> > > > > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > > > > >  3 files changed, 75 insertions(+)
> > > > > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > > > > 
> > > > > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > > > > index 0954a18..942a4bb 100644
> > > > > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > > > > +obj-y += core.o
> > > > > > > > > >  
> > > > > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000..d8caf37
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/hw/cpu/core.c
> > > > > > > > > > @@ -0,0 +1,44 @@
> > > > > > > > > > +/*
> > > > > > > > > > + * CPU core abstract device
> > > > > > > > > > + *
> > > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > + *
> > > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > > + */
> > > > > > > > > > +#include "hw/cpu/core.h"
> > > > > > > > > > +
> > > > > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > > > > +{
> > > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > > +
> > > > > > > > > > +    return g_strdup(core->slot);
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > > > > +{
> > > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > > +
> > > > > > > > > > +    core->slot = g_strdup(val);
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > > > > +{
> > > > > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > > > > +                            NULL);
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > > > > +    .abstract = true,
> > > > > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > > > > +};
> > > > > > > > > > +
> > > > > > > > > > +static void cpu_core_register_types(void)
> > > > > > > > > > +{
> > > > > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +type_init(cpu_core_register_types)
> > > > > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000..2daa724
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > > > > @@ -0,0 +1,30 @@
> > > > > > > > > > +/*
> > > > > > > > > > + * CPU core abstract device
> > > > > > > > > > + *
> > > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > + *
> > > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > > + */
> > > > > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > > > > +#define HW_CPU_CORE_H
> > > > > > > > > > +
> > > > > > > > > > +#include "qemu/osdep.h"
> > > > > > > > > > +#include "hw/qdev.h"
> > > > > > > > > > +
> > > > > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > > > > +
> > > > > > > > > > +#define CPU_CORE(obj) \
> > > > > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > > > > +
> > > > > > > > > > +typedef struct CPUCore {
> > > > > > > > > > +    /*< private >*/
> > > > > > > > > > +    DeviceState parent_obj;
> > > > > > > > > > +
> > > > > > > > > > +    /*< public >*/
> > > > > > > > > > +    char *slot;
> > > > > > > > > > +} CPUCore;
> > > > > > > > > > +
> > > > > > > > > > +#define CPU_CORE_SLOT_PROP "slot"        
> > > > > > > > > as it's generic property I'd rename to 'core' so it would fit all users        
> > > > > > > > 
> > > > > > > > Ok. Also note that this is a string property which is associated with the
> > > > > > > > link name (string) that we created from machine object to this core. I think
> > > > > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > > > > becomes easier to feed the core location into your CPUSlotProperties.core.      
> > > > > > > agreed, it should be core number.      
> > > > > > 
> > > > > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > > > > some kind of "address" value, but how best to do it is not clear to
> > > > > > me.
> > > > > > 
> > > > > > Changing this to an integer sounds like it's probably a good idea.
> > > > > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > > > > even necessarily have a core id?
> > > > > > 
> > > > > > I'm wondering if the addressing is something that needs to move the
> > > > > > the platform specific subtypes, while some other stuff can move to the
> > > > > > generic base type.
> > > > > >       
> > > > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > > > all derived cores would inherit it.
> > > > > > > > > 
> > > > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > > > make every created cpu-core object to have threads set
> > > > > > > > > at instance_init() time (device_init).
> > > > > > > > > 
> > > > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > > > >   device_add spapr-core,core=x
> > > > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > > > providing thread property at device_add time:
> > > > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > > > 
> > > > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > > > but since realize is allowed to fail it should be fine do so.        
> > > > > > > > 
> > > > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > > > of creating the threads from either of the property setters. I always
> > > > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > > > is fine, it is good.      
> > > > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > > > to create internal objects there, as far as proper cleanups are done
> > > > > > > for failure path.      
> > > > > > 
> > > > > > Right, moving the "nr_threads" property to the base type seems like a
> > > > > > good idea to me.      
> > > > > 
> > > > > And we will also move the cpu_model property (now being tracked by
> > > > > an ObjectClass pointer) to the base type ?    
> > > > I'm not sure that moving cpu_model to the base class is the right thing,
> > > > I'd keep it local to platform for now.    
> > > 
> > > I tend to agree, although I'm not sure that I could really explain why
> > > :/
> > >   
> > > > Could you have several spapr core types? One per CPU model?
> > > > That way you won't need to track cpu_model when using device_add.    
> > > 
> > > We could in theory, but it would be pretty inconvenient.  Because this
> > > is a paravirt platform, there really can't be any core-level
> > > difference between them, and it would mean creating a fair batch of
> > > core types for the various minor POWER7 and POWER8 variants - and
> > > needing to update this whenever IBM makes a new version.  I suspect it
> > > would also introduce more wrinkles in order to have a correct
> > > "spapr-core-host" type matching the "HOST" cpu thread type.  Since KVM
> > > (HV) only supports the HOST thread type, that's a fairly big issue.  
> > Welcome to x86 world, that's roughly what we have there.  
> 
> I don't really follow you.  x86 doesn't have core devices at all for
> the moment.
> 
> What I'm saying here is that using different core subtypes for every
> cpu subtype on power would mean 2 types for each minor variant, rather
> than just 1.
I think the same applies to cpu_model and transitioning CPUs to -device.
X86 also has a bunch of cpu_model-s which have some minor variations
but it still generates a type per cpu_model.
If some day we are to implement socket objects for x86 that would also
mean to have a socket types per each cpu model.
As I understand cpu_model is a legacy option which should translate to
a corresponding QOM type (CPU device) which could be used with
-device/device_add.

As analogy, QEMU has legacy -net model=foo[1234] option, but when network cards
were converted to -device interface that in the end became a set of QOM types
like -device foo[1234], it was easier in case of network cards as
they where a separate devices models to begin with, the thing to note
here is that they weren't converted to a single 'network_card' type with
'model' property.

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 12752 bytes --]

On Wed, Mar 09, 2016 at 11:32:28AM +0100, Igor Mammedov wrote:
> On Wed, 9 Mar 2016 13:55:51 +1100
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Tue, Mar 08, 2016 at 10:11:17AM +0100, Igor Mammedov wrote:
> > > On Tue, 8 Mar 2016 14:57:10 +1100
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > >   
> > > > On Mon, Mar 07, 2016 at 11:40:11AM +0100, Igor Mammedov wrote:  
> > > > > On Mon, 7 Mar 2016 14:01:55 +0530
> > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > >     
> > > > > > On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:    
> > > > > > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:      
> > > > > > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > > >       
> > > > > > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:      
> > > > > > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > > > > >         
> > > > > > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > > > > > 
> > > > > > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > > > > > >  3 files changed, 75 insertions(+)
> > > > > > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > > > > > 
> > > > > > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > > > > > index 0954a18..942a4bb 100644
> > > > > > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > > > > > +obj-y += core.o
> > > > > > > > > > >  
> > > > > > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 0000000..d8caf37
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/hw/cpu/core.c
> > > > > > > > > > > @@ -0,0 +1,44 @@
> > > > > > > > > > > +/*
> > > > > > > > > > > + * CPU core abstract device
> > > > > > > > > > > + *
> > > > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > > + *
> > > > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > > > + */
> > > > > > > > > > > +#include "hw/cpu/core.h"
> > > > > > > > > > > +
> > > > > > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > > > > > +{
> > > > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > > > +
> > > > > > > > > > > +    return g_strdup(core->slot);
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > > > > > +{
> > > > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > > > +
> > > > > > > > > > > +    core->slot = g_strdup(val);
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > > > > > +{
> > > > > > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > > > > > +                            NULL);
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > > > > > +    .abstract = true,
> > > > > > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > > > > > +};
> > > > > > > > > > > +
> > > > > > > > > > > +static void cpu_core_register_types(void)
> > > > > > > > > > > +{
> > > > > > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +type_init(cpu_core_register_types)
> > > > > > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 0000000..2daa724
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > > > > > @@ -0,0 +1,30 @@
> > > > > > > > > > > +/*
> > > > > > > > > > > + * CPU core abstract device
> > > > > > > > > > > + *
> > > > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > > + *
> > > > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > > > + */
> > > > > > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > > > > > +#define HW_CPU_CORE_H
> > > > > > > > > > > +
> > > > > > > > > > > +#include "qemu/osdep.h"
> > > > > > > > > > > +#include "hw/qdev.h"
> > > > > > > > > > > +
> > > > > > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > > > > > +
> > > > > > > > > > > +#define CPU_CORE(obj) \
> > > > > > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > > > > > +
> > > > > > > > > > > +typedef struct CPUCore {
> > > > > > > > > > > +    /*< private >*/
> > > > > > > > > > > +    DeviceState parent_obj;
> > > > > > > > > > > +
> > > > > > > > > > > +    /*< public >*/
> > > > > > > > > > > +    char *slot;
> > > > > > > > > > > +} CPUCore;
> > > > > > > > > > > +
> > > > > > > > > > > +#define CPU_CORE_SLOT_PROP "slot"        
> > > > > > > > > > as it's generic property I'd rename to 'core' so it would fit all users        
> > > > > > > > > 
> > > > > > > > > Ok. Also note that this is a string property which is associated with the
> > > > > > > > > link name (string) that we created from machine object to this core. I think
> > > > > > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > > > > > becomes easier to feed the core location into your CPUSlotProperties.core.      
> > > > > > > > agreed, it should be core number.      
> > > > > > > 
> > > > > > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > > > > > some kind of "address" value, but how best to do it is not clear to
> > > > > > > me.
> > > > > > > 
> > > > > > > Changing this to an integer sounds like it's probably a good idea.
> > > > > > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > > > > > even necessarily have a core id?
> > > > > > > 
> > > > > > > I'm wondering if the addressing is something that needs to move the
> > > > > > > the platform specific subtypes, while some other stuff can move to the
> > > > > > > generic base type.
> > > > > > >       
> > > > > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > > > > all derived cores would inherit it.
> > > > > > > > > > 
> > > > > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > > > > make every created cpu-core object to have threads set
> > > > > > > > > > at instance_init() time (device_init).
> > > > > > > > > > 
> > > > > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > > > > >   device_add spapr-core,core=x
> > > > > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > > > > providing thread property at device_add time:
> > > > > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > > > > 
> > > > > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > > > > but since realize is allowed to fail it should be fine do so.        
> > > > > > > > > 
> > > > > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > > > > of creating the threads from either of the property setters. I always
> > > > > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > > > > is fine, it is good.      
> > > > > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > > > > to create internal objects there, as far as proper cleanups are done
> > > > > > > > for failure path.      
> > > > > > > 
> > > > > > > Right, moving the "nr_threads" property to the base type seems like a
> > > > > > > good idea to me.      
> > > > > > 
> > > > > > And we will also move the cpu_model property (now being tracked by
> > > > > > an ObjectClass pointer) to the base type ?    
> > > > > I'm not sure that moving cpu_model to the base class is the right thing,
> > > > > I'd keep it local to platform for now.    
> > > > 
> > > > I tend to agree, although I'm not sure that I could really explain why
> > > > :/
> > > >   
> > > > > Could you have several spapr core types? One per CPU model?
> > > > > That way you won't need to track cpu_model when using device_add.    
> > > > 
> > > > We could in theory, but it would be pretty inconvenient.  Because this
> > > > is a paravirt platform, there really can't be any core-level
> > > > difference between them, and it would mean creating a fair batch of
> > > > core types for the various minor POWER7 and POWER8 variants - and
> > > > needing to update this whenever IBM makes a new version.  I suspect it
> > > > would also introduce more wrinkles in order to have a correct
> > > > "spapr-core-host" type matching the "HOST" cpu thread type.  Since KVM
> > > > (HV) only supports the HOST thread type, that's a fairly big issue.  
> > > Welcome to x86 world, that's roughly what we have there.  
> > 
> > I don't really follow you.  x86 doesn't have core devices at all for
> > the moment.
> > 
> > What I'm saying here is that using different core subtypes for every
> > cpu subtype on power would mean 2 types for each minor variant, rather
> > than just 1.
> I think the same applies to cpu_model and transitioning CPUs to -device.
> X86 also has a bunch of cpu_model-s which have some minor variations
> but it still generates a type per cpu_model.
> If some day we are to implement socket objects for x86 that would also
> mean to have a socket types per each cpu model.

Hmm.  I guess.

What concerns me is the possible combinatorial explosion of # cpu
models * # of machine types leading to an enormous number of
core/socket types.

The other thing is that the platform specific core types belong with
the machine type code, which means they can't naturally use macro
magic or whatever to generate them in parallel with the thread device
types in the core target-ppc code.

> As I understand cpu_model is a legacy option which should translate to
> a corresponding QOM type (CPU device) which could be used with
> -device/device_add.

Sure, but the "cpu_model" parameter could be "thread type name" just
as easily.

> As analogy, QEMU has legacy -net model=foo[1234] option, but when network cards
> were converted to -device interface that in the end became a set of QOM types
> like -device foo[1234], it was easier in case of network cards as
> they where a separate devices models to begin with, the thing to note
> here is that they weren't converted to a single 'network_card' type with
> 'model' property.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Thu, 10 Mar 2016 16:04:29 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, Mar 09, 2016 at 11:32:28AM +0100, Igor Mammedov wrote:
> > On Wed, 9 Mar 2016 13:55:51 +1100
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > On Tue, Mar 08, 2016 at 10:11:17AM +0100, Igor Mammedov wrote:  
> > > > On Tue, 8 Mar 2016 14:57:10 +1100
> > > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > > >     
> > > > > On Mon, Mar 07, 2016 at 11:40:11AM +0100, Igor Mammedov wrote:    
> > > > > > On Mon, 7 Mar 2016 14:01:55 +0530
> > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > >       
> > > > > > > On Mon, Mar 07, 2016 at 02:36:55PM +1100, David Gibson wrote:      
> > > > > > > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:        
> > > > > > > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > > > >         
> > > > > > > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:        
> > > > > > > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > > > > > > >           
> > > > > > > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > > > > > > 
> > > > > > > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > > > ---
> > > > > > > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > > > > > > >  3 files changed, 75 insertions(+)
> > > > > > > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > > > > > > 
> > > > > > > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > > > > > > index 0954a18..942a4bb 100644
> > > > > > > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > > > > > > +obj-y += core.o
> > > > > > > > > > > >  
> > > > > > > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > index 0000000..d8caf37
> > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > +++ b/hw/cpu/core.c
> > > > > > > > > > > > @@ -0,0 +1,44 @@
> > > > > > > > > > > > +/*
> > > > > > > > > > > > + * CPU core abstract device
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > > > > + */
> > > > > > > > > > > > +#include "hw/cpu/core.h"
> > > > > > > > > > > > +
> > > > > > > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > > > > +
> > > > > > > > > > > > +    return g_strdup(core->slot);
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > > > > > > +
> > > > > > > > > > > > +    core->slot = g_strdup(val);
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > > > > > > +                            NULL);
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > > > > > > +    .abstract = true,
> > > > > > > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > > > > > > +};
> > > > > > > > > > > > +
> > > > > > > > > > > > +static void cpu_core_register_types(void)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > > +type_init(cpu_core_register_types)
> > > > > > > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > index 0000000..2daa724
> > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > > > > > > @@ -0,0 +1,30 @@
> > > > > > > > > > > > +/*
> > > > > > > > > > > > + * CPU core abstract device
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > > > > > > + */
> > > > > > > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > > > > > > +#define HW_CPU_CORE_H
> > > > > > > > > > > > +
> > > > > > > > > > > > +#include "qemu/osdep.h"
> > > > > > > > > > > > +#include "hw/qdev.h"
> > > > > > > > > > > > +
> > > > > > > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > > > > > > +
> > > > > > > > > > > > +#define CPU_CORE(obj) \
> > > > > > > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > > > > > > +
> > > > > > > > > > > > +typedef struct CPUCore {
> > > > > > > > > > > > +    /*< private >*/
> > > > > > > > > > > > +    DeviceState parent_obj;
> > > > > > > > > > > > +
> > > > > > > > > > > > +    /*< public >*/
> > > > > > > > > > > > +    char *slot;
> > > > > > > > > > > > +} CPUCore;
> > > > > > > > > > > > +
> > > > > > > > > > > > +#define CPU_CORE_SLOT_PROP "slot"          
> > > > > > > > > > > as it's generic property I'd rename to 'core' so it would fit all users          
> > > > > > > > > > 
> > > > > > > > > > Ok. Also note that this is a string property which is associated with the
> > > > > > > > > > link name (string) that we created from machine object to this core. I think
> > > > > > > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > > > > > > becomes easier to feed the core location into your CPUSlotProperties.core.        
> > > > > > > > > agreed, it should be core number.        
> > > > > > > > 
> > > > > > > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > > > > > > some kind of "address" value, but how best to do it is not clear to
> > > > > > > > me.
> > > > > > > > 
> > > > > > > > Changing this to an integer sounds like it's probably a good idea.
> > > > > > > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > > > > > > even necessarily have a core id?
> > > > > > > > 
> > > > > > > > I'm wondering if the addressing is something that needs to move the
> > > > > > > > the platform specific subtypes, while some other stuff can move to the
> > > > > > > > generic base type.
> > > > > > > >         
> > > > > > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > > > > > all derived cores would inherit it.
> > > > > > > > > > > 
> > > > > > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > > > > > make every created cpu-core object to have threads set
> > > > > > > > > > > at instance_init() time (device_init).
> > > > > > > > > > > 
> > > > > > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > > > > > >   device_add spapr-core,core=x
> > > > > > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > > > > > providing thread property at device_add time:
> > > > > > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > > > > > 
> > > > > > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > > > > > but since realize is allowed to fail it should be fine do so.          
> > > > > > > > > > 
> > > > > > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > > > > > of creating the threads from either of the property setters. I always
> > > > > > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > > > > > is fine, it is good.        
> > > > > > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > > > > > to create internal objects there, as far as proper cleanups are done
> > > > > > > > > for failure path.        
> > > > > > > > 
> > > > > > > > Right, moving the "nr_threads" property to the base type seems like a
> > > > > > > > good idea to me.        
> > > > > > > 
> > > > > > > And we will also move the cpu_model property (now being tracked by
> > > > > > > an ObjectClass pointer) to the base type ?      
> > > > > > I'm not sure that moving cpu_model to the base class is the right thing,
> > > > > > I'd keep it local to platform for now.      
> > > > > 
> > > > > I tend to agree, although I'm not sure that I could really explain why
> > > > > :/
> > > > >     
> > > > > > Could you have several spapr core types? One per CPU model?
> > > > > > That way you won't need to track cpu_model when using device_add.      
> > > > > 
> > > > > We could in theory, but it would be pretty inconvenient.  Because this
> > > > > is a paravirt platform, there really can't be any core-level
> > > > > difference between them, and it would mean creating a fair batch of
> > > > > core types for the various minor POWER7 and POWER8 variants - and
> > > > > needing to update this whenever IBM makes a new version.  I suspect it
> > > > > would also introduce more wrinkles in order to have a correct
> > > > > "spapr-core-host" type matching the "HOST" cpu thread type.  Since KVM
> > > > > (HV) only supports the HOST thread type, that's a fairly big issue.    
> > > > Welcome to x86 world, that's roughly what we have there.    
> > > 
> > > I don't really follow you.  x86 doesn't have core devices at all for
> > > the moment.
> > > 
> > > What I'm saying here is that using different core subtypes for every
> > > cpu subtype on power would mean 2 types for each minor variant, rather
> > > than just 1.  
> > I think the same applies to cpu_model and transitioning CPUs to -device.
> > X86 also has a bunch of cpu_model-s which have some minor variations
> > but it still generates a type per cpu_model.
> > If some day we are to implement socket objects for x86 that would also
> > mean to have a socket types per each cpu model.  
> 
> Hmm.  I guess.
> 
> What concerns me is the possible combinatorial explosion of # cpu
> models * # of machine types leading to an enormous number of
> core/socket types.
>
> The other thing is that the platform specific core types belong with
> the machine type code, which means they can't naturally use macro
> magic or whatever to generate them in parallel with the thread device
> types in the core target-ppc code.
on x86 when introducing a new feature or changing behavior 
of existing cpu_model, cpu type stays the same but we amend some of
it's properties using compat machine infrastructure.
Though I'm not sure if it could help in SPAPR case.
 

> > As I understand cpu_model is a legacy option which should translate to
> > a corresponding QOM type (CPU device) which could be used with
> > -device/device_add.  
> 
> Sure, but the "cpu_model" parameter could be "thread type name" just
> as easily.
> 
> > As analogy, QEMU has legacy -net model=foo[1234] option, but when network cards
> > were converted to -device interface that in the end became a set of QOM types
> > like -device foo[1234], it was easier in case of network cards as
> > they where a separate devices models to begin with, the thing to note
> > here is that they weren't converted to a single 'network_card' type with
> > 'model' property.  
> 

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Mon, 7 Mar 2016 14:36:55 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:
> > On Fri, 4 Mar 2016 16:32:53 +0530
> > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> >   
> > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:  
> > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > >     
> > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > to define and hotplug CPUs in core granularity.
> > > > > 
> > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > ---
> > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > >  3 files changed, 75 insertions(+)
> > > > >  create mode 100644 hw/cpu/core.c
> > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > 
> > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > index 0954a18..942a4bb 100644
> > > > > --- a/hw/cpu/Makefile.objs
> > > > > +++ b/hw/cpu/Makefile.objs
> > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > +obj-y += core.o
> > > > >  
> > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > new file mode 100644
> > > > > index 0000000..d8caf37
> > > > > --- /dev/null
> > > > > +++ b/hw/cpu/core.c
> > > > > @@ -0,0 +1,44 @@
> > > > > +/*
> > > > > + * CPU core abstract device
> > > > > + *
> > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > + * See the COPYING file in the top-level directory.
> > > > > + */
> > > > > +#include "hw/cpu/core.h"
> > > > > +
> > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > +{
> > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > +
> > > > > +    return g_strdup(core->slot);
> > > > > +}
> > > > > +
> > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > +{
> > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > +
> > > > > +    core->slot = g_strdup(val);
> > > > > +}
> > > > > +
> > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > +{
> > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > +                            NULL);
> > > > > +}
> > > > > +
> > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > +    .name = TYPE_CPU_CORE,
> > > > > +    .parent = TYPE_DEVICE,
> > > > > +    .abstract = true,
> > > > > +    .instance_size = sizeof(CPUCore),
> > > > > +    .instance_init = cpu_core_instance_init,
> > > > > +};
> > > > > +
> > > > > +static void cpu_core_register_types(void)
> > > > > +{
> > > > > +    type_register_static(&cpu_core_type_info);
> > > > > +}
> > > > > +
> > > > > +type_init(cpu_core_register_types)
> > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > new file mode 100644
> > > > > index 0000000..2daa724
> > > > > --- /dev/null
> > > > > +++ b/include/hw/cpu/core.h
> > > > > @@ -0,0 +1,30 @@
> > > > > +/*
> > > > > + * CPU core abstract device
> > > > > + *
> > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > + * See the COPYING file in the top-level directory.
> > > > > + */
> > > > > +#ifndef HW_CPU_CORE_H
> > > > > +#define HW_CPU_CORE_H
> > > > > +
> > > > > +#include "qemu/osdep.h"
> > > > > +#include "hw/qdev.h"
> > > > > +
> > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > +
> > > > > +#define CPU_CORE(obj) \
> > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > +
> > > > > +typedef struct CPUCore {
> > > > > +    /*< private >*/
> > > > > +    DeviceState parent_obj;
> > > > > +
> > > > > +    /*< public >*/
> > > > > +    char *slot;
> > > > > +} CPUCore;
> > > > > +
> > > > > +#define CPU_CORE_SLOT_PROP "slot"    
> > > > as it's generic property I'd rename to 'core' so it would fit all users    
> > > 
> > > Ok. Also note that this is a string property which is associated with the
> > > link name (string) that we created from machine object to this core. I think
> > > it would be ideal if this becomes an interger  property in which case it
> > > becomes easier to feed the core location into your CPUSlotProperties.core.  
> > agreed, it should be core number.  
> 
> The slot stuff is continuing to confuse me a bit.  I see that we need
> some kind of "address" value, but how best to do it is not clear to
> me.
> 
> Changing this to an integer sounds like it's probably a good idea.
> I'm a bit wary of just calling it "core" though.  Do all platforms
> even necessarily have a core id?
platform's that don't have core concept could or even should
use its own base type (i.e. not cpu-core).
Numeric code id should work for x86, ARM and Power.

> I'm wondering if the addressing is something that needs to move the
> the platform specific subtypes, while some other stuff can move to the
> generic base type.
core id looks to me as cpu-core property but I won't object if
it will be moved to subtype as so far only Power would have it.

What I'd prefer to keep is consistent naming of properties
if it's possible, i.e. property 'core' which makes sense for x86, ARM, and Power
from enduser point of view.

> 
> > > > on top of that I'd add numeric 'threads' property to base class so
> > > > all derived cores would inherit it.
> > > > 
> > > > Then as easy integration with -smp threads=x, a machine could push
> > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > make every created cpu-core object to have threads set
> > > > at instance_init() time (device_init).
> > > > 
> > > > That way user won't have to specify 'threads=y' for every
> > > >   device_add spapr-core,core=x
> > > > as it will be taken from global property 'cpu-core.threads'
> > > > but if user wishes he/she still could override global by explicitly
> > > > providing thread property at device_add time:
> > > >   device_add spapr-core,core=x,threads=y
> > > > 
> > > > wrt this series it would mean, instead of creating threads in property
> > > > setter, delaying threads creation to core.realize() time,
> > > > but since realize is allowed to fail it should be fine do so.    
> > > 
> > > Ok that would suit us as there are two properties on which thread creation
> > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > created at core realize time, then we don't have to resort to the ugliness
> > > of creating the threads from either of the property setters. I always
> > > assumed that we shouldn't be creating objects from realize, but if that
> > > is fine, it is good.  
> > since realize is allowed to fail, it should be safe from hotplug pov
> > to create internal objects there, as far as proper cleanups are done
> > for failure path.  
> 
[...]
> I'm not clear from the above if you're also intending to move at least
> the adding of the threads as child properties is supposed to go into
> the base type,
I'm not sure that I've got question, could you please rephrase?

> but that also sounds like a good idea, again for
> consistency.
> 

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 9335 bytes --]

On Mon, Mar 07, 2016 at 11:29:29AM +0100, Igor Mammedov wrote:
> On Mon, 7 Mar 2016 14:36:55 +1100
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:
> > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > >   
> > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:  
> > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > >     
> > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > to define and hotplug CPUs in core granularity.
> > > > > > 
> > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > ---
> > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > >  3 files changed, 75 insertions(+)
> > > > > >  create mode 100644 hw/cpu/core.c
> > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > 
> > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > index 0954a18..942a4bb 100644
> > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > +obj-y += core.o
> > > > > >  
> > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > new file mode 100644
> > > > > > index 0000000..d8caf37
> > > > > > --- /dev/null
> > > > > > +++ b/hw/cpu/core.c
> > > > > > @@ -0,0 +1,44 @@
> > > > > > +/*
> > > > > > + * CPU core abstract device
> > > > > > + *
> > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > + *
> > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > + * See the COPYING file in the top-level directory.
> > > > > > + */
> > > > > > +#include "hw/cpu/core.h"
> > > > > > +
> > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > +{
> > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > +
> > > > > > +    return g_strdup(core->slot);
> > > > > > +}
> > > > > > +
> > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > +{
> > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > +
> > > > > > +    core->slot = g_strdup(val);
> > > > > > +}
> > > > > > +
> > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > +{
> > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > +                            NULL);
> > > > > > +}
> > > > > > +
> > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > +    .parent = TYPE_DEVICE,
> > > > > > +    .abstract = true,
> > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > +};
> > > > > > +
> > > > > > +static void cpu_core_register_types(void)
> > > > > > +{
> > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > +}
> > > > > > +
> > > > > > +type_init(cpu_core_register_types)
> > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > new file mode 100644
> > > > > > index 0000000..2daa724
> > > > > > --- /dev/null
> > > > > > +++ b/include/hw/cpu/core.h
> > > > > > @@ -0,0 +1,30 @@
> > > > > > +/*
> > > > > > + * CPU core abstract device
> > > > > > + *
> > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > + *
> > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > + * See the COPYING file in the top-level directory.
> > > > > > + */
> > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > +#define HW_CPU_CORE_H
> > > > > > +
> > > > > > +#include "qemu/osdep.h"
> > > > > > +#include "hw/qdev.h"
> > > > > > +
> > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > +
> > > > > > +#define CPU_CORE(obj) \
> > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > +
> > > > > > +typedef struct CPUCore {
> > > > > > +    /*< private >*/
> > > > > > +    DeviceState parent_obj;
> > > > > > +
> > > > > > +    /*< public >*/
> > > > > > +    char *slot;
> > > > > > +} CPUCore;
> > > > > > +
> > > > > > +#define CPU_CORE_SLOT_PROP "slot"    
> > > > > as it's generic property I'd rename to 'core' so it would fit all users    
> > > > 
> > > > Ok. Also note that this is a string property which is associated with the
> > > > link name (string) that we created from machine object to this core. I think
> > > > it would be ideal if this becomes an interger  property in which case it
> > > > becomes easier to feed the core location into your CPUSlotProperties.core.  
> > > agreed, it should be core number.  
> > 
> > The slot stuff is continuing to confuse me a bit.  I see that we need
> > some kind of "address" value, but how best to do it is not clear to
> > me.
> > 
> > Changing this to an integer sounds like it's probably a good idea.
> > I'm a bit wary of just calling it "core" though.  Do all platforms
> > even necessarily have a core id?
> platform's that don't have core concept could or even should
> use its own base type (i.e. not cpu-core).

Hmm.. that's a good point.  And actually makes me inclined to
suggest including the cpu model property in the base type, contrary to
my suggestion earlier.

I can think of (somewhat contrived) cases of cpu packages where
cpu_model doesn't make sense (e.g. a multi-chip bigLITTLE system,
since there are multiple CPU types in a package), but in that case the
package doesn't really resemble a "core" in any normal sense.

> Numeric code id should work for x86, ARM and Power.

Yes.  I think a numeric id should be fine in general.  Whether it's
actually meaningful with regard to platform docs, or completely
arbitrary might vary by platform, but it should be possible to create
something.

> > I'm wondering if the addressing is something that needs to move the
> > the platform specific subtypes, while some other stuff can move to the
> > generic base type.
> core id looks to me as cpu-core property but I won't object if
> it will be moved to subtype as so far only Power would have it.
> 
> What I'd prefer to keep is consistent naming of properties
> if it's possible, i.e. property 'core' which makes sense for x86, ARM, and Power
> from enduser point of view.
> 
> > 
> > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > all derived cores would inherit it.
> > > > > 
> > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > make every created cpu-core object to have threads set
> > > > > at instance_init() time (device_init).
> > > > > 
> > > > > That way user won't have to specify 'threads=y' for every
> > > > >   device_add spapr-core,core=x
> > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > but if user wishes he/she still could override global by explicitly
> > > > > providing thread property at device_add time:
> > > > >   device_add spapr-core,core=x,threads=y
> > > > > 
> > > > > wrt this series it would mean, instead of creating threads in property
> > > > > setter, delaying threads creation to core.realize() time,
> > > > > but since realize is allowed to fail it should be fine do so.    
> > > > 
> > > > Ok that would suit us as there are two properties on which thread creation
> > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > created at core realize time, then we don't have to resort to the ugliness
> > > > of creating the threads from either of the property setters. I always
> > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > is fine, it is good.  
> > > since realize is allowed to fail, it should be safe from hotplug pov
> > > to create internal objects there, as far as proper cleanups are done
> > > for failure path.  
> > 
> [...]
> > I'm not clear from the above if you're also intending to move at least
> > the adding of the threads as child properties is supposed to go into
> > the base type,
> I'm not sure that I've got question, could you please rephrase?

So, it seems like we're agreed that moving the nr_threads property to
the base type is a good idea.

My question is, do we also move the object_property_add_child() calls
for each thread to the base type (possibly via a helper function or
method the base type provides to derived types)?

> > but that also sounds like a good idea, again for
> > consistency.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Tue, 8 Mar 2016 15:26:27 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Mon, Mar 07, 2016 at 11:29:29AM +0100, Igor Mammedov wrote:
> > On Mon, 7 Mar 2016 14:36:55 +1100
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > On Fri, Mar 04, 2016 at 07:07:20PM +0100, Igor Mammedov wrote:  
> > > > On Fri, 4 Mar 2016 16:32:53 +0530
> > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > >     
> > > > > On Fri, Mar 04, 2016 at 11:38:45AM +0100, Igor Mammedov wrote:    
> > > > > > On Fri,  4 Mar 2016 12:24:16 +0530
> > > > > > Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> > > > > >       
> > > > > > > Add an abstract CPU core type that could be used by machines that want
> > > > > > > to define and hotplug CPUs in core granularity.
> > > > > > > 
> > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > ---
> > > > > > >  hw/cpu/Makefile.objs  |  1 +
> > > > > > >  hw/cpu/core.c         | 44 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > > >  include/hw/cpu/core.h | 30 ++++++++++++++++++++++++++++++
> > > > > > >  3 files changed, 75 insertions(+)
> > > > > > >  create mode 100644 hw/cpu/core.c
> > > > > > >  create mode 100644 include/hw/cpu/core.h
> > > > > > > 
> > > > > > > diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
> > > > > > > index 0954a18..942a4bb 100644
> > > > > > > --- a/hw/cpu/Makefile.objs
> > > > > > > +++ b/hw/cpu/Makefile.objs
> > > > > > > @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
> > > > > > >  obj-$(CONFIG_REALVIEW) += realview_mpcore.o
> > > > > > >  obj-$(CONFIG_A9MPCORE) += a9mpcore.o
> > > > > > >  obj-$(CONFIG_A15MPCORE) += a15mpcore.o
> > > > > > > +obj-y += core.o
> > > > > > >  
> > > > > > > diff --git a/hw/cpu/core.c b/hw/cpu/core.c
> > > > > > > new file mode 100644
> > > > > > > index 0000000..d8caf37
> > > > > > > --- /dev/null
> > > > > > > +++ b/hw/cpu/core.c
> > > > > > > @@ -0,0 +1,44 @@
> > > > > > > +/*
> > > > > > > + * CPU core abstract device
> > > > > > > + *
> > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > + *
> > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > + */
> > > > > > > +#include "hw/cpu/core.h"
> > > > > > > +
> > > > > > > +static char *core_prop_get_slot(Object *obj, Error **errp)
> > > > > > > +{
> > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > +
> > > > > > > +    return g_strdup(core->slot);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void core_prop_set_slot(Object *obj, const char *val, Error **errp)
> > > > > > > +{
> > > > > > > +    CPUCore *core = CPU_CORE(obj);
> > > > > > > +
> > > > > > > +    core->slot = g_strdup(val);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void cpu_core_instance_init(Object *obj)
> > > > > > > +{
> > > > > > > +    object_property_add_str(obj, "slot", core_prop_get_slot, core_prop_set_slot,
> > > > > > > +                            NULL);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static const TypeInfo cpu_core_type_info = {
> > > > > > > +    .name = TYPE_CPU_CORE,
> > > > > > > +    .parent = TYPE_DEVICE,
> > > > > > > +    .abstract = true,
> > > > > > > +    .instance_size = sizeof(CPUCore),
> > > > > > > +    .instance_init = cpu_core_instance_init,
> > > > > > > +};
> > > > > > > +
> > > > > > > +static void cpu_core_register_types(void)
> > > > > > > +{
> > > > > > > +    type_register_static(&cpu_core_type_info);
> > > > > > > +}
> > > > > > > +
> > > > > > > +type_init(cpu_core_register_types)
> > > > > > > diff --git a/include/hw/cpu/core.h b/include/hw/cpu/core.h
> > > > > > > new file mode 100644
> > > > > > > index 0000000..2daa724
> > > > > > > --- /dev/null
> > > > > > > +++ b/include/hw/cpu/core.h
> > > > > > > @@ -0,0 +1,30 @@
> > > > > > > +/*
> > > > > > > + * CPU core abstract device
> > > > > > > + *
> > > > > > > + * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > + *
> > > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > > > > > > + * See the COPYING file in the top-level directory.
> > > > > > > + */
> > > > > > > +#ifndef HW_CPU_CORE_H
> > > > > > > +#define HW_CPU_CORE_H
> > > > > > > +
> > > > > > > +#include "qemu/osdep.h"
> > > > > > > +#include "hw/qdev.h"
> > > > > > > +
> > > > > > > +#define TYPE_CPU_CORE "cpu-core"
> > > > > > > +
> > > > > > > +#define CPU_CORE(obj) \
> > > > > > > +    OBJECT_CHECK(CPUCore, (obj), TYPE_CPU_CORE)
> > > > > > > +
> > > > > > > +typedef struct CPUCore {
> > > > > > > +    /*< private >*/
> > > > > > > +    DeviceState parent_obj;
> > > > > > > +
> > > > > > > +    /*< public >*/
> > > > > > > +    char *slot;
> > > > > > > +} CPUCore;
> > > > > > > +
> > > > > > > +#define CPU_CORE_SLOT_PROP "slot"      
> > > > > > as it's generic property I'd rename to 'core' so it would fit all users      
> > > > > 
> > > > > Ok. Also note that this is a string property which is associated with the
> > > > > link name (string) that we created from machine object to this core. I think
> > > > > it would be ideal if this becomes an interger  property in which case it
> > > > > becomes easier to feed the core location into your CPUSlotProperties.core.    
> > > > agreed, it should be core number.    
> > > 
> > > The slot stuff is continuing to confuse me a bit.  I see that we need
> > > some kind of "address" value, but how best to do it is not clear to
> > > me.
> > > 
> > > Changing this to an integer sounds like it's probably a good idea.
> > > I'm a bit wary of just calling it "core" though.  Do all platforms
> > > even necessarily have a core id?  
> > platform's that don't have core concept could or even should
> > use its own base type (i.e. not cpu-core).  
> 
> Hmm.. that's a good point.  And actually makes me inclined to
> suggest including the cpu model property in the base type, contrary to
> my suggestion earlier.
I've answered wrt cpu_model in "v1 05/10] cpu: Abstract CPU core type" thread.
You can try to ping/ask Andreas on IRC what he thinks about it,
but I think he would be against cpu_model property in core.

> 
> I can think of (somewhat contrived) cases of cpu packages where
> cpu_model doesn't make sense (e.g. a multi-chip bigLITTLE system,
> since there are multiple CPU types in a package), but in that case the
> package doesn't really resemble a "core" in any normal sense.
> 
> > Numeric code id should work for x86, ARM and Power.  
> 
> Yes.  I think a numeric id should be fine in general.  Whether it's
> actually meaningful with regard to platform docs, or completely
> arbitrary might vary by platform, but it should be possible to create
> something.
> 
> > > I'm wondering if the addressing is something that needs to move the
> > > the platform specific subtypes, while some other stuff can move to the
> > > generic base type.  
> > core id looks to me as cpu-core property but I won't object if
> > it will be moved to subtype as so far only Power would have it.
> > 
> > What I'd prefer to keep is consistent naming of properties
> > if it's possible, i.e. property 'core' which makes sense for x86, ARM, and Power
> > from enduser point of view.
> >   
> > >   
> > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > all derived cores would inherit it.
> > > > > > 
> > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > make every created cpu-core object to have threads set
> > > > > > at instance_init() time (device_init).
> > > > > > 
> > > > > > That way user won't have to specify 'threads=y' for every
> > > > > >   device_add spapr-core,core=x
> > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > providing thread property at device_add time:
> > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > 
> > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > but since realize is allowed to fail it should be fine do so.      
> > > > > 
> > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > of creating the threads from either of the property setters. I always
> > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > is fine, it is good.    
> > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > to create internal objects there, as far as proper cleanups are done
> > > > for failure path.    
> > >   
> > [...]  
> > > I'm not clear from the above if you're also intending to move at least
> > > the adding of the threads as child properties is supposed to go into
> > > the base type,  
> > I'm not sure that I've got question, could you please rephrase?  
> 
> So, it seems like we're agreed that moving the nr_threads property to
> the base type is a good idea.
> 
> My question is, do we also move the object_property_add_child() calls
> for each thread to the base type (possibly via a helper function or
> method the base type provides to derived types)?
I can't think of a reason to do so,
why can't subtype-core.realize() do it?
What would one gain creating callbacks and calling them from base class?

> 
> > > but that also sounds like a good idea, again for
> > > consistency.  
> 

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3279 bytes --]

On Wed, Mar 09, 2016 at 11:40:53AM +0100, Igor Mammedov wrote:
> On Tue, 8 Mar 2016 15:26:27 +1100
> David Gibson <david@gibson.dropbear.id.au> wrote:
> > On Mon, Mar 07, 2016 at 11:29:29AM +0100, Igor Mammedov wrote:
> > > On Mon, 7 Mar 2016 14:36:55 +1100
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
[snip]
> > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > all derived cores would inherit it.
> > > > > > > 
> > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > make every created cpu-core object to have threads set
> > > > > > > at instance_init() time (device_init).
> > > > > > > 
> > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > >   device_add spapr-core,core=x
> > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > providing thread property at device_add time:
> > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > 
> > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > but since realize is allowed to fail it should be fine do so.      
> > > > > > 
> > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > of creating the threads from either of the property setters. I always
> > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > is fine, it is good.    
> > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > to create internal objects there, as far as proper cleanups are done
> > > > > for failure path.    
> > > >   
> > > [...]  
> > > > I'm not clear from the above if you're also intending to move at least
> > > > the adding of the threads as child properties is supposed to go into
> > > > the base type,  
> > > I'm not sure that I've got question, could you please rephrase?  
> > 
> > So, it seems like we're agreed that moving the nr_threads property to
> > the base type is a good idea.
> > 
> > My question is, do we also move the object_property_add_child() calls
> > for each thread to the base type (possibly via a helper function or
> > method the base type provides to derived types)?
> I can't think of a reason to do so,
> why can't subtype-core.realize() do it?

It can, but I'm always suspicious of boilerplate stuff that every
subtype *has* to do in order to work properly.

> What would one gain creating callbacks and calling them from base class?

Enforcing - or at least making as easy as possible - consistency in
the child object naming.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 05/10] cpu: Abstract CPU core type

[Igor Mammedov]

On Thu, 10 Mar 2016 10:42:26 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, Mar 09, 2016 at 11:40:53AM +0100, Igor Mammedov wrote:
> > On Tue, 8 Mar 2016 15:26:27 +1100
> > David Gibson <david@gibson.dropbear.id.au> wrote:  
> > > On Mon, Mar 07, 2016 at 11:29:29AM +0100, Igor Mammedov wrote:  
> > > > On Mon, 7 Mar 2016 14:36:55 +1100
> > > > David Gibson <david@gibson.dropbear.id.au> wrote:  
> [snip]
> > > > > > > > on top of that I'd add numeric 'threads' property to base class so
> > > > > > > > all derived cores would inherit it.
> > > > > > > > 
> > > > > > > > Then as easy integration with -smp threads=x, a machine could push
> > > > > > > > a global variable 'cpu-core.threads=[smp_threads]' which would
> > > > > > > > make every created cpu-core object to have threads set
> > > > > > > > at instance_init() time (device_init).
> > > > > > > > 
> > > > > > > > That way user won't have to specify 'threads=y' for every
> > > > > > > >   device_add spapr-core,core=x
> > > > > > > > as it will be taken from global property 'cpu-core.threads'
> > > > > > > > but if user wishes he/she still could override global by explicitly
> > > > > > > > providing thread property at device_add time:
> > > > > > > >   device_add spapr-core,core=x,threads=y
> > > > > > > > 
> > > > > > > > wrt this series it would mean, instead of creating threads in property
> > > > > > > > setter, delaying threads creation to core.realize() time,
> > > > > > > > but since realize is allowed to fail it should be fine do so.        
> > > > > > > 
> > > > > > > Ok that would suit us as there are two properties on which thread creation
> > > > > > > is dependent upon: nr_threads and cpu_model. If thread objects can be
> > > > > > > created at core realize time, then we don't have to resort to the ugliness
> > > > > > > of creating the threads from either of the property setters. I always
> > > > > > > assumed that we shouldn't be creating objects from realize, but if that
> > > > > > > is fine, it is good.      
> > > > > > since realize is allowed to fail, it should be safe from hotplug pov
> > > > > > to create internal objects there, as far as proper cleanups are done
> > > > > > for failure path.      
> > > > >     
> > > > [...]    
> > > > > I'm not clear from the above if you're also intending to move at least
> > > > > the adding of the threads as child properties is supposed to go into
> > > > > the base type,    
> > > > I'm not sure that I've got question, could you please rephrase?    
> > > 
> > > So, it seems like we're agreed that moving the nr_threads property to
> > > the base type is a good idea.
> > > 
> > > My question is, do we also move the object_property_add_child() calls
> > > for each thread to the base type (possibly via a helper function or
> > > method the base type provides to derived types)?  
> > I can't think of a reason to do so,
> > why can't subtype-core.realize() do it?  
> 
> It can, but I'm always suspicious of boilerplate stuff that every
> subtype *has* to do in order to work properly.
> 
> > What would one gain creating callbacks and calling them from base class?  
> 
> Enforcing - or at least making as easy as possible - consistency in
> the child object naming.
> 

I guess a patch would help to understand if it's worth of effort of adding
extra callbacks in base class.

[Qemu-devel] [RFC PATCH v1 06/10] spapr: CPU core device

[Bharata B Rao]

Add sPAPR specific CPU core device that is based on generic CPU core device.
Creating this core device will result in creation of all the CPU thread
devices that are part of this core.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 hw/ppc/Makefile.objs            |   1 +
 hw/ppc/spapr_cpu_core.c         | 208 ++++++++++++++++++++++++++++++++++++++++
 include/hw/ppc/spapr_cpu_core.h |  29 ++++++
 3 files changed, 238 insertions(+)
 create mode 100644 hw/ppc/spapr_cpu_core.c
 create mode 100644 include/hw/ppc/spapr_cpu_core.h

diff --git a/hw/ppc/Makefile.objs b/hw/ppc/Makefile.objs
index c1ffc77..5cc6608 100644
--- a/hw/ppc/Makefile.objs
+++ b/hw/ppc/Makefile.objs
@@ -4,6 +4,7 @@ obj-y += ppc.o ppc_booke.o
 obj-$(CONFIG_PSERIES) += spapr.o spapr_vio.o spapr_events.o
 obj-$(CONFIG_PSERIES) += spapr_hcall.o spapr_iommu.o spapr_rtas.o
 obj-$(CONFIG_PSERIES) += spapr_pci.o spapr_rtc.o spapr_drc.o spapr_rng.o
+obj-$(CONFIG_PSERIES) += spapr_cpu_core.o
 ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES)$(CONFIG_LINUX), yyy)
 obj-y += spapr_pci_vfio.o
 endif
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
new file mode 100644
index 0000000..3f3440c
--- /dev/null
+++ b/hw/ppc/spapr_cpu_core.c
@@ -0,0 +1,208 @@
+/*
+ * sPAPR CPU core device, acts as container of CPU thread devices.
+ *
+ * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include "hw/cpu/core.h"
+#include "hw/ppc/spapr_cpu_core.h"
+#include "hw/ppc/spapr.h"
+#include "hw/boards.h"
+#include "qemu/error-report.h"
+#include "qapi/visitor.h"
+#include <sysemu/cpus.h>
+
+static int spapr_cpu_core_realize_child(Object *child, void *opaque)
+{
+    Error **errp = opaque;
+
+    object_property_set_bool(child, true, "realized", errp);
+    if (*errp) {
+        return 1;
+    }
+    return 0;
+}
+
+static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
+{
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
+    sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
+    char *slot;
+    Error *local_err = NULL;
+
+    if (!core->nr_threads) {
+        error_setg(errp, "nr_threads property can't be 0");
+        return;
+    }
+
+    if (!core->oc) {
+        error_setg(errp, "cpu_model property isn't set");
+        return;
+    }
+
+    /*
+     * TODO: If slot isn't specified, plug this core into
+     * an existing empty slot.
+     */
+    slot = object_property_get_str(OBJECT(dev), CPU_CORE_SLOT_PROP, &local_err);
+    if (!slot) {
+        error_setg(errp, "slot property isn't set");
+        return;
+    }
+
+    object_property_set_link(OBJECT(spapr), OBJECT(core), slot, &local_err);
+    g_free(slot);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    object_child_foreach(OBJECT(dev), spapr_cpu_core_realize_child, errp);
+}
+
+/*
+ * This creates the CPU threads for a given @core.
+ *
+ * In order to create the threads, we need two inputs - number of
+ * threads and the cpu_model. These are set as core object's properties.
+ * When both of them become available/set, this routine will be called from
+ * either property's set handler to create the threads.
+ *
+ * TODO: Dependence of threads creation on two properties is resulting
+ * in this not-so-clean way of creating threads from either of the
+ * property setters based on the order in which they get set. Check if
+ * this can be handled in a better manner.
+ */
+static void spapr_cpu_core_create_threads(sPAPRCPUCore *core, Error **errp)
+{
+    int i;
+    Error *local_err = NULL;
+
+    for (i = 0; i < core->nr_threads; i++) {
+        char id[32];
+
+        object_initialize(&core->threads[i], sizeof(core->threads[i]),
+                          object_class_get_name(core->oc));
+        snprintf(id, sizeof(id), "thread[%d]", i);
+        object_property_add_child(OBJECT(core), id, OBJECT(&core->threads[i]),
+                                  &local_err);
+        if (local_err) {
+            goto err;
+        }
+    }
+    return;
+
+err:
+    while (--i) {
+        object_unparent(OBJECT(&core->threads[i]));
+    }
+    error_propagate(errp, local_err);
+}
+
+static char *spapr_cpu_core_prop_get_cpu_model(Object *obj, Error **errp)
+{
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(obj);
+
+    return g_strdup(object_class_get_name(core->oc));
+}
+
+static void spapr_cpu_core_prop_set_cpu_model(Object *obj, const char *val,
+                                              Error **errp)
+{
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(obj);
+    MachineState *machine = MACHINE(qdev_get_machine());
+    ObjectClass *oc = cpu_class_by_name(TYPE_POWERPC_CPU, val);
+    ObjectClass *oc_base = cpu_class_by_name(TYPE_POWERPC_CPU,
+                                             machine->cpu_model);
+    if (!oc) {
+        error_setg(errp, "Unknown CPU model %s", val);
+        return;
+    }
+
+    /*
+     * Currently cpu_model can't be different from what is specified with -cpu
+     */
+    if (strcmp(object_class_get_name(oc), object_class_get_name(oc_base))) {
+        error_setg(errp, "cpu_model must be %s", machine->cpu_model);
+        return;
+    }
+
+    core->oc = oc;
+    if (core->nr_threads && core->oc) {
+        spapr_cpu_core_create_threads(core, errp);
+    }
+}
+
+static void spapr_cpu_core_prop_get_nr_threads(Object *obj, Visitor *v,
+                                               const char *name, void *opaque,
+                                               Error **errp)
+{
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(obj);
+    int64_t value = core->nr_threads;
+
+    visit_type_int(v, name, &value, errp);
+}
+
+static void spapr_cpu_core_prop_set_nr_threads(Object *obj, Visitor *v,
+                                               const char *name, void *opaque,
+                                               Error **errp)
+{
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(obj);
+    Error *local_err = NULL;
+    int64_t value;
+
+    visit_type_int(v, name, &value, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    /* Allow only homogeneous configuration */
+    if (value != smp_threads) {
+        error_setg(errp, "nr_threads must be %d", smp_threads);
+        return;
+    }
+
+    core->nr_threads = value;
+    core->threads = g_new0(PowerPCCPU, core->nr_threads);
+
+    if (core->nr_threads && core->oc) {
+        spapr_cpu_core_create_threads(core, errp);
+    }
+}
+
+static void spapr_cpu_core_instance_init(Object *obj)
+{
+    object_property_add(obj, "nr_threads", "int",
+                        spapr_cpu_core_prop_get_nr_threads,
+                        spapr_cpu_core_prop_set_nr_threads,
+                        NULL, NULL, NULL);
+    object_property_add_str(obj, "cpu_model",
+                            spapr_cpu_core_prop_get_cpu_model,
+                            spapr_cpu_core_prop_set_cpu_model,
+                            NULL);
+}
+
+static void spapr_cpu_core_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+
+    dc->realize = spapr_cpu_core_realize;
+}
+
+static const TypeInfo spapr_cpu_core_type_info = {
+    .name = TYPE_SPAPR_CPU_CORE,
+    .parent = TYPE_CPU_CORE,
+    .instance_init = spapr_cpu_core_instance_init,
+    .instance_size = sizeof(sPAPRCPUCore),
+    .class_init = spapr_cpu_core_class_init,
+};
+
+static void spapr_cpu_core_register_types(void)
+{
+    type_register_static(&spapr_cpu_core_type_info);
+}
+
+type_init(spapr_cpu_core_register_types)
diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
new file mode 100644
index 0000000..ddaff6f
--- /dev/null
+++ b/include/hw/ppc/spapr_cpu_core.h
@@ -0,0 +1,29 @@
+/*
+ * sPAPR CPU core device.
+ *
+ * Copyright (C) 2016 Bharata B Rao <bharata@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#ifndef HW_SPAPR_CPU_CORE_H
+#define HW_SPAPR_CPU_CORE_H
+
+#include "hw/qdev.h"
+#include "hw/cpu/core.h"
+
+#define TYPE_SPAPR_CPU_CORE "spapr-cpu-core"
+#define SPAPR_CPU_CORE(obj) \
+    OBJECT_CHECK(sPAPRCPUCore, (obj), TYPE_SPAPR_CPU_CORE)
+
+typedef struct sPAPRCPUCore {
+    /*< private >*/
+    CPUCore parent_obj;
+
+    /*< public >*/
+    int nr_threads;
+    ObjectClass *oc;
+    PowerPCCPU *threads;
+} sPAPRCPUCore;
+
+#endif
-- 
2.1.0

[Qemu-devel] [RFC PATCH v1 07/10] spapr: Represent boot CPUs as spapr-cpu-core devices

[Bharata B Rao]

Initialize boot CPUs as spapr-cpu-core devices and create links from
machine object to these core devices. These links can be considered
as CPU slots in which core devices will get hot-plugged. spapr-cpu-core
device's slot property indicates the slot where it is plugged. Information
about all the CPU slots can be obtained by walking these links.

Also prevent topologies that have or can result in incomplete cores.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 hw/ppc/spapr.c          | 85 ++++++++++++++++++++++++++++++++++++++++++-------
 hw/ppc/spapr_cpu_core.c |  9 ++++++
 include/hw/ppc/spapr.h  |  4 +++
 3 files changed, 87 insertions(+), 11 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index e9d4abf..5acb612 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -64,6 +64,7 @@
 
 #include "hw/compat.h"
 #include "qemu-common.h"
+#include "hw/ppc/spapr_cpu_core.h"
 
 #include <libfdt.h>
 
@@ -1614,8 +1615,11 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
     machine->boot_order = g_strdup(boot_device);
 }
 
-static void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu,
-                           Error **errp)
+/*
+ * TODO: Check if some of these can be moved to rtas_start_cpu() where
+ * a few other things required for hotplugged CPUs are being done.
+ */
+void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
 {
     CPUPPCState *env = &cpu->env;
 
@@ -1643,7 +1647,6 @@ static void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu,
     }
 
     xics_cpu_setup(spapr->icp, cpu);
-
     qemu_register_reset(spapr_cpu_reset, cpu);
 }
 
@@ -1720,6 +1723,28 @@ static void spapr_validate_node_memory(MachineState *machine, Error **errp)
     }
 }
 
+/*
+ * Check to see if core is being hot-plugged into an already populated slot.
+ */
+static void spapr_cpu_core_allow_set_link(Object *obj, const char *name,
+                                          Object *val, Error **errp)
+{
+    Object *core = object_property_get_link(qdev_get_machine(), name, NULL);
+
+    /*
+     * Allow the link to be unset when the core is unplugged.
+     */
+    if (!val) {
+        return;
+    }
+
+    if (core) {
+        char *path = object_get_canonical_path(core);
+        error_setg(errp, "Slot %s already populated with %s", name, path);
+        g_free(path);
+    }
+}
+
 /* pSeries LPAR / sPAPR hardware init */
 static void ppc_spapr_init(MachineState *machine)
 {
@@ -1728,7 +1753,6 @@ static void ppc_spapr_init(MachineState *machine)
     const char *kernel_filename = machine->kernel_filename;
     const char *kernel_cmdline = machine->kernel_cmdline;
     const char *initrd_filename = machine->initrd_filename;
-    PowerPCCPU *cpu;
     PCIHostState *phb;
     int i;
     MemoryRegion *sysmem = get_system_memory();
@@ -1742,6 +1766,20 @@ static void ppc_spapr_init(MachineState *machine)
     long load_limit, fw_size;
     bool kernel_le = false;
     char *filename;
+    int spapr_cores = smp_cpus / smp_threads;
+    int spapr_max_cores = max_cpus / smp_threads;
+
+    if (smp_cpus % smp_threads) {
+        error_report("smp_cpus (%u) must be multiple of threads (%u)",
+                     smp_cpus, smp_threads);
+        exit(1);
+    }
+
+    if (max_cpus % smp_threads) {
+        error_report("max_cpus (%u) must be multiple of threads (%u)",
+                     max_cpus, smp_threads);
+        exit(1);
+    }
 
     msi_supported = true;
 
@@ -1800,13 +1838,37 @@ static void ppc_spapr_init(MachineState *machine)
     if (machine->cpu_model == NULL) {
         machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
     }
-    for (i = 0; i < smp_cpus; i++) {
-        cpu = cpu_ppc_init(machine->cpu_model);
-        if (cpu == NULL) {
-            error_report("Unable to find PowerPC CPU definition");
-            exit(1);
+
+    spapr->cores = g_new0(Object *, spapr_max_cores);
+
+    for (i = 0; i < spapr_max_cores; i++) {
+        char name[32];
+
+        /*
+         * Create links from machine objects to all possible cores.
+         */
+        snprintf(name, sizeof(name), "%s[%d]", SPAPR_MACHINE_CPU_CORE_PROP, i);
+        object_property_add_link(OBJECT(spapr), name, TYPE_SPAPR_CPU_CORE,
+                                 (Object **)&spapr->cores[i],
+                                 spapr_cpu_core_allow_set_link,
+                                 OBJ_PROP_LINK_UNREF_ON_RELEASE,
+                                 &error_fatal);
+
+        /*
+         * Create cores and set link from machine object to core object for
+         * boot time CPUs and realize them.
+         */
+        if (i < spapr_cores) {
+            Object *core  = object_new(TYPE_SPAPR_CPU_CORE);
+
+            object_property_set_str(core, machine->cpu_model, "cpu_model",
+                                    &error_fatal);
+            object_property_set_int(core, smp_threads, "nr_threads",
+                                    &error_fatal);
+            object_property_set_str(core, name, CPU_CORE_SLOT_PROP,
+                                    &error_fatal);
+            object_property_set_bool(core, true, "realized", &error_fatal);
         }
-        spapr_cpu_init(spapr, cpu, &error_fatal);
     }
 
     if (kvm_enabled()) {
@@ -2259,7 +2321,8 @@ static void spapr_machine_device_unplug(HotplugHandler *hotplug_dev,
 static HotplugHandler *spapr_get_hotpug_handler(MachineState *machine,
                                              DeviceState *dev)
 {
-    if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
+    if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) ||
+        object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
         return HOTPLUG_HANDLER(machine);
     }
     return NULL;
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index 3f3440c..9ddf3ce 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -17,11 +17,20 @@
 static int spapr_cpu_core_realize_child(Object *child, void *opaque)
 {
     Error **errp = opaque;
+    sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
+    CPUState *cs = CPU(child);
+    PowerPCCPU *cpu = POWERPC_CPU(cs);
 
     object_property_set_bool(child, true, "realized", errp);
     if (*errp) {
         return 1;
     }
+
+    spapr_cpu_init(spapr, cpu, errp);
+    if (*errp) {
+        return 1;
+    }
+
     return 0;
 }
 
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 098d85d..06e6cec 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -79,8 +79,11 @@ struct sPAPRMachineState {
     /*< public >*/
     char *kvm_type;
     MemoryHotplugState hotplug_memory;
+    Object **cores;
 };
 
+#define SPAPR_MACHINE_CPU_CORE_PROP "core"
+
 #define H_SUCCESS         0
 #define H_BUSY            1        /* Hardware busy -- retry later */
 #define H_CLOSED          2        /* Resource closed */
@@ -585,6 +588,7 @@ void spapr_hotplug_req_add_by_count(sPAPRDRConnectorType drc_type,
                                        uint32_t count);
 void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type,
                                           uint32_t count);
+void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp);
 
 /* rtas-configure-connector state */
 struct sPAPRConfigureConnectorState {
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 07/10] spapr: Represent boot CPUs as spapr-cpu-core devices

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 8737 bytes --]

On Fri, Mar 04, 2016 at 12:24:18PM +0530, Bharata B Rao wrote:
> Initialize boot CPUs as spapr-cpu-core devices and create links from
> machine object to these core devices. These links can be considered
> as CPU slots in which core devices will get hot-plugged. spapr-cpu-core
> device's slot property indicates the slot where it is plugged. Information
> about all the CPU slots can be obtained by walking these links.
> 
> Also prevent topologies that have or can result in incomplete cores.
> 
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> ---
>  hw/ppc/spapr.c          | 85 ++++++++++++++++++++++++++++++++++++++++++-------
>  hw/ppc/spapr_cpu_core.c |  9 ++++++
>  include/hw/ppc/spapr.h  |  4 +++
>  3 files changed, 87 insertions(+), 11 deletions(-)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index e9d4abf..5acb612 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -64,6 +64,7 @@
>  
>  #include "hw/compat.h"
>  #include "qemu-common.h"
> +#include "hw/ppc/spapr_cpu_core.h"
>  
>  #include <libfdt.h>
>  
> @@ -1614,8 +1615,11 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
>      machine->boot_order = g_strdup(boot_device);
>  }
>  
> -static void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu,
> -                           Error **errp)
> +/*
> + * TODO: Check if some of these can be moved to rtas_start_cpu() where
> + * a few other things required for hotplugged CPUs are being done.
> + */
> +void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>  {
>      CPUPPCState *env = &cpu->env;
>  
> @@ -1643,7 +1647,6 @@ static void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu,
>      }
>  
>      xics_cpu_setup(spapr->icp, cpu);
> -
>      qemu_register_reset(spapr_cpu_reset, cpu);
>  }
>  
> @@ -1720,6 +1723,28 @@ static void spapr_validate_node_memory(MachineState *machine, Error **errp)
>      }
>  }
>  
> +/*
> + * Check to see if core is being hot-plugged into an already populated slot.
> + */
> +static void spapr_cpu_core_allow_set_link(Object *obj, const char *name,
> +                                          Object *val, Error **errp)
> +{
> +    Object *core = object_property_get_link(qdev_get_machine(), name, NULL);
> +
> +    /*
> +     * Allow the link to be unset when the core is unplugged.
> +     */
> +    if (!val) {
> +        return;
> +    }
> +
> +    if (core) {
> +        char *path = object_get_canonical_path(core);
> +        error_setg(errp, "Slot %s already populated with %s", name, path);
> +        g_free(path);
> +    }
> +}
> +
>  /* pSeries LPAR / sPAPR hardware init */
>  static void ppc_spapr_init(MachineState *machine)
>  {
> @@ -1728,7 +1753,6 @@ static void ppc_spapr_init(MachineState *machine)
>      const char *kernel_filename = machine->kernel_filename;
>      const char *kernel_cmdline = machine->kernel_cmdline;
>      const char *initrd_filename = machine->initrd_filename;
> -    PowerPCCPU *cpu;
>      PCIHostState *phb;
>      int i;
>      MemoryRegion *sysmem = get_system_memory();
> @@ -1742,6 +1766,20 @@ static void ppc_spapr_init(MachineState *machine)
>      long load_limit, fw_size;
>      bool kernel_le = false;
>      char *filename;
> +    int spapr_cores = smp_cpus / smp_threads;
> +    int spapr_max_cores = max_cpus / smp_threads;
> +
> +    if (smp_cpus % smp_threads) {
> +        error_report("smp_cpus (%u) must be multiple of threads (%u)",
> +                     smp_cpus, smp_threads);
> +        exit(1);
> +    }
> +
> +    if (max_cpus % smp_threads) {
> +        error_report("max_cpus (%u) must be multiple of threads (%u)",
> +                     max_cpus, smp_threads);
> +        exit(1);
> +    }
>  
>      msi_supported = true;
>  
> @@ -1800,13 +1838,37 @@ static void ppc_spapr_init(MachineState *machine)
>      if (machine->cpu_model == NULL) {
>          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
>      }
> -    for (i = 0; i < smp_cpus; i++) {
> -        cpu = cpu_ppc_init(machine->cpu_model);
> -        if (cpu == NULL) {
> -            error_report("Unable to find PowerPC CPU definition");
> -            exit(1);
> +
> +    spapr->cores = g_new0(Object *, spapr_max_cores);
> +
> +    for (i = 0; i < spapr_max_cores; i++) {
> +        char name[32];
> +
> +        /*
> +         * Create links from machine objects to all possible cores.
> +         */
> +        snprintf(name, sizeof(name), "%s[%d]", SPAPR_MACHINE_CPU_CORE_PROP, i);
> +        object_property_add_link(OBJECT(spapr), name, TYPE_SPAPR_CPU_CORE,
> +                                 (Object **)&spapr->cores[i],
> +                                 spapr_cpu_core_allow_set_link,
> +                                 OBJ_PROP_LINK_UNREF_ON_RELEASE,
> +                                 &error_fatal);

These links no longer make sense to me.  When it was a set of links to
the hotpluggable units on the machine, I could see why they might be
useful.  But now that they're links explicitly to *core* which are
only the hotpluggable units on certain platforms, I don't see why
these are useful.

Could we maybe not create the links for now, and add them in a later
patch if they do turn out to be useful for something?

> +        /*
> +         * Create cores and set link from machine object to core object for
> +         * boot time CPUs and realize them.
> +         */
> +        if (i < spapr_cores) {
> +            Object *core  = object_new(TYPE_SPAPR_CPU_CORE);
> +
> +            object_property_set_str(core, machine->cpu_model, "cpu_model",
> +                                    &error_fatal);
> +            object_property_set_int(core, smp_threads, "nr_threads",
> +                                    &error_fatal);
> +            object_property_set_str(core, name, CPU_CORE_SLOT_PROP,
> +                                    &error_fatal);
> +            object_property_set_bool(core, true, "realized", &error_fatal);
>          }
> -        spapr_cpu_init(spapr, cpu, &error_fatal);
>      }
>  
>      if (kvm_enabled()) {
> @@ -2259,7 +2321,8 @@ static void spapr_machine_device_unplug(HotplugHandler *hotplug_dev,
>  static HotplugHandler *spapr_get_hotpug_handler(MachineState *machine,
>                                               DeviceState *dev)
>  {
> -    if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
> +    if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) ||
> +        object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
>          return HOTPLUG_HANDLER(machine);
>      }
>      return NULL;
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index 3f3440c..9ddf3ce 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -17,11 +17,20 @@
>  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
>  {
>      Error **errp = opaque;
> +    sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> +    CPUState *cs = CPU(child);
> +    PowerPCCPU *cpu = POWERPC_CPU(cs);
>  
>      object_property_set_bool(child, true, "realized", errp);
>      if (*errp) {
>          return 1;
>      }
> +
> +    spapr_cpu_init(spapr, cpu, errp);
> +    if (*errp) {
> +        return 1;
> +    }
> +

Doesn't this change belong in the previous patch, which created the
spapr core type?

>      return 0;
>  }
>  
> diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
> index 098d85d..06e6cec 100644
> --- a/include/hw/ppc/spapr.h
> +++ b/include/hw/ppc/spapr.h
> @@ -79,8 +79,11 @@ struct sPAPRMachineState {
>      /*< public >*/
>      char *kvm_type;
>      MemoryHotplugState hotplug_memory;
> +    Object **cores;
>  };
>  
> +#define SPAPR_MACHINE_CPU_CORE_PROP "core"
> +
>  #define H_SUCCESS         0
>  #define H_BUSY            1        /* Hardware busy -- retry later */
>  #define H_CLOSED          2        /* Resource closed */
> @@ -585,6 +588,7 @@ void spapr_hotplug_req_add_by_count(sPAPRDRConnectorType drc_type,
>                                         uint32_t count);
>  void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type,
>                                            uint32_t count);
> +void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp);
>  
>  /* rtas-configure-connector state */
>  struct sPAPRConfigureConnectorState {

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 07/10] spapr: Represent boot CPUs as spapr-cpu-core devices

[Bharata B Rao]

On Mon, Mar 07, 2016 at 02:45:09PM +1100, David Gibson wrote:
> On Fri, Mar 04, 2016 at 12:24:18PM +0530, Bharata B Rao wrote:
> > Initialize boot CPUs as spapr-cpu-core devices and create links from
> > machine object to these core devices. These links can be considered
> > as CPU slots in which core devices will get hot-plugged. spapr-cpu-core
> > device's slot property indicates the slot where it is plugged. Information
> > about all the CPU slots can be obtained by walking these links.
> > 
> > Also prevent topologies that have or can result in incomplete cores.
> > 
> > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > ---
> >  hw/ppc/spapr.c          | 85 ++++++++++++++++++++++++++++++++++++++++++-------
> >  hw/ppc/spapr_cpu_core.c |  9 ++++++
> >  include/hw/ppc/spapr.h  |  4 +++
> >  3 files changed, 87 insertions(+), 11 deletions(-)
> > 
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index e9d4abf..5acb612 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -64,6 +64,7 @@
> >  
> >  #include "hw/compat.h"
> >  #include "qemu-common.h"
> > +#include "hw/ppc/spapr_cpu_core.h"
> >  
> >  #include <libfdt.h>
> >  
> > @@ -1614,8 +1615,11 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> >      machine->boot_order = g_strdup(boot_device);
> >  }
> >  
> > -static void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu,
> > -                           Error **errp)
> > +/*
> > + * TODO: Check if some of these can be moved to rtas_start_cpu() where
> > + * a few other things required for hotplugged CPUs are being done.
> > + */
> > +void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> >  {
> >      CPUPPCState *env = &cpu->env;
> >  
> > @@ -1643,7 +1647,6 @@ static void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu,
> >      }
> >  
> >      xics_cpu_setup(spapr->icp, cpu);
> > -
> >      qemu_register_reset(spapr_cpu_reset, cpu);
> >  }
> >  
> > @@ -1720,6 +1723,28 @@ static void spapr_validate_node_memory(MachineState *machine, Error **errp)
> >      }
> >  }
> >  
> > +/*
> > + * Check to see if core is being hot-plugged into an already populated slot.
> > + */
> > +static void spapr_cpu_core_allow_set_link(Object *obj, const char *name,
> > +                                          Object *val, Error **errp)
> > +{
> > +    Object *core = object_property_get_link(qdev_get_machine(), name, NULL);
> > +
> > +    /*
> > +     * Allow the link to be unset when the core is unplugged.
> > +     */
> > +    if (!val) {
> > +        return;
> > +    }
> > +
> > +    if (core) {
> > +        char *path = object_get_canonical_path(core);
> > +        error_setg(errp, "Slot %s already populated with %s", name, path);
> > +        g_free(path);
> > +    }
> > +}
> > +
> >  /* pSeries LPAR / sPAPR hardware init */
> >  static void ppc_spapr_init(MachineState *machine)
> >  {
> > @@ -1728,7 +1753,6 @@ static void ppc_spapr_init(MachineState *machine)
> >      const char *kernel_filename = machine->kernel_filename;
> >      const char *kernel_cmdline = machine->kernel_cmdline;
> >      const char *initrd_filename = machine->initrd_filename;
> > -    PowerPCCPU *cpu;
> >      PCIHostState *phb;
> >      int i;
> >      MemoryRegion *sysmem = get_system_memory();
> > @@ -1742,6 +1766,20 @@ static void ppc_spapr_init(MachineState *machine)
> >      long load_limit, fw_size;
> >      bool kernel_le = false;
> >      char *filename;
> > +    int spapr_cores = smp_cpus / smp_threads;
> > +    int spapr_max_cores = max_cpus / smp_threads;
> > +
> > +    if (smp_cpus % smp_threads) {
> > +        error_report("smp_cpus (%u) must be multiple of threads (%u)",
> > +                     smp_cpus, smp_threads);
> > +        exit(1);
> > +    }
> > +
> > +    if (max_cpus % smp_threads) {
> > +        error_report("max_cpus (%u) must be multiple of threads (%u)",
> > +                     max_cpus, smp_threads);
> > +        exit(1);
> > +    }
> >  
> >      msi_supported = true;
> >  
> > @@ -1800,13 +1838,37 @@ static void ppc_spapr_init(MachineState *machine)
> >      if (machine->cpu_model == NULL) {
> >          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
> >      }
> > -    for (i = 0; i < smp_cpus; i++) {
> > -        cpu = cpu_ppc_init(machine->cpu_model);
> > -        if (cpu == NULL) {
> > -            error_report("Unable to find PowerPC CPU definition");
> > -            exit(1);
> > +
> > +    spapr->cores = g_new0(Object *, spapr_max_cores);
> > +
> > +    for (i = 0; i < spapr_max_cores; i++) {
> > +        char name[32];
> > +
> > +        /*
> > +         * Create links from machine objects to all possible cores.
> > +         */
> > +        snprintf(name, sizeof(name), "%s[%d]", SPAPR_MACHINE_CPU_CORE_PROP, i);
> > +        object_property_add_link(OBJECT(spapr), name, TYPE_SPAPR_CPU_CORE,
> > +                                 (Object **)&spapr->cores[i],
> > +                                 spapr_cpu_core_allow_set_link,
> > +                                 OBJ_PROP_LINK_UNREF_ON_RELEASE,
> > +                                 &error_fatal);
> 
> These links no longer make sense to me.  When it was a set of links to
> the hotpluggable units on the machine, I could see why they might be
> useful.  But now that they're links explicitly to *core* which are
> only the hotpluggable units on certain platforms, I don't see why
> these are useful.
> 
> Could we maybe not create the links for now, and add them in a later
> patch if they do turn out to be useful for something?

Dropped the links part.

> 
> > +        /*
> > +         * Create cores and set link from machine object to core object for
> > +         * boot time CPUs and realize them.
> > +         */
> > +        if (i < spapr_cores) {
> > +            Object *core  = object_new(TYPE_SPAPR_CPU_CORE);
> > +
> > +            object_property_set_str(core, machine->cpu_model, "cpu_model",
> > +                                    &error_fatal);
> > +            object_property_set_int(core, smp_threads, "nr_threads",
> > +                                    &error_fatal);
> > +            object_property_set_str(core, name, CPU_CORE_SLOT_PROP,
> > +                                    &error_fatal);
> > +            object_property_set_bool(core, true, "realized", &error_fatal);
> >          }
> > -        spapr_cpu_init(spapr, cpu, &error_fatal);
> >      }
> >  
> >      if (kvm_enabled()) {
> > @@ -2259,7 +2321,8 @@ static void spapr_machine_device_unplug(HotplugHandler *hotplug_dev,
> >  static HotplugHandler *spapr_get_hotpug_handler(MachineState *machine,
> >                                               DeviceState *dev)
> >  {
> > -    if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
> > +    if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) ||
> > +        object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
> >          return HOTPLUG_HANDLER(machine);
> >      }
> >      return NULL;
> > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > index 3f3440c..9ddf3ce 100644
> > --- a/hw/ppc/spapr_cpu_core.c
> > +++ b/hw/ppc/spapr_cpu_core.c
> > @@ -17,11 +17,20 @@
> >  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> >  {
> >      Error **errp = opaque;
> > +    sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > +    CPUState *cs = CPU(child);
> > +    PowerPCCPU *cpu = POWERPC_CPU(cs);
> >  
> >      object_property_set_bool(child, true, "realized", errp);
> >      if (*errp) {
> >          return 1;
> >      }
> > +
> > +    spapr_cpu_init(spapr, cpu, errp);
> > +    if (*errp) {
> > +        return 1;
> > +    }
> > +
> 
> Doesn't this change belong in the previous patch, which created the
> spapr core type?

I merged this patch with the previous one so that we have one patch
that creates spapr-cpu-core and converts boot cpus to cores.

Regards,
Bharata.

[Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Bharata B Rao]

Set up device tree entries for the hotplugged CPU core and use the
exising EPOW event infrastructure to send CPU hotplug notification to
the guest.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
 hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
 hw/ppc/spapr_events.c           |  3 ++
 hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
 include/hw/ppc/spapr.h          |  4 +++
 include/hw/ppc/spapr_cpu_core.h |  2 ++
 6 files changed, 165 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 5acb612..6c4ac50 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
     size_t page_sizes_prop_size;
     uint32_t vcpus_per_socket = smp_threads * smp_cores;
     uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
+    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
+    sPAPRDRConnector *drc;
+    sPAPRDRConnectorClass *drck;
+    int drc_index;
+
+    if (smc->dr_cpu_enabled) {
+        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
+        g_assert(drc);
+        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
+        drc_index = drck->get_index(drc);
+        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
+    }
 
     /* Note: we keep CI large pages off for now because a 64K capable guest
      * provisioned with large pages might otherwise try to map a qemu
@@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
         _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
     }
 
+    if (smc->dr_cpu_enabled) {
+        int offset = fdt_path_offset(fdt, "/cpus");
+        ret = spapr_drc_populate_dt(fdt, offset, NULL,
+                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
+        if (ret < 0) {
+            error_report("Couldn't set up CPU DR device tree properties");
+            exit(1);
+        }
+    }
+
     _FDT((fdt_pack(fdt)));
 
     if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
@@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
 
 }
 
-static void spapr_cpu_reset(void *opaque)
+void spapr_cpu_reset(void *opaque)
 {
     sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
     PowerPCCPU *cpu = opaque;
@@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
 void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
 {
     CPUPPCState *env = &cpu->env;
+    CPUState *cs = CPU(cpu);
+    int i;
 
     /* Set time-base frequency to 512 MHz */
     cpu_ppc_tb_init(env, TIMEBASE_FREQ);
@@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
         }
     }
 
+    /* Set NUMA node for the added CPUs  */
+    for (i = 0; i < nb_numa_nodes; i++) {
+        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
+            cs->numa_node = i;
+            break;
+        }
+    }
+
     xics_cpu_setup(spapr->icp, cpu);
     qemu_register_reset(spapr_cpu_reset, cpu);
 }
@@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine)
     char *filename;
     int spapr_cores = smp_cpus / smp_threads;
     int spapr_max_cores = max_cpus / smp_threads;
+    int smt = kvmppc_smt_threads();
 
     if (smp_cpus % smp_threads) {
         error_report("smp_cpus (%u) must be multiple of threads (%u)",
@@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine)
         spapr_validate_node_memory(machine, &error_fatal);
     }
 
+    if (smc->dr_cpu_enabled) {
+        for (i = 0; i < spapr_max_cores; i++) {
+            sPAPRDRConnector *drc =
+                spapr_dr_connector_new(OBJECT(spapr),
+                                       SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt);
+            qemu_register_reset(spapr_drc_reset, drc);
+        }
+    }
+
     /* init CPUs */
     if (machine->cpu_model == NULL) {
         machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
@@ -2267,6 +2309,27 @@ out:
     error_propagate(errp, local_err);
 }
 
+void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
+                                    int *fdt_offset, sPAPRMachineState *spapr)
+{
+    PowerPCCPU *cpu = POWERPC_CPU(cs);
+    DeviceClass *dc = DEVICE_GET_CLASS(cs);
+    int id = ppc_get_vcpu_dt_id(cpu);
+    void *fdt;
+    int offset, fdt_size;
+    char *nodename;
+
+    fdt = create_device_tree(&fdt_size);
+    nodename = g_strdup_printf("%s@%x", dc->fw_name, id);
+    offset = fdt_add_subnode(fdt, 0, nodename);
+
+    spapr_populate_cpu_dt(cs, fdt, offset, spapr);
+    g_free(nodename);
+
+    *fdt_offset = offset;
+    return fdt;
+}
+
 static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
                                       DeviceState *dev, Error **errp)
 {
@@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
         }
 
         spapr_memory_plug(hotplug_dev, dev, node, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
+        if (!smc->dr_cpu_enabled && dev->hotplugged) {
+            error_setg(errp, "CPU hotplug not supported for this machine");
+            return;
+        }
+        spapr_core_plug(hotplug_dev, dev, errp);
     }
 }
 
@@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
 
     smc->dr_lmb_enabled = true;
+    smc->dr_cpu_enabled = true;
     fwc->get_dev_path = spapr_get_fw_dev_path;
     nc->nmi_monitor_handler = spapr_nmi;
 }
@@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc)
 
     spapr_machine_2_6_class_options(mc);
     smc->use_ohci_by_default = true;
+    smc->dr_cpu_enabled = false;
     SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5);
 }
 
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index 9ddf3ce..4c233d7 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -14,6 +14,65 @@
 #include "qapi/visitor.h"
 #include <sysemu/cpus.h>
 
+void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                     Error **errp)
+{
+    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
+    sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine());
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
+    PowerPCCPU *cpu = &core->threads[0];
+    CPUState *cs = CPU(cpu);
+    int id = ppc_get_vcpu_dt_id(cpu);
+    sPAPRDRConnector *drc =
+        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
+    sPAPRDRConnectorClass *drck;
+    Error *local_err = NULL;
+    void *fdt = NULL;
+    int fdt_offset = 0;
+
+    if (!smc->dr_cpu_enabled) {
+        /*
+         * This is a cold plugged CPU core but the machine doesn't support
+         * DR. So skip the hotplug path ensuring that the core is brought
+         * up online with out an associated DR connector.
+         */
+        return;
+    }
+
+    g_assert(drc);
+
+    /*
+     * Setup CPU DT entries only for hotplugged CPUs. For boot time or
+     * coldplugged CPUs DT entries are setup in spapr_finalize_fdt().
+     */
+    if (dev->hotplugged) {
+        fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms);
+        dev->hotplugged = true;
+    }
+
+    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
+    drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err);
+    if (local_err) {
+        g_free(fdt);
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    if (dev->hotplugged) {
+        /*
+         * Send hotplug notification interrupt to the guest only in case
+         * of hotplugged CPUs.
+         */
+        spapr_hotplug_req_add_by_index(drc);
+    } else {
+        /*
+         * Set the right DRC states for cold plugged CPU.
+         */
+        drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE);
+        drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED);
+    }
+}
+
 static int spapr_cpu_core_realize_child(Object *child, void *opaque)
 {
     Error **errp = opaque;
@@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque)
     if (*errp) {
         return 1;
     }
+    spapr_cpu_reset(cpu);
 
     return 0;
 }
diff --git a/hw/ppc/spapr_events.c b/hw/ppc/spapr_events.c
index 39f4682..10340e1 100644
--- a/hw/ppc/spapr_events.c
+++ b/hw/ppc/spapr_events.c
@@ -437,6 +437,9 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action,
     case SPAPR_DR_CONNECTOR_TYPE_LMB:
         hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_MEMORY;
         break;
+    case SPAPR_DR_CONNECTOR_TYPE_CPU:
+        hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_CPU;
+        break;
     default:
         /* we shouldn't be signaling hotplug events for resources
          * that don't support them
diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c
index b7c5ebd..cc0369e 100644
--- a/hw/ppc/spapr_rtas.c
+++ b/hw/ppc/spapr_rtas.c
@@ -34,6 +34,7 @@
 
 #include "hw/ppc/spapr.h"
 #include "hw/ppc/spapr_vio.h"
+#include "hw/ppc/ppc.h"
 #include "qapi-event.h"
 #include "hw/boards.h"
 
@@ -161,6 +162,27 @@ static void rtas_query_cpu_stopped_state(PowerPCCPU *cpu_,
     rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
 }
 
+/*
+ * Set the timebase offset of the CPU to that of first CPU.
+ * This helps hotplugged CPU to have the correct timebase offset.
+ */
+static void spapr_cpu_update_tb_offset(PowerPCCPU *cpu)
+{
+    PowerPCCPU *fcpu = POWERPC_CPU(first_cpu);
+
+    cpu->env.tb_env->tb_offset = fcpu->env.tb_env->tb_offset;
+}
+
+static void spapr_cpu_set_endianness(PowerPCCPU *cpu)
+{
+    PowerPCCPU *fcpu = POWERPC_CPU(first_cpu);
+    PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(fcpu);
+
+    if (!pcc->interrupts_big_endian(fcpu)) {
+        cpu->env.spr[SPR_LPCR] |= LPCR_ILE;
+    }
+}
+
 static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr,
                            uint32_t token, uint32_t nargs,
                            target_ulong args,
@@ -197,6 +219,8 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr,
         env->nip = start;
         env->gpr[3] = r3;
         cs->halted = 0;
+        spapr_cpu_set_endianness(cpu);
+        spapr_cpu_update_tb_offset(cpu);
 
         qemu_cpu_kick(cs);
 
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 06e6cec..d1a0af8 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -36,6 +36,7 @@ struct sPAPRMachineClass {
 
     /*< public >*/
     bool dr_lmb_enabled;       /* enable dynamic-reconfig/hotplug of LMBs */
+    bool dr_cpu_enabled;       /* enable dynamic-reconfig/hotplug of CPUs */
     bool use_ohci_by_default;  /* use USB-OHCI instead of XHCI */
 };
 
@@ -589,6 +590,9 @@ void spapr_hotplug_req_add_by_count(sPAPRDRConnectorType drc_type,
 void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type,
                                           uint32_t count);
 void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp);
+void spapr_cpu_reset(void *opaque);
+void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
+                                    int *fdt_offset, sPAPRMachineState *spapr);
 
 /* rtas-configure-connector state */
 struct sPAPRConfigureConnectorState {
diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
index ddaff6f..9879398 100644
--- a/include/hw/ppc/spapr_cpu_core.h
+++ b/include/hw/ppc/spapr_cpu_core.h
@@ -26,4 +26,6 @@ typedef struct sPAPRCPUCore {
     PowerPCCPU *threads;
 } sPAPRCPUCore;
 
+void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                     Error **errp);
 #endif
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 13391 bytes --]

On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:
> Set up device tree entries for the hotplugged CPU core and use the
> exising EPOW event infrastructure to send CPU hotplug notification to
> the guest.
> 
> Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> ---
>  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
>  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
>  hw/ppc/spapr_events.c           |  3 ++
>  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
>  include/hw/ppc/spapr.h          |  4 +++
>  include/hw/ppc/spapr_cpu_core.h |  2 ++
>  6 files changed, 165 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 5acb612..6c4ac50 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
>      size_t page_sizes_prop_size;
>      uint32_t vcpus_per_socket = smp_threads * smp_cores;
>      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> +    sPAPRDRConnector *drc;
> +    sPAPRDRConnectorClass *drck;
> +    int drc_index;
> +
> +    if (smc->dr_cpu_enabled) {
> +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> +        g_assert(drc);
> +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> +        drc_index = drck->get_index(drc);
> +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> +    }
>  
>      /* Note: we keep CI large pages off for now because a 64K capable guest
>       * provisioned with large pages might otherwise try to map a qemu
> @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
>          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
>      }
>  
> +    if (smc->dr_cpu_enabled) {
> +        int offset = fdt_path_offset(fdt, "/cpus");
> +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> +        if (ret < 0) {
> +            error_report("Couldn't set up CPU DR device tree properties");
> +            exit(1);
> +        }
> +    }
> +
>      _FDT((fdt_pack(fdt)));
>  
>      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
>  
>  }
>  
> -static void spapr_cpu_reset(void *opaque)
> +void spapr_cpu_reset(void *opaque)
>  {
>      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
>      PowerPCCPU *cpu = opaque;
> @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
>  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>  {
>      CPUPPCState *env = &cpu->env;
> +    CPUState *cs = CPU(cpu);
> +    int i;
>  
>      /* Set time-base frequency to 512 MHz */
>      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>          }
>      }
>  
> +    /* Set NUMA node for the added CPUs  */
> +    for (i = 0; i < nb_numa_nodes; i++) {
> +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> +            cs->numa_node = i;
> +            break;
> +        }
> +    }
> +

This hunk seems like it belongs in a different patch.

>      xics_cpu_setup(spapr->icp, cpu);
>      qemu_register_reset(spapr_cpu_reset, cpu);
>  }
> @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine)
>      char *filename;
>      int spapr_cores = smp_cpus / smp_threads;
>      int spapr_max_cores = max_cpus / smp_threads;
> +    int smt = kvmppc_smt_threads();
>  
>      if (smp_cpus % smp_threads) {
>          error_report("smp_cpus (%u) must be multiple of threads (%u)",
> @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine)
>          spapr_validate_node_memory(machine, &error_fatal);
>      }
>  
> +    if (smc->dr_cpu_enabled) {
> +        for (i = 0; i < spapr_max_cores; i++) {
> +            sPAPRDRConnector *drc =
> +                spapr_dr_connector_new(OBJECT(spapr),
> +                                       SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt);
> +            qemu_register_reset(spapr_drc_reset, drc);
> +        }
> +    }
> +

Nit: would this be cleaner to include in the same loop that constructs
the (empty) links and boot-time cpu cores?

>      /* init CPUs */
>      if (machine->cpu_model == NULL) {
>          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
> @@ -2267,6 +2309,27 @@ out:
>      error_propagate(errp, local_err);
>  }
>  
> +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
> +                                    int *fdt_offset, sPAPRMachineState *spapr)
> +{
> +    PowerPCCPU *cpu = POWERPC_CPU(cs);
> +    DeviceClass *dc = DEVICE_GET_CLASS(cs);
> +    int id = ppc_get_vcpu_dt_id(cpu);
> +    void *fdt;
> +    int offset, fdt_size;
> +    char *nodename;
> +
> +    fdt = create_device_tree(&fdt_size);
> +    nodename = g_strdup_printf("%s@%x", dc->fw_name, id);
> +    offset = fdt_add_subnode(fdt, 0, nodename);
> +
> +    spapr_populate_cpu_dt(cs, fdt, offset, spapr);
> +    g_free(nodename);
> +
> +    *fdt_offset = offset;
> +    return fdt;
> +}
> +
>  static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
>                                        DeviceState *dev, Error **errp)
>  {
> @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
>          }
>  
>          spapr_memory_plug(hotplug_dev, dev, node, errp);
> +    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
> +        if (!smc->dr_cpu_enabled && dev->hotplugged) {
> +            error_setg(errp, "CPU hotplug not supported for this machine");
> +            return;
> +        }
> +        spapr_core_plug(hotplug_dev, dev, errp);
>      }
>  }
>  
> @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
>      mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
>  
>      smc->dr_lmb_enabled = true;
> +    smc->dr_cpu_enabled = true;
>      fwc->get_dev_path = spapr_get_fw_dev_path;
>      nc->nmi_monitor_handler = spapr_nmi;
>  }
> @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc)
>  
>      spapr_machine_2_6_class_options(mc);
>      smc->use_ohci_by_default = true;
> +    smc->dr_cpu_enabled = false;
>      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5);
>  }
>  
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index 9ddf3ce..4c233d7 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -14,6 +14,65 @@
>  #include "qapi/visitor.h"
>  #include <sysemu/cpus.h>
>  
> +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> +                     Error **errp)
> +{
> +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> +    sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine());
> +    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
> +    PowerPCCPU *cpu = &core->threads[0];
> +    CPUState *cs = CPU(cpu);
> +    int id = ppc_get_vcpu_dt_id(cpu);
> +    sPAPRDRConnector *drc =
> +        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
> +    sPAPRDRConnectorClass *drck;
> +    Error *local_err = NULL;
> +    void *fdt = NULL;
> +    int fdt_offset = 0;
> +
> +    if (!smc->dr_cpu_enabled) {
> +        /*
> +         * This is a cold plugged CPU core but the machine doesn't support
> +         * DR. So skip the hotplug path ensuring that the core is brought
> +         * up online with out an associated DR connector.
> +         */
> +        return;
> +    }
> +
> +    g_assert(drc);
> +
> +    /*
> +     * Setup CPU DT entries only for hotplugged CPUs. For boot time or
> +     * coldplugged CPUs DT entries are setup in spapr_finalize_fdt().
> +     */
> +    if (dev->hotplugged) {
> +        fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms);
> +        dev->hotplugged = true;
> +    }
> +
> +    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> +    drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err);
> +    if (local_err) {
> +        g_free(fdt);
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    if (dev->hotplugged) {
> +        /*
> +         * Send hotplug notification interrupt to the guest only in case
> +         * of hotplugged CPUs.
> +         */
> +        spapr_hotplug_req_add_by_index(drc);
> +    } else {
> +        /*
> +         * Set the right DRC states for cold plugged CPU.
> +         */
> +        drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE);
> +        drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED);
> +    }
> +}
> +
>  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
>  {
>      Error **errp = opaque;
> @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque)
>      if (*errp) {
>          return 1;
>      }
> +    spapr_cpu_reset(cpu);

This also looks like it belongs in a different patch.

>      return 0;
>  }
> diff --git a/hw/ppc/spapr_events.c b/hw/ppc/spapr_events.c
> index 39f4682..10340e1 100644
> --- a/hw/ppc/spapr_events.c
> +++ b/hw/ppc/spapr_events.c
> @@ -437,6 +437,9 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action,
>      case SPAPR_DR_CONNECTOR_TYPE_LMB:
>          hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_MEMORY;
>          break;
> +    case SPAPR_DR_CONNECTOR_TYPE_CPU:
> +        hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_CPU;
> +        break;
>      default:
>          /* we shouldn't be signaling hotplug events for resources
>           * that don't support them
> diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c
> index b7c5ebd..cc0369e 100644
> --- a/hw/ppc/spapr_rtas.c
> +++ b/hw/ppc/spapr_rtas.c
> @@ -34,6 +34,7 @@
>  
>  #include "hw/ppc/spapr.h"
>  #include "hw/ppc/spapr_vio.h"
> +#include "hw/ppc/ppc.h"
>  #include "qapi-event.h"
>  #include "hw/boards.h"
>  
> @@ -161,6 +162,27 @@ static void rtas_query_cpu_stopped_state(PowerPCCPU *cpu_,
>      rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
>  }
>  
> +/*
> + * Set the timebase offset of the CPU to that of first CPU.
> + * This helps hotplugged CPU to have the correct timebase offset.
> + */
> +static void spapr_cpu_update_tb_offset(PowerPCCPU *cpu)
> +{
> +    PowerPCCPU *fcpu = POWERPC_CPU(first_cpu);
> +
> +    cpu->env.tb_env->tb_offset = fcpu->env.tb_env->tb_offset;
> +}
> +
> +static void spapr_cpu_set_endianness(PowerPCCPU *cpu)
> +{
> +    PowerPCCPU *fcpu = POWERPC_CPU(first_cpu);
> +    PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(fcpu);
> +
> +    if (!pcc->interrupts_big_endian(fcpu)) {
> +        cpu->env.spr[SPR_LPCR] |= LPCR_ILE;
> +    }
> +}
> +
>  static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr,
>                             uint32_t token, uint32_t nargs,
>                             target_ulong args,
> @@ -197,6 +219,8 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr,
>          env->nip = start;
>          env->gpr[3] = r3;
>          cs->halted = 0;
> +        spapr_cpu_set_endianness(cpu);
> +        spapr_cpu_update_tb_offset(cpu);
>  
>          qemu_cpu_kick(cs);
>  
> diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
> index 06e6cec..d1a0af8 100644
> --- a/include/hw/ppc/spapr.h
> +++ b/include/hw/ppc/spapr.h
> @@ -36,6 +36,7 @@ struct sPAPRMachineClass {
>  
>      /*< public >*/
>      bool dr_lmb_enabled;       /* enable dynamic-reconfig/hotplug of LMBs */
> +    bool dr_cpu_enabled;       /* enable dynamic-reconfig/hotplug of CPUs */
>      bool use_ohci_by_default;  /* use USB-OHCI instead of XHCI */
>  };
>  
> @@ -589,6 +590,9 @@ void spapr_hotplug_req_add_by_count(sPAPRDRConnectorType drc_type,
>  void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type,
>                                            uint32_t count);
>  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp);
> +void spapr_cpu_reset(void *opaque);
> +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
> +                                    int *fdt_offset, sPAPRMachineState *spapr);
>  
>  /* rtas-configure-connector state */
>  struct sPAPRConfigureConnectorState {
> diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
> index ddaff6f..9879398 100644
> --- a/include/hw/ppc/spapr_cpu_core.h
> +++ b/include/hw/ppc/spapr_cpu_core.h
> @@ -26,4 +26,6 @@ typedef struct sPAPRCPUCore {
>      PowerPCCPU *threads;
>  } sPAPRCPUCore;
>  
> +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> +                     Error **errp);
>  #endif

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Bharata B Rao]

On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:
> On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:
> > Set up device tree entries for the hotplugged CPU core and use the
> > exising EPOW event infrastructure to send CPU hotplug notification to
> > the guest.
> > 
> > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > ---
> >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> >  hw/ppc/spapr_events.c           |  3 ++
> >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> >  include/hw/ppc/spapr.h          |  4 +++
> >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> >  6 files changed, 165 insertions(+), 1 deletion(-)
> > 
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index 5acb612..6c4ac50 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> >      size_t page_sizes_prop_size;
> >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > +    sPAPRDRConnector *drc;
> > +    sPAPRDRConnectorClass *drck;
> > +    int drc_index;
> > +
> > +    if (smc->dr_cpu_enabled) {
> > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > +        g_assert(drc);
> > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > +        drc_index = drck->get_index(drc);
> > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > +    }
> >  
> >      /* Note: we keep CI large pages off for now because a 64K capable guest
> >       * provisioned with large pages might otherwise try to map a qemu
> > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> >      }
> >  
> > +    if (smc->dr_cpu_enabled) {
> > +        int offset = fdt_path_offset(fdt, "/cpus");
> > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > +        if (ret < 0) {
> > +            error_report("Couldn't set up CPU DR device tree properties");
> > +            exit(1);
> > +        }
> > +    }
> > +
> >      _FDT((fdt_pack(fdt)));
> >  
> >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> >  
> >  }
> >  
> > -static void spapr_cpu_reset(void *opaque)
> > +void spapr_cpu_reset(void *opaque)
> >  {
> >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> >      PowerPCCPU *cpu = opaque;
> > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> >  {
> >      CPUPPCState *env = &cpu->env;
> > +    CPUState *cs = CPU(cpu);
> > +    int i;
> >  
> >      /* Set time-base frequency to 512 MHz */
> >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> >          }
> >      }
> >  
> > +    /* Set NUMA node for the added CPUs  */
> > +    for (i = 0; i < nb_numa_nodes; i++) {
> > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > +            cs->numa_node = i;
> > +            break;
> > +        }
> > +    }
> > +
> 
> This hunk seems like it belongs in a different patch.

It appears that this would be needed by other archs also to set the
NUMA node for the hot-plugged CPU. How about make an API out of this
and use this something like below ? Igor ?

-------------------------------------------------------------------
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 0aeefd2..8347234 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1112,6 +1112,7 @@ void pc_hot_add_cpu(const int64_t id, Error **errp)
         error_propagate(errp, local_err);
         return;
     }
+    numa_set_cpu(CPU(cpu));
     object_unref(OBJECT(cpu));
 }
 
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index a42f8c0..f2b3b67 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1645,7 +1645,6 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
 {
     CPUPPCState *env = &cpu->env;
     CPUState *cs = CPU(cpu);
-    int i;
 
     /* Set time-base frequency to 512 MHz */
     cpu_ppc_tb_init(env, TIMEBASE_FREQ);
@@ -1671,12 +1670,7 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
     }
 
     /* Set NUMA node for the added CPUs  */
-    for (i = 0; i < nb_numa_nodes; i++) {
-        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
-            cs->numa_node = i;
-            break;
-        }
-    }
+    numa_set_cpu(cs);
 
     xics_cpu_setup(spapr->icp, cpu);
     qemu_register_reset(spapr_cpu_reset, cpu);
diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
index bb184c9..648d68b 100644
--- a/include/sysemu/numa.h
+++ b/include/sysemu/numa.h
@@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
 void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
 uint32_t numa_get_node(ram_addr_t addr, Error **errp);
+void numa_set_cpu(CPUState *cpu);
 
 #endif
diff --git a/numa.c b/numa.c
index 4c4f7f5..1b47c15 100644
--- a/numa.c
+++ b/numa.c
@@ -396,20 +396,32 @@ void parse_numa_opts(MachineClass *mc)
     }
 }
 
+static void numa_set_cpu_numa_node(CPUState *cpu)
+{
+    int i;
+
+    for (i = 0; i < nb_numa_nodes; i++) {
+        if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
+            cpu->numa_node = i;
+            break;
+        }
+    }
+}
+
 void numa_post_machine_init(void)
 {
     CPUState *cpu;
-    int i;
 
     CPU_FOREACH(cpu) {
-        for (i = 0; i < nb_numa_nodes; i++) {
-            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
-                cpu->numa_node = i;
-            }
-        }
+        numa_set_cpu_numa_node(cpu);
     }
 }
 
+void numa_set_cpu(CPUState *cpu)
+{
+    numa_set_cpu_numa_node(cpu);
+}
+
 static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
                                            const char *name,
                                            uint64_t ram_size)

-------------------------------------------------------------------
> 
> >      xics_cpu_setup(spapr->icp, cpu);
> >      qemu_register_reset(spapr_cpu_reset, cpu);
> >  }
> > @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine)
> >      char *filename;
> >      int spapr_cores = smp_cpus / smp_threads;
> >      int spapr_max_cores = max_cpus / smp_threads;
> > +    int smt = kvmppc_smt_threads();
> >  
> >      if (smp_cpus % smp_threads) {
> >          error_report("smp_cpus (%u) must be multiple of threads (%u)",
> > @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine)
> >          spapr_validate_node_memory(machine, &error_fatal);
> >      }
> >  
> > +    if (smc->dr_cpu_enabled) {
> > +        for (i = 0; i < spapr_max_cores; i++) {
> > +            sPAPRDRConnector *drc =
> > +                spapr_dr_connector_new(OBJECT(spapr),
> > +                                       SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt);
> > +            qemu_register_reset(spapr_drc_reset, drc);
> > +        }
> > +    }
> > +
> 
> Nit: would this be cleaner to include in the same loop that constructs
> the (empty) links and boot-time cpu cores?

Seems possible, will change.

> 
> >      /* init CPUs */
> >      if (machine->cpu_model == NULL) {
> >          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
> > @@ -2267,6 +2309,27 @@ out:
> >      error_propagate(errp, local_err);
> >  }
> >  
> > +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
> > +                                    int *fdt_offset, sPAPRMachineState *spapr)
> > +{
> > +    PowerPCCPU *cpu = POWERPC_CPU(cs);
> > +    DeviceClass *dc = DEVICE_GET_CLASS(cs);
> > +    int id = ppc_get_vcpu_dt_id(cpu);
> > +    void *fdt;
> > +    int offset, fdt_size;
> > +    char *nodename;
> > +
> > +    fdt = create_device_tree(&fdt_size);
> > +    nodename = g_strdup_printf("%s@%x", dc->fw_name, id);
> > +    offset = fdt_add_subnode(fdt, 0, nodename);
> > +
> > +    spapr_populate_cpu_dt(cs, fdt, offset, spapr);
> > +    g_free(nodename);
> > +
> > +    *fdt_offset = offset;
> > +    return fdt;
> > +}
> > +
> >  static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> >                                        DeviceState *dev, Error **errp)
> >  {
> > @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> >          }
> >  
> >          spapr_memory_plug(hotplug_dev, dev, node, errp);
> > +    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
> > +        if (!smc->dr_cpu_enabled && dev->hotplugged) {
> > +            error_setg(errp, "CPU hotplug not supported for this machine");
> > +            return;
> > +        }
> > +        spapr_core_plug(hotplug_dev, dev, errp);
> >      }
> >  }
> >  
> > @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
> >      mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
> >  
> >      smc->dr_lmb_enabled = true;
> > +    smc->dr_cpu_enabled = true;
> >      fwc->get_dev_path = spapr_get_fw_dev_path;
> >      nc->nmi_monitor_handler = spapr_nmi;
> >  }
> > @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc)
> >  
> >      spapr_machine_2_6_class_options(mc);
> >      smc->use_ohci_by_default = true;
> > +    smc->dr_cpu_enabled = false;
> >      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5);
> >  }
> >  
> > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > index 9ddf3ce..4c233d7 100644
> > --- a/hw/ppc/spapr_cpu_core.c
> > +++ b/hw/ppc/spapr_cpu_core.c
> > @@ -14,6 +14,65 @@
> >  #include "qapi/visitor.h"
> >  #include <sysemu/cpus.h>
> >  
> > +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > +                     Error **errp)
> > +{
> > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > +    sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine());
> > +    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
> > +    PowerPCCPU *cpu = &core->threads[0];
> > +    CPUState *cs = CPU(cpu);
> > +    int id = ppc_get_vcpu_dt_id(cpu);
> > +    sPAPRDRConnector *drc =
> > +        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
> > +    sPAPRDRConnectorClass *drck;
> > +    Error *local_err = NULL;
> > +    void *fdt = NULL;
> > +    int fdt_offset = 0;
> > +
> > +    if (!smc->dr_cpu_enabled) {
> > +        /*
> > +         * This is a cold plugged CPU core but the machine doesn't support
> > +         * DR. So skip the hotplug path ensuring that the core is brought
> > +         * up online with out an associated DR connector.
> > +         */
> > +        return;
> > +    }
> > +
> > +    g_assert(drc);
> > +
> > +    /*
> > +     * Setup CPU DT entries only for hotplugged CPUs. For boot time or
> > +     * coldplugged CPUs DT entries are setup in spapr_finalize_fdt().
> > +     */
> > +    if (dev->hotplugged) {
> > +        fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms);
> > +        dev->hotplugged = true;
> > +    }
> > +
> > +    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > +    drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err);
> > +    if (local_err) {
> > +        g_free(fdt);
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    if (dev->hotplugged) {
> > +        /*
> > +         * Send hotplug notification interrupt to the guest only in case
> > +         * of hotplugged CPUs.
> > +         */
> > +        spapr_hotplug_req_add_by_index(drc);
> > +    } else {
> > +        /*
> > +         * Set the right DRC states for cold plugged CPU.
> > +         */
> > +        drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE);
> > +        drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED);
> > +    }
> > +}
> > +
> >  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> >  {
> >      Error **errp = opaque;
> > @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> >      if (*errp) {
> >          return 1;
> >      }
> > +    spapr_cpu_reset(cpu);
> 
> This also looks like it belongs in a different patch.

You mean a separate patch for this or push this around to an existing
patch of the series ?

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Igor Mammedov]

On Mon, 7 Mar 2016 11:59:42 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:
> > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:  
> > > Set up device tree entries for the hotplugged CPU core and use the
> > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > the guest.
> > > 
> > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > ---
> > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > >  hw/ppc/spapr_events.c           |  3 ++
> > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > >  include/hw/ppc/spapr.h          |  4 +++
> > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > index 5acb612..6c4ac50 100644
> > > --- a/hw/ppc/spapr.c
> > > +++ b/hw/ppc/spapr.c
> > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > >      size_t page_sizes_prop_size;
> > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > +    sPAPRDRConnector *drc;
> > > +    sPAPRDRConnectorClass *drck;
> > > +    int drc_index;
> > > +
> > > +    if (smc->dr_cpu_enabled) {
> > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > +        g_assert(drc);
> > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > +        drc_index = drck->get_index(drc);
> > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > +    }
> > >  
> > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > >       * provisioned with large pages might otherwise try to map a qemu
> > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > >      }
> > >  
> > > +    if (smc->dr_cpu_enabled) {
> > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > +        if (ret < 0) {
> > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > +            exit(1);
> > > +        }
> > > +    }
> > > +
> > >      _FDT((fdt_pack(fdt)));
> > >  
> > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > >  
> > >  }
> > >  
> > > -static void spapr_cpu_reset(void *opaque)
> > > +void spapr_cpu_reset(void *opaque)
> > >  {
> > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > >      PowerPCCPU *cpu = opaque;
> > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > >  {
> > >      CPUPPCState *env = &cpu->env;
> > > +    CPUState *cs = CPU(cpu);
> > > +    int i;
> > >  
> > >      /* Set time-base frequency to 512 MHz */
> > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > >          }
> > >      }
> > >  
> > > +    /* Set NUMA node for the added CPUs  */
> > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > +            cs->numa_node = i;
> > > +            break;
> > > +        }
> > > +    }
> > > +  
> > 
> > This hunk seems like it belongs in a different patch.  
> 
> It appears that this would be needed by other archs also to set the
> NUMA node for the hot-plugged CPU. How about make an API out of this
> and use this something like below ? Igor ?
I don't think we should try to make it generic (at leas for now),
each target might have its own notion of cpu to node mapping.

As I see -numa CLI interface is somewhat broken and allow(s|ed)
to create nonsense configurations. But it's a separate topic,
which I'd leave out this discussion.

--
I have a not completely thought out idea to obsolete -numa CLI
in favor of "-device some-type,node=XX" where 'node' property
could replace current -numa interface.
For example on x86 for memory -numa is used only for initial
memory and we already use -device pc-dimm,node=XX for hotpluggable memory.
However if initial memory is converted to pc-dimm devices, then
it could be possible to replace -numa mem CLI with -device pc-dimm
as well.
Applying the same idea to CPUs could obsolete -numa cpus CLI.

> 
> -------------------------------------------------------------------
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 0aeefd2..8347234 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1112,6 +1112,7 @@ void pc_hot_add_cpu(const int64_t id, Error **errp)
>          error_propagate(errp, local_err);
>          return;
>      }
> +    numa_set_cpu(CPU(cpu));
>      object_unref(OBJECT(cpu));
>  }
>  
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index a42f8c0..f2b3b67 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1645,7 +1645,6 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>  {
>      CPUPPCState *env = &cpu->env;
>      CPUState *cs = CPU(cpu);
> -    int i;
>  
>      /* Set time-base frequency to 512 MHz */
>      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> @@ -1671,12 +1670,7 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>      }
>  
>      /* Set NUMA node for the added CPUs  */
> -    for (i = 0; i < nb_numa_nodes; i++) {
> -        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> -            cs->numa_node = i;
> -            break;
> -        }
> -    }
> +    numa_set_cpu(cs);
>  
>      xics_cpu_setup(spapr->icp, cpu);
>      qemu_register_reset(spapr_cpu_reset, cpu);
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index bb184c9..648d68b 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
>  void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
>  void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
>  uint32_t numa_get_node(ram_addr_t addr, Error **errp);
> +void numa_set_cpu(CPUState *cpu);
>  
>  #endif
> diff --git a/numa.c b/numa.c
> index 4c4f7f5..1b47c15 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -396,20 +396,32 @@ void parse_numa_opts(MachineClass *mc)
>      }
>  }
>  
> +static void numa_set_cpu_numa_node(CPUState *cpu)
> +{
> +    int i;
> +
> +    for (i = 0; i < nb_numa_nodes; i++) {
> +        if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> +            cpu->numa_node = i;
> +            break;
> +        }
> +    }
> +}
> +
>  void numa_post_machine_init(void)
>  {
>      CPUState *cpu;
> -    int i;
>  
>      CPU_FOREACH(cpu) {
> -        for (i = 0; i < nb_numa_nodes; i++) {
> -            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> -                cpu->numa_node = i;
> -            }
> -        }
> +        numa_set_cpu_numa_node(cpu);
>      }
>  }
>  
> +void numa_set_cpu(CPUState *cpu)
> +{
> +    numa_set_cpu_numa_node(cpu);
> +}
> +
>  static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
>                                             const char *name,
>                                             uint64_t ram_size)
> 
> -------------------------------------------------------------------
> >   
> > >      xics_cpu_setup(spapr->icp, cpu);
> > >      qemu_register_reset(spapr_cpu_reset, cpu);
> > >  }
> > > @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine)
> > >      char *filename;
> > >      int spapr_cores = smp_cpus / smp_threads;
> > >      int spapr_max_cores = max_cpus / smp_threads;
> > > +    int smt = kvmppc_smt_threads();
> > >  
> > >      if (smp_cpus % smp_threads) {
> > >          error_report("smp_cpus (%u) must be multiple of threads (%u)",
> > > @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine)
> > >          spapr_validate_node_memory(machine, &error_fatal);
> > >      }
> > >  
> > > +    if (smc->dr_cpu_enabled) {
> > > +        for (i = 0; i < spapr_max_cores; i++) {
> > > +            sPAPRDRConnector *drc =
> > > +                spapr_dr_connector_new(OBJECT(spapr),
> > > +                                       SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt);
> > > +            qemu_register_reset(spapr_drc_reset, drc);
> > > +        }
> > > +    }
> > > +  
> > 
> > Nit: would this be cleaner to include in the same loop that constructs
> > the (empty) links and boot-time cpu cores?  
> 
> Seems possible, will change.
> 
> >   
> > >      /* init CPUs */
> > >      if (machine->cpu_model == NULL) {
> > >          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
> > > @@ -2267,6 +2309,27 @@ out:
> > >      error_propagate(errp, local_err);
> > >  }
> > >  
> > > +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
> > > +                                    int *fdt_offset, sPAPRMachineState *spapr)
> > > +{
> > > +    PowerPCCPU *cpu = POWERPC_CPU(cs);
> > > +    DeviceClass *dc = DEVICE_GET_CLASS(cs);
> > > +    int id = ppc_get_vcpu_dt_id(cpu);
> > > +    void *fdt;
> > > +    int offset, fdt_size;
> > > +    char *nodename;
> > > +
> > > +    fdt = create_device_tree(&fdt_size);
> > > +    nodename = g_strdup_printf("%s@%x", dc->fw_name, id);
> > > +    offset = fdt_add_subnode(fdt, 0, nodename);
> > > +
> > > +    spapr_populate_cpu_dt(cs, fdt, offset, spapr);
> > > +    g_free(nodename);
> > > +
> > > +    *fdt_offset = offset;
> > > +    return fdt;
> > > +}
> > > +
> > >  static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> > >                                        DeviceState *dev, Error **errp)
> > >  {
> > > @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> > >          }
> > >  
> > >          spapr_memory_plug(hotplug_dev, dev, node, errp);
> > > +    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
> > > +        if (!smc->dr_cpu_enabled && dev->hotplugged) {
> > > +            error_setg(errp, "CPU hotplug not supported for this machine");
> > > +            return;
> > > +        }
> > > +        spapr_core_plug(hotplug_dev, dev, errp);
> > >      }
> > >  }
> > >  
> > > @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
> > >      mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
> > >  
> > >      smc->dr_lmb_enabled = true;
> > > +    smc->dr_cpu_enabled = true;
> > >      fwc->get_dev_path = spapr_get_fw_dev_path;
> > >      nc->nmi_monitor_handler = spapr_nmi;
> > >  }
> > > @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc)
> > >  
> > >      spapr_machine_2_6_class_options(mc);
> > >      smc->use_ohci_by_default = true;
> > > +    smc->dr_cpu_enabled = false;
> > >      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5);
> > >  }
> > >  
> > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > > index 9ddf3ce..4c233d7 100644
> > > --- a/hw/ppc/spapr_cpu_core.c
> > > +++ b/hw/ppc/spapr_cpu_core.c
> > > @@ -14,6 +14,65 @@
> > >  #include "qapi/visitor.h"
> > >  #include <sysemu/cpus.h>
> > >  
> > > +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > > +                     Error **errp)
> > > +{
> > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > +    sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine());
> > > +    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
> > > +    PowerPCCPU *cpu = &core->threads[0];
> > > +    CPUState *cs = CPU(cpu);
> > > +    int id = ppc_get_vcpu_dt_id(cpu);
> > > +    sPAPRDRConnector *drc =
> > > +        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
> > > +    sPAPRDRConnectorClass *drck;
> > > +    Error *local_err = NULL;
> > > +    void *fdt = NULL;
> > > +    int fdt_offset = 0;
> > > +
> > > +    if (!smc->dr_cpu_enabled) {
> > > +        /*
> > > +         * This is a cold plugged CPU core but the machine doesn't support
> > > +         * DR. So skip the hotplug path ensuring that the core is brought
> > > +         * up online with out an associated DR connector.
> > > +         */
> > > +        return;
> > > +    }
> > > +
> > > +    g_assert(drc);
> > > +
> > > +    /*
> > > +     * Setup CPU DT entries only for hotplugged CPUs. For boot time or
> > > +     * coldplugged CPUs DT entries are setup in spapr_finalize_fdt().
> > > +     */
> > > +    if (dev->hotplugged) {
> > > +        fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms);
> > > +        dev->hotplugged = true;
> > > +    }
> > > +
> > > +    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > +    drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err);
> > > +    if (local_err) {
> > > +        g_free(fdt);
> > > +        error_propagate(errp, local_err);
> > > +        return;
> > > +    }
> > > +
> > > +    if (dev->hotplugged) {
> > > +        /*
> > > +         * Send hotplug notification interrupt to the guest only in case
> > > +         * of hotplugged CPUs.
> > > +         */
> > > +        spapr_hotplug_req_add_by_index(drc);
> > > +    } else {
> > > +        /*
> > > +         * Set the right DRC states for cold plugged CPU.
> > > +         */
> > > +        drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE);
> > > +        drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED);
> > > +    }
> > > +}
> > > +
> > >  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> > >  {
> > >      Error **errp = opaque;
> > > @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> > >      if (*errp) {
> > >          return 1;
> > >      }
> > > +    spapr_cpu_reset(cpu);  
> > 
> > This also looks like it belongs in a different patch.  
> 
> You mean a separate patch for this or push this around to an existing
> patch of the series ?
> 
> Regards,
> Bharata.
> 

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 14315 bytes --]

On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote:
> On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:
> > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:
> > > Set up device tree entries for the hotplugged CPU core and use the
> > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > the guest.
> > > 
> > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > ---
> > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > >  hw/ppc/spapr_events.c           |  3 ++
> > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > >  include/hw/ppc/spapr.h          |  4 +++
> > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > index 5acb612..6c4ac50 100644
> > > --- a/hw/ppc/spapr.c
> > > +++ b/hw/ppc/spapr.c
> > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > >      size_t page_sizes_prop_size;
> > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > +    sPAPRDRConnector *drc;
> > > +    sPAPRDRConnectorClass *drck;
> > > +    int drc_index;
> > > +
> > > +    if (smc->dr_cpu_enabled) {
> > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > +        g_assert(drc);
> > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > +        drc_index = drck->get_index(drc);
> > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > +    }
> > >  
> > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > >       * provisioned with large pages might otherwise try to map a qemu
> > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > >      }
> > >  
> > > +    if (smc->dr_cpu_enabled) {
> > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > +        if (ret < 0) {
> > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > +            exit(1);
> > > +        }
> > > +    }
> > > +
> > >      _FDT((fdt_pack(fdt)));
> > >  
> > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > >  
> > >  }
> > >  
> > > -static void spapr_cpu_reset(void *opaque)
> > > +void spapr_cpu_reset(void *opaque)
> > >  {
> > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > >      PowerPCCPU *cpu = opaque;
> > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > >  {
> > >      CPUPPCState *env = &cpu->env;
> > > +    CPUState *cs = CPU(cpu);
> > > +    int i;
> > >  
> > >      /* Set time-base frequency to 512 MHz */
> > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > >          }
> > >      }
> > >  
> > > +    /* Set NUMA node for the added CPUs  */
> > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > +            cs->numa_node = i;
> > > +            break;
> > > +        }
> > > +    }
> > > +
> > 
> > This hunk seems like it belongs in a different patch.
> 
> It appears that this would be needed by other archs also to set the
> NUMA node for the hot-plugged CPU. How about make an API out of this
> and use this something like below ? Igor ?

Is there a way we could put this in the the CPU thread initialization
itself?  Rather than requiring every platform to call a helper.

> -------------------------------------------------------------------
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 0aeefd2..8347234 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1112,6 +1112,7 @@ void pc_hot_add_cpu(const int64_t id, Error **errp)
>          error_propagate(errp, local_err);
>          return;
>      }
> +    numa_set_cpu(CPU(cpu));
>      object_unref(OBJECT(cpu));
>  }
>  
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index a42f8c0..f2b3b67 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1645,7 +1645,6 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>  {
>      CPUPPCState *env = &cpu->env;
>      CPUState *cs = CPU(cpu);
> -    int i;
>  
>      /* Set time-base frequency to 512 MHz */
>      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> @@ -1671,12 +1670,7 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
>      }
>  
>      /* Set NUMA node for the added CPUs  */
> -    for (i = 0; i < nb_numa_nodes; i++) {
> -        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> -            cs->numa_node = i;
> -            break;
> -        }
> -    }
> +    numa_set_cpu(cs);
>  
>      xics_cpu_setup(spapr->icp, cpu);
>      qemu_register_reset(spapr_cpu_reset, cpu);
> diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> index bb184c9..648d68b 100644
> --- a/include/sysemu/numa.h
> +++ b/include/sysemu/numa.h
> @@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
>  void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
>  void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
>  uint32_t numa_get_node(ram_addr_t addr, Error **errp);
> +void numa_set_cpu(CPUState *cpu);
>  
>  #endif
> diff --git a/numa.c b/numa.c
> index 4c4f7f5..1b47c15 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -396,20 +396,32 @@ void parse_numa_opts(MachineClass *mc)
>      }
>  }
>  
> +static void numa_set_cpu_numa_node(CPUState *cpu)
> +{
> +    int i;
> +
> +    for (i = 0; i < nb_numa_nodes; i++) {
> +        if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> +            cpu->numa_node = i;
> +            break;
> +        }
> +    }
> +}
> +
>  void numa_post_machine_init(void)
>  {
>      CPUState *cpu;
> -    int i;
>  
>      CPU_FOREACH(cpu) {
> -        for (i = 0; i < nb_numa_nodes; i++) {
> -            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> -                cpu->numa_node = i;
> -            }
> -        }
> +        numa_set_cpu_numa_node(cpu);
>      }
>  }
>  
> +void numa_set_cpu(CPUState *cpu)
> +{
> +    numa_set_cpu_numa_node(cpu);
> +}
> +
>  static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
>                                             const char *name,
>                                             uint64_t ram_size)
> 
> -------------------------------------------------------------------
> > 
> > >      xics_cpu_setup(spapr->icp, cpu);
> > >      qemu_register_reset(spapr_cpu_reset, cpu);
> > >  }
> > > @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine)
> > >      char *filename;
> > >      int spapr_cores = smp_cpus / smp_threads;
> > >      int spapr_max_cores = max_cpus / smp_threads;
> > > +    int smt = kvmppc_smt_threads();
> > >  
> > >      if (smp_cpus % smp_threads) {
> > >          error_report("smp_cpus (%u) must be multiple of threads (%u)",
> > > @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine)
> > >          spapr_validate_node_memory(machine, &error_fatal);
> > >      }
> > >  
> > > +    if (smc->dr_cpu_enabled) {
> > > +        for (i = 0; i < spapr_max_cores; i++) {
> > > +            sPAPRDRConnector *drc =
> > > +                spapr_dr_connector_new(OBJECT(spapr),
> > > +                                       SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt);
> > > +            qemu_register_reset(spapr_drc_reset, drc);
> > > +        }
> > > +    }
> > > +
> > 
> > Nit: would this be cleaner to include in the same loop that constructs
> > the (empty) links and boot-time cpu cores?
> 
> Seems possible, will change.
> 
> > 
> > >      /* init CPUs */
> > >      if (machine->cpu_model == NULL) {
> > >          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
> > > @@ -2267,6 +2309,27 @@ out:
> > >      error_propagate(errp, local_err);
> > >  }
> > >  
> > > +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
> > > +                                    int *fdt_offset, sPAPRMachineState *spapr)
> > > +{
> > > +    PowerPCCPU *cpu = POWERPC_CPU(cs);
> > > +    DeviceClass *dc = DEVICE_GET_CLASS(cs);
> > > +    int id = ppc_get_vcpu_dt_id(cpu);
> > > +    void *fdt;
> > > +    int offset, fdt_size;
> > > +    char *nodename;
> > > +
> > > +    fdt = create_device_tree(&fdt_size);
> > > +    nodename = g_strdup_printf("%s@%x", dc->fw_name, id);
> > > +    offset = fdt_add_subnode(fdt, 0, nodename);
> > > +
> > > +    spapr_populate_cpu_dt(cs, fdt, offset, spapr);
> > > +    g_free(nodename);
> > > +
> > > +    *fdt_offset = offset;
> > > +    return fdt;
> > > +}
> > > +
> > >  static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> > >                                        DeviceState *dev, Error **errp)
> > >  {
> > > @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> > >          }
> > >  
> > >          spapr_memory_plug(hotplug_dev, dev, node, errp);
> > > +    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
> > > +        if (!smc->dr_cpu_enabled && dev->hotplugged) {
> > > +            error_setg(errp, "CPU hotplug not supported for this machine");
> > > +            return;
> > > +        }
> > > +        spapr_core_plug(hotplug_dev, dev, errp);
> > >      }
> > >  }
> > >  
> > > @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
> > >      mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
> > >  
> > >      smc->dr_lmb_enabled = true;
> > > +    smc->dr_cpu_enabled = true;
> > >      fwc->get_dev_path = spapr_get_fw_dev_path;
> > >      nc->nmi_monitor_handler = spapr_nmi;
> > >  }
> > > @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc)
> > >  
> > >      spapr_machine_2_6_class_options(mc);
> > >      smc->use_ohci_by_default = true;
> > > +    smc->dr_cpu_enabled = false;
> > >      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5);
> > >  }
> > >  
> > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > > index 9ddf3ce..4c233d7 100644
> > > --- a/hw/ppc/spapr_cpu_core.c
> > > +++ b/hw/ppc/spapr_cpu_core.c
> > > @@ -14,6 +14,65 @@
> > >  #include "qapi/visitor.h"
> > >  #include <sysemu/cpus.h>
> > >  
> > > +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > > +                     Error **errp)
> > > +{
> > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > +    sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine());
> > > +    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
> > > +    PowerPCCPU *cpu = &core->threads[0];
> > > +    CPUState *cs = CPU(cpu);
> > > +    int id = ppc_get_vcpu_dt_id(cpu);
> > > +    sPAPRDRConnector *drc =
> > > +        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
> > > +    sPAPRDRConnectorClass *drck;
> > > +    Error *local_err = NULL;
> > > +    void *fdt = NULL;
> > > +    int fdt_offset = 0;
> > > +
> > > +    if (!smc->dr_cpu_enabled) {
> > > +        /*
> > > +         * This is a cold plugged CPU core but the machine doesn't support
> > > +         * DR. So skip the hotplug path ensuring that the core is brought
> > > +         * up online with out an associated DR connector.
> > > +         */
> > > +        return;
> > > +    }
> > > +
> > > +    g_assert(drc);
> > > +
> > > +    /*
> > > +     * Setup CPU DT entries only for hotplugged CPUs. For boot time or
> > > +     * coldplugged CPUs DT entries are setup in spapr_finalize_fdt().
> > > +     */
> > > +    if (dev->hotplugged) {
> > > +        fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms);
> > > +        dev->hotplugged = true;
> > > +    }
> > > +
> > > +    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > +    drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err);
> > > +    if (local_err) {
> > > +        g_free(fdt);
> > > +        error_propagate(errp, local_err);
> > > +        return;
> > > +    }
> > > +
> > > +    if (dev->hotplugged) {
> > > +        /*
> > > +         * Send hotplug notification interrupt to the guest only in case
> > > +         * of hotplugged CPUs.
> > > +         */
> > > +        spapr_hotplug_req_add_by_index(drc);
> > > +    } else {
> > > +        /*
> > > +         * Set the right DRC states for cold plugged CPU.
> > > +         */
> > > +        drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE);
> > > +        drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED);
> > > +    }
> > > +}
> > > +
> > >  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> > >  {
> > >      Error **errp = opaque;
> > > @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> > >      if (*errp) {
> > >          return 1;
> > >      }
> > > +    spapr_cpu_reset(cpu);
> > 
> > This also looks like it belongs in a different patch.
> 
> You mean a separate patch for this or push this around to an existing
> patch of the series ?
> 
> Regards,
> Bharata.
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Igor Mammedov]

On Tue, 8 Mar 2016 15:27:39 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote:
> > On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:  
> > > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:  
> > > > Set up device tree entries for the hotplugged CPU core and use the
> > > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > > the guest.
> > > > 
> > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > ---
> > > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > > >  hw/ppc/spapr_events.c           |  3 ++
> > > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > > >  include/hw/ppc/spapr.h          |  4 +++
> > > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > index 5acb612..6c4ac50 100644
> > > > --- a/hw/ppc/spapr.c
> > > > +++ b/hw/ppc/spapr.c
> > > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > > >      size_t page_sizes_prop_size;
> > > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > > +    sPAPRDRConnector *drc;
> > > > +    sPAPRDRConnectorClass *drck;
> > > > +    int drc_index;
> > > > +
> > > > +    if (smc->dr_cpu_enabled) {
> > > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > > +        g_assert(drc);
> > > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > > +        drc_index = drck->get_index(drc);
> > > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > > +    }
> > > >  
> > > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > > >       * provisioned with large pages might otherwise try to map a qemu
> > > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > > >      }
> > > >  
> > > > +    if (smc->dr_cpu_enabled) {
> > > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > > +        if (ret < 0) {
> > > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > > +            exit(1);
> > > > +        }
> > > > +    }
> > > > +
> > > >      _FDT((fdt_pack(fdt)));
> > > >  
> > > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > > >  
> > > >  }
> > > >  
> > > > -static void spapr_cpu_reset(void *opaque)
> > > > +void spapr_cpu_reset(void *opaque)
> > > >  {
> > > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > > >      PowerPCCPU *cpu = opaque;
> > > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > >  {
> > > >      CPUPPCState *env = &cpu->env;
> > > > +    CPUState *cs = CPU(cpu);
> > > > +    int i;
> > > >  
> > > >      /* Set time-base frequency to 512 MHz */
> > > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > >          }
> > > >      }
> > > >  
> > > > +    /* Set NUMA node for the added CPUs  */
> > > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > > +            cs->numa_node = i;
> > > > +            break;
> > > > +        }
> > > > +    }
> > > > +  
> > > 
> > > This hunk seems like it belongs in a different patch.  
> > 
> > It appears that this would be needed by other archs also to set the
> > NUMA node for the hot-plugged CPU. How about make an API out of this
> > and use this something like below ? Igor ?  
> 
> Is there a way we could put this in the the CPU thread initialization
> itself?  Rather than requiring every platform to call a helper.
I'd suggest hotplugable CPU entity to have 'node' property, like we have
in pc-dimm.

However machine owns numa mapping, so setting it from thread
initialization seems to be wrong.
Could that be done from machine's plug() handler (spapr_core_plug)?

Also I notice that there is in no way to check/set 'address' properties
at machine level before calling cpu->realize(), which makes us
to attempt checking them inside cpu->realize() or at post realize time
in HotplugHandler(machine)->plug() hook with following full rollback
in case of failure.
Maybe we need add an HotplugHandler->pre_realize() hook to allow machine
verify/set properties of CPU before it's realized.

> 
> > -------------------------------------------------------------------
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > index 0aeefd2..8347234 100644
> > --- a/hw/i386/pc.c
> > +++ b/hw/i386/pc.c
> > @@ -1112,6 +1112,7 @@ void pc_hot_add_cpu(const int64_t id, Error **errp)
> >          error_propagate(errp, local_err);
> >          return;
> >      }
> > +    numa_set_cpu(CPU(cpu));
> >      object_unref(OBJECT(cpu));
> >  }
> >  
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index a42f8c0..f2b3b67 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -1645,7 +1645,6 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> >  {
> >      CPUPPCState *env = &cpu->env;
> >      CPUState *cs = CPU(cpu);
> > -    int i;
> >  
> >      /* Set time-base frequency to 512 MHz */
> >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > @@ -1671,12 +1670,7 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> >      }
> >  
> >      /* Set NUMA node for the added CPUs  */
> > -    for (i = 0; i < nb_numa_nodes; i++) {
> > -        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > -            cs->numa_node = i;
> > -            break;
> > -        }
> > -    }
> > +    numa_set_cpu(cs);
> >  
> >      xics_cpu_setup(spapr->icp, cpu);
> >      qemu_register_reset(spapr_cpu_reset, cpu);
> > diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h
> > index bb184c9..648d68b 100644
> > --- a/include/sysemu/numa.h
> > +++ b/include/sysemu/numa.h
> > @@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts;
> >  void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
> >  void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node);
> >  uint32_t numa_get_node(ram_addr_t addr, Error **errp);
> > +void numa_set_cpu(CPUState *cpu);
> >  
> >  #endif
> > diff --git a/numa.c b/numa.c
> > index 4c4f7f5..1b47c15 100644
> > --- a/numa.c
> > +++ b/numa.c
> > @@ -396,20 +396,32 @@ void parse_numa_opts(MachineClass *mc)
> >      }
> >  }
> >  
> > +static void numa_set_cpu_numa_node(CPUState *cpu)
> > +{
> > +    int i;
> > +
> > +    for (i = 0; i < nb_numa_nodes; i++) {
> > +        if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> > +            cpu->numa_node = i;
> > +            break;
> > +        }
> > +    }
> > +}
> > +
> >  void numa_post_machine_init(void)
> >  {
> >      CPUState *cpu;
> > -    int i;
> >  
> >      CPU_FOREACH(cpu) {
> > -        for (i = 0; i < nb_numa_nodes; i++) {
> > -            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
> > -                cpu->numa_node = i;
> > -            }
> > -        }
> > +        numa_set_cpu_numa_node(cpu);
> >      }
> >  }
> >  
> > +void numa_set_cpu(CPUState *cpu)
> > +{
> > +    numa_set_cpu_numa_node(cpu);
> > +}
> > +
> >  static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
> >                                             const char *name,
> >                                             uint64_t ram_size)
> > 
> > -------------------------------------------------------------------  
> > >   
> > > >      xics_cpu_setup(spapr->icp, cpu);
> > > >      qemu_register_reset(spapr_cpu_reset, cpu);
> > > >  }
> > > > @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine)
> > > >      char *filename;
> > > >      int spapr_cores = smp_cpus / smp_threads;
> > > >      int spapr_max_cores = max_cpus / smp_threads;
> > > > +    int smt = kvmppc_smt_threads();
> > > >  
> > > >      if (smp_cpus % smp_threads) {
> > > >          error_report("smp_cpus (%u) must be multiple of threads (%u)",
> > > > @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine)
> > > >          spapr_validate_node_memory(machine, &error_fatal);
> > > >      }
> > > >  
> > > > +    if (smc->dr_cpu_enabled) {
> > > > +        for (i = 0; i < spapr_max_cores; i++) {
> > > > +            sPAPRDRConnector *drc =
> > > > +                spapr_dr_connector_new(OBJECT(spapr),
> > > > +                                       SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt);
> > > > +            qemu_register_reset(spapr_drc_reset, drc);
> > > > +        }
> > > > +    }
> > > > +  
> > > 
> > > Nit: would this be cleaner to include in the same loop that constructs
> > > the (empty) links and boot-time cpu cores?  
> > 
> > Seems possible, will change.
> >   
> > >   
> > > >      /* init CPUs */
> > > >      if (machine->cpu_model == NULL) {
> > > >          machine->cpu_model = kvm_enabled() ? "host" : "POWER7";
> > > > @@ -2267,6 +2309,27 @@ out:
> > > >      error_propagate(errp, local_err);
> > > >  }
> > > >  
> > > > +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
> > > > +                                    int *fdt_offset, sPAPRMachineState *spapr)
> > > > +{
> > > > +    PowerPCCPU *cpu = POWERPC_CPU(cs);
> > > > +    DeviceClass *dc = DEVICE_GET_CLASS(cs);
> > > > +    int id = ppc_get_vcpu_dt_id(cpu);
> > > > +    void *fdt;
> > > > +    int offset, fdt_size;
> > > > +    char *nodename;
> > > > +
> > > > +    fdt = create_device_tree(&fdt_size);
> > > > +    nodename = g_strdup_printf("%s@%x", dc->fw_name, id);
> > > > +    offset = fdt_add_subnode(fdt, 0, nodename);
> > > > +
> > > > +    spapr_populate_cpu_dt(cs, fdt, offset, spapr);
> > > > +    g_free(nodename);
> > > > +
> > > > +    *fdt_offset = offset;
> > > > +    return fdt;
> > > > +}
> > > > +
> > > >  static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> > > >                                        DeviceState *dev, Error **errp)
> > > >  {
> > > > @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
> > > >          }
> > > >  
> > > >          spapr_memory_plug(hotplug_dev, dev, node, errp);
> > > > +    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
> > > > +        if (!smc->dr_cpu_enabled && dev->hotplugged) {
> > > > +            error_setg(errp, "CPU hotplug not supported for this machine");
> > > > +            return;
> > > > +        }
> > > > +        spapr_core_plug(hotplug_dev, dev, errp);
> > > >      }
> > > >  }
> > > >  
> > > > @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
> > > >      mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id;
> > > >  
> > > >      smc->dr_lmb_enabled = true;
> > > > +    smc->dr_cpu_enabled = true;
> > > >      fwc->get_dev_path = spapr_get_fw_dev_path;
> > > >      nc->nmi_monitor_handler = spapr_nmi;
> > > >  }
> > > > @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc)
> > > >  
> > > >      spapr_machine_2_6_class_options(mc);
> > > >      smc->use_ohci_by_default = true;
> > > > +    smc->dr_cpu_enabled = false;
> > > >      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5);
> > > >  }
> > > >  
> > > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> > > > index 9ddf3ce..4c233d7 100644
> > > > --- a/hw/ppc/spapr_cpu_core.c
> > > > +++ b/hw/ppc/spapr_cpu_core.c
> > > > @@ -14,6 +14,65 @@
> > > >  #include "qapi/visitor.h"
> > > >  #include <sysemu/cpus.h>
> > > >  
> > > > +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
> > > > +                     Error **errp)
> > > > +{
> > > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > > +    sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine());
> > > > +    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
> > > > +    PowerPCCPU *cpu = &core->threads[0];
> > > > +    CPUState *cs = CPU(cpu);
> > > > +    int id = ppc_get_vcpu_dt_id(cpu);
> > > > +    sPAPRDRConnector *drc =
> > > > +        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
> > > > +    sPAPRDRConnectorClass *drck;
> > > > +    Error *local_err = NULL;
> > > > +    void *fdt = NULL;
> > > > +    int fdt_offset = 0;
> > > > +
> > > > +    if (!smc->dr_cpu_enabled) {
> > > > +        /*
> > > > +         * This is a cold plugged CPU core but the machine doesn't support
> > > > +         * DR. So skip the hotplug path ensuring that the core is brought
> > > > +         * up online with out an associated DR connector.
> > > > +         */
> > > > +        return;
> > > > +    }
> > > > +
> > > > +    g_assert(drc);
> > > > +
> > > > +    /*
> > > > +     * Setup CPU DT entries only for hotplugged CPUs. For boot time or
> > > > +     * coldplugged CPUs DT entries are setup in spapr_finalize_fdt().
> > > > +     */
> > > > +    if (dev->hotplugged) {
> > > > +        fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms);
> > > > +        dev->hotplugged = true;
> > > > +    }
> > > > +
> > > > +    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > > +    drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err);
> > > > +    if (local_err) {
> > > > +        g_free(fdt);
> > > > +        error_propagate(errp, local_err);
> > > > +        return;
> > > > +    }
> > > > +
> > > > +    if (dev->hotplugged) {
> > > > +        /*
> > > > +         * Send hotplug notification interrupt to the guest only in case
> > > > +         * of hotplugged CPUs.
> > > > +         */
> > > > +        spapr_hotplug_req_add_by_index(drc);
> > > > +    } else {
> > > > +        /*
> > > > +         * Set the right DRC states for cold plugged CPU.
> > > > +         */
> > > > +        drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE);
> > > > +        drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED);
> > > > +    }
> > > > +}
> > > > +
> > > >  static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> > > >  {
> > > >      Error **errp = opaque;
> > > > @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque)
> > > >      if (*errp) {
> > > >          return 1;
> > > >      }
> > > > +    spapr_cpu_reset(cpu);  
> > > 
> > > This also looks like it belongs in a different patch.  
> > 
> > You mean a separate patch for this or push this around to an existing
> > patch of the series ?
> > 
> > Regards,
> > Bharata.
> >   
> 

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 6100 bytes --]

On Tue, Mar 08, 2016 at 10:37:08AM +0100, Igor Mammedov wrote:
> On Tue, 8 Mar 2016 15:27:39 +1100
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote:
> > > On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:  
> > > > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:  
> > > > > Set up device tree entries for the hotplugged CPU core and use the
> > > > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > > > the guest.
> > > > > 
> > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > ---
> > > > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > > > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > > > >  hw/ppc/spapr_events.c           |  3 ++
> > > > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > > > >  include/hw/ppc/spapr.h          |  4 +++
> > > > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > > > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > > index 5acb612..6c4ac50 100644
> > > > > --- a/hw/ppc/spapr.c
> > > > > +++ b/hw/ppc/spapr.c
> > > > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > > > >      size_t page_sizes_prop_size;
> > > > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > > > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > > > +    sPAPRDRConnector *drc;
> > > > > +    sPAPRDRConnectorClass *drck;
> > > > > +    int drc_index;
> > > > > +
> > > > > +    if (smc->dr_cpu_enabled) {
> > > > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > > > +        g_assert(drc);
> > > > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > > > +        drc_index = drck->get_index(drc);
> > > > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > > > +    }
> > > > >  
> > > > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > > > >       * provisioned with large pages might otherwise try to map a qemu
> > > > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > > > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > > > >      }
> > > > >  
> > > > > +    if (smc->dr_cpu_enabled) {
> > > > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > > > +        if (ret < 0) {
> > > > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > > > +            exit(1);
> > > > > +        }
> > > > > +    }
> > > > > +
> > > > >      _FDT((fdt_pack(fdt)));
> > > > >  
> > > > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > > > >  
> > > > >  }
> > > > >  
> > > > > -static void spapr_cpu_reset(void *opaque)
> > > > > +void spapr_cpu_reset(void *opaque)
> > > > >  {
> > > > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > > > >      PowerPCCPU *cpu = opaque;
> > > > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > > > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > >  {
> > > > >      CPUPPCState *env = &cpu->env;
> > > > > +    CPUState *cs = CPU(cpu);
> > > > > +    int i;
> > > > >  
> > > > >      /* Set time-base frequency to 512 MHz */
> > > > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > >          }
> > > > >      }
> > > > >  
> > > > > +    /* Set NUMA node for the added CPUs  */
> > > > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > > > +            cs->numa_node = i;
> > > > > +            break;
> > > > > +        }
> > > > > +    }
> > > > > +  
> > > > 
> > > > This hunk seems like it belongs in a different patch.  
> > > 
> > > It appears that this would be needed by other archs also to set the
> > > NUMA node for the hot-plugged CPU. How about make an API out of this
> > > and use this something like below ? Igor ?  
> > 
> > Is there a way we could put this in the the CPU thread initialization
> > itself?  Rather than requiring every platform to call a helper.
> I'd suggest hotplugable CPU entity to have 'node' property, like we have
> in pc-dimm.

Ok.  Do you think that makes sense for the base core type (which will
sometimes be a hotpluggable CPU entity and sometimes might not be)?

> However machine owns numa mapping, so setting it from thread
> initialization seems to be wrong.

Ah.. good point.

> Could that be done from machine's plug() handler (spapr_core_plug)?

Yes, I think so.  The core might need a "set node" method, but we
should be able to control this from the plug() handler.

> Also I notice that there is in no way to check/set 'address' properties
> at machine level before calling cpu->realize(), which makes us
> to attempt checking them inside cpu->realize() or at post realize time
> in HotplugHandler(machine)->plug() hook with following full rollback
> in case of failure.
> Maybe we need add an HotplugHandler->pre_realize() hook to allow machine
> verify/set properties of CPU before it's realized.

I don't really see what the problem is with checking inside realize().

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Bharata B Rao]

On Wed, Mar 09, 2016 at 01:58:53PM +1100, David Gibson wrote:
> On Tue, Mar 08, 2016 at 10:37:08AM +0100, Igor Mammedov wrote:
> > On Tue, 8 Mar 2016 15:27:39 +1100
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> > 
> > > On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote:
> > > > On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:  
> > > > > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:  
> > > > > > Set up device tree entries for the hotplugged CPU core and use the
> > > > > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > > > > the guest.
> > > > > > 
> > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > ---
> > > > > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > > > > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > > > > >  hw/ppc/spapr_events.c           |  3 ++
> > > > > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > > > > >  include/hw/ppc/spapr.h          |  4 +++
> > > > > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > > > > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > > > > 
> > > > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > > > index 5acb612..6c4ac50 100644
> > > > > > --- a/hw/ppc/spapr.c
> > > > > > +++ b/hw/ppc/spapr.c
> > > > > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > > > > >      size_t page_sizes_prop_size;
> > > > > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > > > > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > > > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > > > > +    sPAPRDRConnector *drc;
> > > > > > +    sPAPRDRConnectorClass *drck;
> > > > > > +    int drc_index;
> > > > > > +
> > > > > > +    if (smc->dr_cpu_enabled) {
> > > > > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > > > > +        g_assert(drc);
> > > > > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > > > > +        drc_index = drck->get_index(drc);
> > > > > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > > > > +    }
> > > > > >  
> > > > > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > > > > >       * provisioned with large pages might otherwise try to map a qemu
> > > > > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > > > > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > > > > >      }
> > > > > >  
> > > > > > +    if (smc->dr_cpu_enabled) {
> > > > > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > > > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > > > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > > > > +        if (ret < 0) {
> > > > > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > > > > +            exit(1);
> > > > > > +        }
> > > > > > +    }
> > > > > > +
> > > > > >      _FDT((fdt_pack(fdt)));
> > > > > >  
> > > > > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > > > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > > > > >  
> > > > > >  }
> > > > > >  
> > > > > > -static void spapr_cpu_reset(void *opaque)
> > > > > > +void spapr_cpu_reset(void *opaque)
> > > > > >  {
> > > > > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > > > > >      PowerPCCPU *cpu = opaque;
> > > > > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > > > > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > > >  {
> > > > > >      CPUPPCState *env = &cpu->env;
> > > > > > +    CPUState *cs = CPU(cpu);
> > > > > > +    int i;
> > > > > >  
> > > > > >      /* Set time-base frequency to 512 MHz */
> > > > > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > > > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > > >          }
> > > > > >      }
> > > > > >  
> > > > > > +    /* Set NUMA node for the added CPUs  */
> > > > > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > > > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > > > > +            cs->numa_node = i;
> > > > > > +            break;
> > > > > > +        }
> > > > > > +    }
> > > > > > +  
> > > > > 
> > > > > This hunk seems like it belongs in a different patch.  
> > > > 
> > > > It appears that this would be needed by other archs also to set the
> > > > NUMA node for the hot-plugged CPU. How about make an API out of this
> > > > and use this something like below ? Igor ?  
> > > 
> > > Is there a way we could put this in the the CPU thread initialization
> > > itself?  Rather than requiring every platform to call a helper.
> > I'd suggest hotplugable CPU entity to have 'node' property, like we have
> > in pc-dimm.
> 
> Ok.  Do you think that makes sense for the base core type (which will
> sometimes be a hotpluggable CPU entity and sometimes might not be)?
> 
> > However machine owns numa mapping, so setting it from thread
> > initialization seems to be wrong.
> 
> Ah.. good point.
> 
> > Could that be done from machine's plug() handler (spapr_core_plug)?
> 
> Yes, I think so.  The core might need a "set node" method, but we
> should be able to control this from the plug() handler.

I am inclined to leave it wherever it is right now which is in the
thread realization path as it is setting a field in CPUState. When we have
node property for core and when other issues with NUMA cli get addressed
we can move this to core plug handler, ok ?

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Igor Mammedov]

On Wed, 9 Mar 2016 13:23:59 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> On Wed, Mar 09, 2016 at 01:58:53PM +1100, David Gibson wrote:
> > On Tue, Mar 08, 2016 at 10:37:08AM +0100, Igor Mammedov wrote:  
> > > On Tue, 8 Mar 2016 15:27:39 +1100
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > >   
> > > > On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote:  
> > > > > On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:    
> > > > > > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:    
> > > > > > > Set up device tree entries for the hotplugged CPU core and use the
> > > > > > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > > > > > the guest.
> > > > > > > 
> > > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > > ---
> > > > > > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > > > > > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > > > > > >  hw/ppc/spapr_events.c           |  3 ++
> > > > > > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > > > > > >  include/hw/ppc/spapr.h          |  4 +++
> > > > > > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > > > > > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > > > > > 
> > > > > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > > > > index 5acb612..6c4ac50 100644
> > > > > > > --- a/hw/ppc/spapr.c
> > > > > > > +++ b/hw/ppc/spapr.c
> > > > > > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > > > > > >      size_t page_sizes_prop_size;
> > > > > > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > > > > > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > > > > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > > > > > +    sPAPRDRConnector *drc;
> > > > > > > +    sPAPRDRConnectorClass *drck;
> > > > > > > +    int drc_index;
> > > > > > > +
> > > > > > > +    if (smc->dr_cpu_enabled) {
> > > > > > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > > > > > +        g_assert(drc);
> > > > > > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > > > > > +        drc_index = drck->get_index(drc);
> > > > > > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > > > > > +    }
> > > > > > >  
> > > > > > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > > > > > >       * provisioned with large pages might otherwise try to map a qemu
> > > > > > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > > > > > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > > > > > >      }
> > > > > > >  
> > > > > > > +    if (smc->dr_cpu_enabled) {
> > > > > > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > > > > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > > > > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > > > > > +        if (ret < 0) {
> > > > > > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > > > > > +            exit(1);
> > > > > > > +        }
> > > > > > > +    }
> > > > > > > +
> > > > > > >      _FDT((fdt_pack(fdt)));
> > > > > > >  
> > > > > > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > > > > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > > > > > >  
> > > > > > >  }
> > > > > > >  
> > > > > > > -static void spapr_cpu_reset(void *opaque)
> > > > > > > +void spapr_cpu_reset(void *opaque)
> > > > > > >  {
> > > > > > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > > > > > >      PowerPCCPU *cpu = opaque;
> > > > > > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > > > > > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > > > >  {
> > > > > > >      CPUPPCState *env = &cpu->env;
> > > > > > > +    CPUState *cs = CPU(cpu);
> > > > > > > +    int i;
> > > > > > >  
> > > > > > >      /* Set time-base frequency to 512 MHz */
> > > > > > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > > > > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > > > >          }
> > > > > > >      }
> > > > > > >  
> > > > > > > +    /* Set NUMA node for the added CPUs  */
> > > > > > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > > > > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > > > > > +            cs->numa_node = i;
> > > > > > > +            break;
> > > > > > > +        }
> > > > > > > +    }
> > > > > > > +    
> > > > > > 
> > > > > > This hunk seems like it belongs in a different patch.    
> > > > > 
> > > > > It appears that this would be needed by other archs also to set the
> > > > > NUMA node for the hot-plugged CPU. How about make an API out of this
> > > > > and use this something like below ? Igor ?    
> > > > 
> > > > Is there a way we could put this in the the CPU thread initialization
> > > > itself?  Rather than requiring every platform to call a helper.  
> > > I'd suggest hotplugable CPU entity to have 'node' property, like we have
> > > in pc-dimm.  
> > 
> > Ok.  Do you think that makes sense for the base core type (which will
> > sometimes be a hotpluggable CPU entity and sometimes might not be)?
> >   
> > > However machine owns numa mapping, so setting it from thread
> > > initialization seems to be wrong.  
> > 
> > Ah.. good point.
> >   
> > > Could that be done from machine's plug() handler (spapr_core_plug)?  
> > 
> > Yes, I think so.  The core might need a "set node" method, but we
> > should be able to control this from the plug() handler.  
> 
> I am inclined to leave it wherever it is right now which is in the
> thread realization path as it is setting a field in CPUState. When we have
> node property for core and when other issues with NUMA cli get addressed
> we can move this to core plug handler, ok ?
It's up to target maintainers as far as it's not in generic code.

> 
> Regards,
> Bharata.
> 
> 

Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support

[Igor Mammedov]

On Wed, 9 Mar 2016 13:58:53 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Tue, Mar 08, 2016 at 10:37:08AM +0100, Igor Mammedov wrote:
> > On Tue, 8 Mar 2016 15:27:39 +1100
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote:  
> > > > On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote:    
> > > > > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote:    
> > > > > > Set up device tree entries for the hotplugged CPU core and use the
> > > > > > exising EPOW event infrastructure to send CPU hotplug notification to
> > > > > > the guest.
> > > > > > 
> > > > > > Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
> > > > > > ---
> > > > > >  hw/ppc/spapr.c                  | 73 ++++++++++++++++++++++++++++++++++++++++-
> > > > > >  hw/ppc/spapr_cpu_core.c         | 60 +++++++++++++++++++++++++++++++++
> > > > > >  hw/ppc/spapr_events.c           |  3 ++
> > > > > >  hw/ppc/spapr_rtas.c             | 24 ++++++++++++++
> > > > > >  include/hw/ppc/spapr.h          |  4 +++
> > > > > >  include/hw/ppc/spapr_cpu_core.h |  2 ++
> > > > > >  6 files changed, 165 insertions(+), 1 deletion(-)
> > > > > > 
> > > > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > > > index 5acb612..6c4ac50 100644
> > > > > > --- a/hw/ppc/spapr.c
> > > > > > +++ b/hw/ppc/spapr.c
> > > > > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset,
> > > > > >      size_t page_sizes_prop_size;
> > > > > >      uint32_t vcpus_per_socket = smp_threads * smp_cores;
> > > > > >      uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)};
> > > > > > +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> > > > > > +    sPAPRDRConnector *drc;
> > > > > > +    sPAPRDRConnectorClass *drck;
> > > > > > +    int drc_index;
> > > > > > +
> > > > > > +    if (smc->dr_cpu_enabled) {
> > > > > > +        drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index);
> > > > > > +        g_assert(drc);
> > > > > > +        drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > > > > > +        drc_index = drck->get_index(drc);
> > > > > > +        _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index)));
> > > > > > +    }
> > > > > >  
> > > > > >      /* Note: we keep CI large pages off for now because a 64K capable guest
> > > > > >       * provisioned with large pages might otherwise try to map a qemu
> > > > > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr,
> > > > > >          _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB));
> > > > > >      }
> > > > > >  
> > > > > > +    if (smc->dr_cpu_enabled) {
> > > > > > +        int offset = fdt_path_offset(fdt, "/cpus");
> > > > > > +        ret = spapr_drc_populate_dt(fdt, offset, NULL,
> > > > > > +                                    SPAPR_DR_CONNECTOR_TYPE_CPU);
> > > > > > +        if (ret < 0) {
> > > > > > +            error_report("Couldn't set up CPU DR device tree properties");
> > > > > > +            exit(1);
> > > > > > +        }
> > > > > > +    }
> > > > > > +
> > > > > >      _FDT((fdt_pack(fdt)));
> > > > > >  
> > > > > >      if (fdt_totalsize(fdt) > FDT_MAX_SIZE) {
> > > > > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void)
> > > > > >  
> > > > > >  }
> > > > > >  
> > > > > > -static void spapr_cpu_reset(void *opaque)
> > > > > > +void spapr_cpu_reset(void *opaque)
> > > > > >  {
> > > > > >      sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
> > > > > >      PowerPCCPU *cpu = opaque;
> > > > > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device,
> > > > > >  void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > > >  {
> > > > > >      CPUPPCState *env = &cpu->env;
> > > > > > +    CPUState *cs = CPU(cpu);
> > > > > > +    int i;
> > > > > >  
> > > > > >      /* Set time-base frequency to 512 MHz */
> > > > > >      cpu_ppc_tb_init(env, TIMEBASE_FREQ);
> > > > > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp)
> > > > > >          }
> > > > > >      }
> > > > > >  
> > > > > > +    /* Set NUMA node for the added CPUs  */
> > > > > > +    for (i = 0; i < nb_numa_nodes; i++) {
> > > > > > +        if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) {
> > > > > > +            cs->numa_node = i;
> > > > > > +            break;
> > > > > > +        }
> > > > > > +    }
> > > > > > +    
> > > > > 
> > > > > This hunk seems like it belongs in a different patch.    
> > > > 
> > > > It appears that this would be needed by other archs also to set the
> > > > NUMA node for the hot-plugged CPU. How about make an API out of this
> > > > and use this something like below ? Igor ?    
> > > 
> > > Is there a way we could put this in the the CPU thread initialization
> > > itself?  Rather than requiring every platform to call a helper.  
> > I'd suggest hotplugable CPU entity to have 'node' property, like we have
> > in pc-dimm.  
> 
> Ok.  Do you think that makes sense for the base core type (which will
> sometimes be a hotpluggable CPU entity and sometimes might not be)?
I wouldn't push it into generic base type, as in general target
implementing it in subtype might not have a numa concept at all.
I'd be better to leave decision up to target and corresponding subtype.

> 
> > However machine owns numa mapping, so setting it from thread
> > initialization seems to be wrong.  
> 
> Ah.. good point.
> 
> > Could that be done from machine's plug() handler (spapr_core_plug)?  
> 
> Yes, I think so.  The core might need a "set node" method, but we
> should be able to control this from the plug() handler.
> 
> > Also I notice that there is in no way to check/set 'address' properties
> > at machine level before calling cpu->realize(), which makes us
> > to attempt checking them inside cpu->realize() or at post realize time
> > in HotplugHandler(machine)->plug() hook with following full rollback
> > in case of failure.
> > Maybe we need add an HotplugHandler->pre_realize() hook to allow machine
> > verify/set properties of CPU before it's realized.  
> 
> I don't really see what the problem is with checking inside realize().
pls see http://patchwork.ozlabs.org/patch/594196/ for reasoning behind this

[Qemu-devel] [RFC PATCH v1 09/10] xics, xics_kvm: Handle CPU unplug correctly

[Bharata B Rao]

XICS is setup for each CPU during initialization. Provide a routine
to undo the same when CPU is unplugged. While here, move ss->cs management
into xics from xics_kvm since there is nothing KVM specific in it.
Also ensure xics reset doesn't set irq for CPUs that are already unplugged.

This allows reboot of a VM that has undergone CPU hotplug and unplug
to work correctly.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 hw/intc/xics.c        | 14 ++++++++++++++
 hw/intc/xics_kvm.c    |  8 ++++----
 include/hw/ppc/xics.h |  1 +
 3 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/hw/intc/xics.c b/hw/intc/xics.c
index 213a370..9fdb551 100644
--- a/hw/intc/xics.c
+++ b/hw/intc/xics.c
@@ -45,6 +45,18 @@ static int get_cpu_index_by_dt_id(int cpu_dt_id)
     return -1;
 }
 
+void xics_cpu_destroy(XICSState *icp, PowerPCCPU *cpu)
+{
+    CPUState *cs = CPU(cpu);
+    ICPState *ss = &icp->ss[cs->cpu_index];
+
+    assert(cs->cpu_index < icp->nr_servers);
+    assert(cs == ss->cs);
+
+    ss->output = NULL;
+    ss->cs = NULL;
+}
+
 void xics_cpu_setup(XICSState *icp, PowerPCCPU *cpu)
 {
     CPUState *cs = CPU(cpu);
@@ -54,6 +66,8 @@ void xics_cpu_setup(XICSState *icp, PowerPCCPU *cpu)
 
     assert(cs->cpu_index < icp->nr_servers);
 
+    ss->cs = cs;
+
     if (info->cpu_setup) {
         info->cpu_setup(icp, cpu);
     }
diff --git a/hw/intc/xics_kvm.c b/hw/intc/xics_kvm.c
index 9fe0667..7aab4a1 100644
--- a/hw/intc/xics_kvm.c
+++ b/hw/intc/xics_kvm.c
@@ -110,8 +110,10 @@ static void icp_kvm_reset(DeviceState *dev)
     icp->pending_priority = 0xff;
     icp->mfrr = 0xff;
 
-    /* Make all outputs are deasserted */
-    qemu_set_irq(icp->output, 0);
+    /* Make all outputs as deasserted only if the CPU thread is in use */
+    if (icp->output) {
+        qemu_set_irq(icp->output, 0);
+    }
 
     icp_set_kvm_state(icp, 1);
 }
@@ -344,8 +346,6 @@ static void xics_kvm_cpu_setup(XICSState *icp, PowerPCCPU *cpu)
     if (icpkvm->kernel_xics_fd != -1) {
         int ret;
 
-        ss->cs = cs;
-
         ret = kvm_vcpu_enable_cap(cs, KVM_CAP_IRQ_XICS, 0,
                                   icpkvm->kernel_xics_fd, kvm_arch_vcpu_id(cs));
         if (ret < 0) {
diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
index f60b06a..9091054 100644
--- a/include/hw/ppc/xics.h
+++ b/include/hw/ppc/xics.h
@@ -167,5 +167,6 @@ int xics_alloc_block(XICSState *icp, int src, int num, bool lsi, bool align,
 void xics_free(XICSState *icp, int irq, int num);
 
 void xics_cpu_setup(XICSState *icp, PowerPCCPU *cpu);
+void xics_cpu_destroy(XICSState *icp, PowerPCCPU *cpu);
 
 #endif /* __XICS_H__ */
-- 
2.1.0

[Qemu-devel] [RFC PATCH v1 10/10] spapr: CPU hot unplug support

[Bharata B Rao]

Remove the CPU core device by removing the underlying CPU thread devices.
Hot removal of CPU for sPAPR guests is supported by sending the hot unplug
notification to the guest via EPOW interrupt. Release the vCPU object
after CPU hot unplug so that vCPU fd can be parked and reused.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---
 hw/ppc/spapr.c                  | 16 ++++++++
 hw/ppc/spapr_cpu_core.c         | 84 +++++++++++++++++++++++++++++++++++++++++
 include/hw/ppc/spapr.h          |  1 +
 include/hw/ppc/spapr_cpu_core.h | 11 ++++++
 4 files changed, 112 insertions(+)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 6c4ac50..a42f8c0 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -2379,11 +2379,27 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev,
     }
 }
 
+void spapr_cpu_destroy(PowerPCCPU *cpu)
+{
+    sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
+
+    xics_cpu_destroy(spapr->icp, cpu);
+    qemu_unregister_reset(spapr_cpu_reset, cpu);
+}
+
 static void spapr_machine_device_unplug(HotplugHandler *hotplug_dev,
                                       DeviceState *dev, Error **errp)
 {
+    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
+
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         error_setg(errp, "Memory hot unplug not supported by sPAPR");
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) {
+        if (!smc->dr_cpu_enabled) {
+            error_setg(errp, "CPU hot unplug not supported on this machine");
+            return;
+        }
+        spapr_core_unplug(hotplug_dev, dev, errp);
     }
 }
 
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index 4c233d7..5156eb3 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -73,6 +73,90 @@ void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
     }
 }
 
+static void spapr_cpu_core_cleanup(struct sPAPRCPUUnplugList *unplug_list)
+{
+    sPAPRCPUUnplug *unplug, *next;
+    Object *cpu;
+
+    QLIST_FOREACH_SAFE(unplug, unplug_list, node, next) {
+        cpu = unplug->cpu;
+        object_unparent(cpu);
+        QLIST_REMOVE(unplug, node);
+        g_free(unplug);
+    }
+}
+
+static void spapr_add_cpu_to_unplug_list(Object *cpu,
+                                         struct sPAPRCPUUnplugList *unplug_list)
+{
+    sPAPRCPUUnplug *unplug = g_malloc(sizeof(*unplug));
+
+    unplug->cpu = cpu;
+    QLIST_INSERT_HEAD(unplug_list, unplug, node);
+}
+
+static int spapr_cpu_release(Object *obj, void *opaque)
+{
+    DeviceState *dev = DEVICE(obj);
+    CPUState *cs = CPU(dev);
+    PowerPCCPU *cpu = POWERPC_CPU(cs);
+    struct sPAPRCPUUnplugList *unplug_list = opaque;
+
+    spapr_cpu_destroy(cpu);
+    cpu_remove_sync(cs);
+
+    /*
+     * We are still walking the core object's children list, and
+     * hence can't cleanup this CPU thread object just yet. Put
+     * it on a list for later removal.
+     */
+    spapr_add_cpu_to_unplug_list(obj, unplug_list);
+    return 0;
+}
+
+static void spapr_core_release(DeviceState *dev, void *opaque)
+{
+    struct sPAPRCPUUnplugList unplug_list;
+    sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
+    char *slot = object_property_get_str(OBJECT(dev), CPU_CORE_SLOT_PROP,
+                 &error_fatal);
+
+    QLIST_INIT(&unplug_list);
+    object_child_foreach(OBJECT(dev), spapr_cpu_release, &unplug_list);
+    spapr_cpu_core_cleanup(&unplug_list);
+
+    /* Unset the link from machine object to this core */
+    object_property_set_link(OBJECT(spapr), NULL, slot, NULL);
+    g_free(slot);
+
+    g_free(core->threads);
+    object_unparent(OBJECT(dev));
+}
+
+void spapr_core_unplug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                       Error **errp)
+{
+    sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev));
+    PowerPCCPU *cpu = &core->threads[0];
+    int id = ppc_get_vcpu_dt_id(cpu);
+    sPAPRDRConnector *drc =
+        spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id);
+    sPAPRDRConnectorClass *drck;
+    Error *local_err = NULL;
+
+    g_assert(drc);
+
+    drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
+    drck->detach(drc, dev, spapr_core_release, NULL, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    spapr_hotplug_req_remove_by_index(drc);
+}
+
 static int spapr_cpu_core_realize_child(Object *child, void *opaque)
 {
     Error **errp = opaque;
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index d1a0af8..8fab27b 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -593,6 +593,7 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp);
 void spapr_cpu_reset(void *opaque);
 void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs,
                                     int *fdt_offset, sPAPRMachineState *spapr);
+void spapr_cpu_destroy(PowerPCCPU *cpu);
 
 /* rtas-configure-connector state */
 struct sPAPRConfigureConnectorState {
diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h
index 9879398..4638f96 100644
--- a/include/hw/ppc/spapr_cpu_core.h
+++ b/include/hw/ppc/spapr_cpu_core.h
@@ -28,4 +28,15 @@ typedef struct sPAPRCPUCore {
 
 void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
                      Error **errp);
+void spapr_core_unplug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                       Error **errp);
+
+/* List to store unplugged CPU objects for cleanup during unplug */
+typedef struct sPAPRCPUUnplug {
+    Object *cpu;
+    QLIST_ENTRY(sPAPRCPUUnplug) node;
+} sPAPRCPUUnplug;
+
+QLIST_HEAD(sPAPRCPUUnplugList, sPAPRCPUUnplug);
+
 #endif
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH v1 00/10] Core based CPU hotplug for PowerPC sPAPR

[Igor Mammedov]

On Fri,  4 Mar 2016 12:24:11 +0530
Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:

> Hi,
> 
> This is the next version of "Core based CPU hotplug for PowerPC sPAPR" that
> was posted at
> https://lists.gnu.org/archive/html/qemu-ppc/2016-02/msg00286.html
> 
> Here is a quick summary on how this approach is different from the
> previous approaches that I have been pursuing with the last one being
> v7 that posted here:
> https://lists.gnu.org/archive/html/qemu-ppc/2016-01/msg00477.html
> 
> - Earlier approaches used an independent PowerPC specific core device while
>   this approach uses an sPAPR specific core device that is derived from
>   generic core device.
> - The earlier approach didn't have the notion of where the boot time as
>   well as hot-plugged cores would sit. In this approach QOM links are
>   created from machine object to all possible core objects. While the
>   links are set for boot time cores during machine init, the same is done
>   for hotplugged cores at hotplug time. The name of this link property
>   is standardized as "core[N]" since these links link the machine object
>   with core devices. The link name ("core[N]") is used with core device's
>   "slot" property to identify the QOM link to set for this core.
>   (qemu) device_add spapr-cpu-core,id=core2,nr_threads=8,cpu_model=host,slot=core[2]
> - The ealier approach created threads from instance_init of the core object
>   using a modified cpu_generic_init() routine. It used smp_threads and
>   MachineState->cpu_model globals. In the current approach, nr_threads and
>   cpu_model are obtained as properties and threads are created from
>   property setters. The thread objects are allocated as part of core
>   device and object_initialize() is used to initialize them.
> 
> Some open questions remain:
> 
> - Does this device_add semantics look ok ?
>   (qemu) device_add spapr-cpu-core,id=core2,nr_threads=8,cpu_model=host,slot=core[2]
looks fine to me, only I'd do following changes:
s/nr_threads/threads/
s/slot/core/ and make it numeric property

> - Is there need for machine to core object links ? If not, what would
>   determine the slot/location of the core device ?
I'd drop links altogether for this series as they were a part of an attempt
to implement QOM based introspection interface, which is not must have
provided QMP interface would provide all necessary for hotplug information
which links aren't capable of to do.

Back in the days Anthony suggested that we 'might' use links to implement
hotplug because there wasn't any other infrastructure hotplug
infrastructure for bus-less devices. But now we have HOTPLUG_HANDLER
interface that obsoleted, what links were supposed to handle at
link set time hooks.

But somehow we stuck on links idea, though their initial role in hotplug
were obsoleted and we still trying to fit them in design where
they are not really necessary.

> - How to fit this with CPUSlotProperties HMP interface that Igor is
>   working on ?
I'll try to re-factor QMP interface RFC on top of this series
so it would be easier for you to review it wrt spapr.


> This version has the following changes:
> 
> - Dropped QMP and HMP enumeration patches for now since it isn't clear
>   if the approach I took would be preferrable by all archs. Will wait
>   and see how Igor's patches evolve here.
> - Added the following pre-req patches to support CPU removal:
>   exec: Remove cpu from cpus list during cpu_exec_exit()
>   exec: Do vmstate unregistration from cpu_exec_exit()
>   cpu: Reclaim vCPU objects
>   cpu: Add a sync version of cpu_remove()
> - Added sPAPR CPU hot removal support
>   xics,xics_kvm: Handle CPU unplug correctly
>   spapr: CPU hot unplug support
> - Moved up the "slot" property into abstract cpu-core device.
> - Recover when thread creation fails (spapr_cpu_core_create_threads())
> - cpu_model property in spapr-cpu-core deivce is now tracked using
>   ObjectClass pointer instead of string.
> - Fail topologies with incomplete cores from within sPAPR's machine init.
> - Fixes in spapr-cpu-core object creation from machine init to create
>   only boottime CPUs.
> - Moved board specific wiring code for CPU threads (spapr_cpu_init)
>   into core's realize method to be called after each thread's realization.
> - No specific action under TYPE_CPU in ->plug() handler either for hotplug
>   or hot removal.
> - Moved all core related CPU hotplug routines into spapr_cpu_core.c where
>   it truly belongs.
> - Setting of NUMA node for hotplugged CPUs moved to spapr_cpu_init()).
> - Compared to previous implementation of hot removal that was part of
>   different series earlier, this implementation moves all core removal
>   logic into spapr_cpu_core.c.
> - Some minor cleanups like use of g_new0 instead of g_malloc0 etc.
> 
> This patchset is present at:
> https://github.com/bharata/qemu/commits/spapr-cpu-core
> 
> Bharata B Rao (9):
>   exec: Remove cpu from cpus list during cpu_exec_exit()
>   exec: Do vmstate unregistration from cpu_exec_exit()
>   cpu: Add a sync version of cpu_remove()
>   cpu: Abstract CPU core type
>   spapr: CPU core device
>   spapr: Represent boot CPUs as spapr-cpu-core devices
>   spapr: CPU hotplug support
>   xics,xics_kvm: Handle CPU unplug correctly
>   spapr: CPU hot unplug support
> 
> Gu Zheng (1):
>   cpu: Reclaim vCPU objects
> 
>  cpus.c                          |  50 ++++++
>  exec.c                          |  44 ++++-
>  hw/cpu/Makefile.objs            |   1 +
>  hw/cpu/core.c                   |  44 +++++
>  hw/intc/xics.c                  |  14 ++
>  hw/intc/xics_kvm.c              |   8 +-
>  hw/ppc/Makefile.objs            |   1 +
>  hw/ppc/spapr.c                  | 174 +++++++++++++++++--
>  hw/ppc/spapr_cpu_core.c         | 361 ++++++++++++++++++++++++++++++++++++++++
>  hw/ppc/spapr_events.c           |   3 +
>  hw/ppc/spapr_rtas.c             |  24 +++
>  include/hw/cpu/core.h           |  30 ++++
>  include/hw/ppc/spapr.h          |   9 +
>  include/hw/ppc/spapr_cpu_core.h |  42 +++++
>  include/hw/ppc/xics.h           |   1 +
>  include/qom/cpu.h               |  18 ++
>  include/sysemu/kvm.h            |   1 +
>  kvm-all.c                       |  57 ++++++-
>  kvm-stub.c                      |   5 +
>  19 files changed, 862 insertions(+), 25 deletions(-)
>  create mode 100644 hw/cpu/core.c
>  create mode 100644 hw/ppc/spapr_cpu_core.c
>  create mode 100644 include/hw/cpu/core.h
>  create mode 100644 include/hw/ppc/spapr_cpu_core.h
> 

Re: [Qemu-devel] [RFC PATCH v1 00/10] Core based CPU hotplug for PowerPC sPAPR

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 7204 bytes --]

On Fri, Mar 04, 2016 at 11:57:18AM +0100, Igor Mammedov wrote:
> On Fri,  4 Mar 2016 12:24:11 +0530
> Bharata B Rao <bharata@linux.vnet.ibm.com> wrote:
> 
> > Hi,
> > 
> > This is the next version of "Core based CPU hotplug for PowerPC sPAPR" that
> > was posted at
> > https://lists.gnu.org/archive/html/qemu-ppc/2016-02/msg00286.html
> > 
> > Here is a quick summary on how this approach is different from the
> > previous approaches that I have been pursuing with the last one being
> > v7 that posted here:
> > https://lists.gnu.org/archive/html/qemu-ppc/2016-01/msg00477.html
> > 
> > - Earlier approaches used an independent PowerPC specific core device while
> >   this approach uses an sPAPR specific core device that is derived from
> >   generic core device.
> > - The earlier approach didn't have the notion of where the boot time as
> >   well as hot-plugged cores would sit. In this approach QOM links are
> >   created from machine object to all possible core objects. While the
> >   links are set for boot time cores during machine init, the same is done
> >   for hotplugged cores at hotplug time. The name of this link property
> >   is standardized as "core[N]" since these links link the machine object
> >   with core devices. The link name ("core[N]") is used with core device's
> >   "slot" property to identify the QOM link to set for this core.
> >   (qemu) device_add spapr-cpu-core,id=core2,nr_threads=8,cpu_model=host,slot=core[2]
> > - The ealier approach created threads from instance_init of the core object
> >   using a modified cpu_generic_init() routine. It used smp_threads and
> >   MachineState->cpu_model globals. In the current approach, nr_threads and
> >   cpu_model are obtained as properties and threads are created from
> >   property setters. The thread objects are allocated as part of core
> >   device and object_initialize() is used to initialize them.
> > 
> > Some open questions remain:
> > 
> > - Does this device_add semantics look ok ?
> >   (qemu) device_add spapr-cpu-core,id=core2,nr_threads=8,cpu_model=host,slot=core[2]
> looks fine to me, only I'd do following changes:
> s/nr_threads/threads/
> s/slot/core/ and make it numeric property
> 
> > - Is there need for machine to core object links ? If not, what would
> >   determine the slot/location of the core device ?
> I'd drop links altogether for this series as they were a part of an attempt
> to implement QOM based introspection interface, which is not must have
> provided QMP interface would provide all necessary for hotplug information
> which links aren't capable of to do.

Yeah, I tend to agree.

> Back in the days Anthony suggested that we 'might' use links to implement
> hotplug because there wasn't any other infrastructure hotplug
> infrastructure for bus-less devices. But now we have HOTPLUG_HANDLER
> interface that obsoleted, what links were supposed to handle at
> link set time hooks.
> 
> But somehow we stuck on links idea, though their initial role in hotplug
> were obsoleted and we still trying to fit them in design where
> they are not really necessary.
> 
> > - How to fit this with CPUSlotProperties HMP interface that Igor is
> >   working on ?
> I'll try to re-factor QMP interface RFC on top of this series
> so it would be easier for you to review it wrt spapr.
> 
> 
> > This version has the following changes:
> > 
> > - Dropped QMP and HMP enumeration patches for now since it isn't clear
> >   if the approach I took would be preferrable by all archs. Will wait
> >   and see how Igor's patches evolve here.
> > - Added the following pre-req patches to support CPU removal:
> >   exec: Remove cpu from cpus list during cpu_exec_exit()
> >   exec: Do vmstate unregistration from cpu_exec_exit()
> >   cpu: Reclaim vCPU objects
> >   cpu: Add a sync version of cpu_remove()
> > - Added sPAPR CPU hot removal support
> >   xics,xics_kvm: Handle CPU unplug correctly
> >   spapr: CPU hot unplug support
> > - Moved up the "slot" property into abstract cpu-core device.
> > - Recover when thread creation fails (spapr_cpu_core_create_threads())
> > - cpu_model property in spapr-cpu-core deivce is now tracked using
> >   ObjectClass pointer instead of string.
> > - Fail topologies with incomplete cores from within sPAPR's machine init.
> > - Fixes in spapr-cpu-core object creation from machine init to create
> >   only boottime CPUs.
> > - Moved board specific wiring code for CPU threads (spapr_cpu_init)
> >   into core's realize method to be called after each thread's realization.
> > - No specific action under TYPE_CPU in ->plug() handler either for hotplug
> >   or hot removal.
> > - Moved all core related CPU hotplug routines into spapr_cpu_core.c where
> >   it truly belongs.
> > - Setting of NUMA node for hotplugged CPUs moved to spapr_cpu_init()).
> > - Compared to previous implementation of hot removal that was part of
> >   different series earlier, this implementation moves all core removal
> >   logic into spapr_cpu_core.c.
> > - Some minor cleanups like use of g_new0 instead of g_malloc0 etc.
> > 
> > This patchset is present at:
> > https://github.com/bharata/qemu/commits/spapr-cpu-core
> > 
> > Bharata B Rao (9):
> >   exec: Remove cpu from cpus list during cpu_exec_exit()
> >   exec: Do vmstate unregistration from cpu_exec_exit()
> >   cpu: Add a sync version of cpu_remove()
> >   cpu: Abstract CPU core type
> >   spapr: CPU core device
> >   spapr: Represent boot CPUs as spapr-cpu-core devices
> >   spapr: CPU hotplug support
> >   xics,xics_kvm: Handle CPU unplug correctly
> >   spapr: CPU hot unplug support
> > 
> > Gu Zheng (1):
> >   cpu: Reclaim vCPU objects
> > 
> >  cpus.c                          |  50 ++++++
> >  exec.c                          |  44 ++++-
> >  hw/cpu/Makefile.objs            |   1 +
> >  hw/cpu/core.c                   |  44 +++++
> >  hw/intc/xics.c                  |  14 ++
> >  hw/intc/xics_kvm.c              |   8 +-
> >  hw/ppc/Makefile.objs            |   1 +
> >  hw/ppc/spapr.c                  | 174 +++++++++++++++++--
> >  hw/ppc/spapr_cpu_core.c         | 361 ++++++++++++++++++++++++++++++++++++++++
> >  hw/ppc/spapr_events.c           |   3 +
> >  hw/ppc/spapr_rtas.c             |  24 +++
> >  include/hw/cpu/core.h           |  30 ++++
> >  include/hw/ppc/spapr.h          |   9 +
> >  include/hw/ppc/spapr_cpu_core.h |  42 +++++
> >  include/hw/ppc/xics.h           |   1 +
> >  include/qom/cpu.h               |  18 ++
> >  include/sysemu/kvm.h            |   1 +
> >  kvm-all.c                       |  57 ++++++-
> >  kvm-stub.c                      |   5 +
> >  19 files changed, 862 insertions(+), 25 deletions(-)
> >  create mode 100644 hw/cpu/core.c
> >  create mode 100644 hw/ppc/spapr_cpu_core.c
> >  create mode 100644 include/hw/cpu/core.h
> >  create mode 100644 include/hw/ppc/spapr_cpu_core.h
> > 
> 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]