From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60241) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YqAg5-0005pe-Vg for qemu-devel@nongnu.org; Wed, 06 May 2015 21:32:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YqAg3-0005zl-PG for qemu-devel@nongnu.org; Wed, 06 May 2015 21:32:41 -0400 Date: Thu, 7 May 2015 11:03:47 +1000 From: David Gibson Message-ID: <20150507010347.GE1407@voom.redhat.com> References: <1429858066-12088-1-git-send-email-bharata@linux.vnet.ibm.com> <1429858066-12088-13-git-send-email-bharata@linux.vnet.ibm.com> <20150505065951.GJ14090@voom.redhat.com> <20150506061420.GO18380@in.ibm.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+JUInw4efm7IfTNU" Content-Disposition: inline In-Reply-To: <20150506061420.GO18380@in.ibm.com> Subject: Re: [Qemu-devel] [RFC PATCH v3 12/24] spapr: CPU hotplug support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Bharata B Rao Cc: mdroth@linux.vnet.ibm.com, aik@ozlabs.ru, agraf@suse.de, qemu-devel@nongnu.org, qemu-ppc@nongnu.org, tyreld@linux.vnet.ibm.com, nfont@linux.vnet.ibm.com, imammedo@redhat.com, afaerber@suse.de --+JUInw4efm7IfTNU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 06, 2015 at 11:44:20AM +0530, Bharata B Rao wrote: > On Tue, May 05, 2015 at 04:59:51PM +1000, David Gibson wrote: > > On Fri, Apr 24, 2015 at 12:17:34PM +0530, Bharata B Rao wrote: > > > Support CPU hotplug via device-add command. Set up device tree > > > entries for the hotplugged CPU core and use the exising EPOW event > > > infrastructure to send CPU hotplug notification to the guest. > > >=20 > > > Also support cold plugged CPUs that are specified by -device option > > > on cmdline. > > >=20 > > > Signed-off-by: Bharata B Rao > > > --- > > > hw/ppc/spapr.c | 129 ++++++++++++++++++++++++++++++++++++++++= ++++++++++ > > > hw/ppc/spapr_events.c | 8 ++-- > > > hw/ppc/spapr_rtas.c | 11 +++++ > > > 3 files changed, 145 insertions(+), 3 deletions(-) > > >=20 > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > > > index b526b7d..9b0701c 100644 > > > --- a/hw/ppc/spapr.c > > > +++ b/hw/ppc/spapr.c > > > @@ -33,6 +33,7 @@ > > > #include "sysemu/block-backend.h" > > > #include "sysemu/cpus.h" > > > #include "sysemu/kvm.h" > > > +#include "sysemu/device_tree.h" > > > #include "kvm_ppc.h" > > > #include "mmu-hash64.h" > > > #include "qom/cpu.h" > > > @@ -662,6 +663,17 @@ static void spapr_populate_cpu_dt(CPUState *cs, = void *fdt, int offset) > > > unsigned sockets =3D opts ? qemu_opt_get_number(opts, "sockets",= 0) : 0; > > > uint32_t cpus_per_socket =3D sockets ? (smp_cpus / sockets) : 1; > > > uint32_t pft_size_prop[] =3D {0, cpu_to_be32(spapr->htab_shift)}; > > > + sPAPRDRConnector *drc; > > > + sPAPRDRConnectorClass *drck; > > > + int drc_index; > > > + > > > + if (spapr->dr_cpu_enabled) { > > > + drc =3D spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU= , index); > > > + g_assert(drc); > > > + drck =3D SPAPR_DR_CONNECTOR_GET_CLASS(drc); > > > + drc_index =3D drck->get_index(drc); > > > + _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_= index))); > > > + } > > > =20 > > > _FDT((fdt_setprop_cell(fdt, offset, "reg", index))); > > > _FDT((fdt_setprop_string(fdt, offset, "device_type", "cpu"))); > > > @@ -1850,6 +1862,114 @@ static void spapr_nmi(NMIState *n, int cpu_in= dex, Error **errp) > > > } > > > } > > > =20 > > > +static void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUStat= e *cs, > > > + int *fdt_offset) > > > +{ > > > + PowerPCCPU *cpu =3D POWERPC_CPU(cs); > > > + DeviceClass *dc =3D DEVICE_GET_CLASS(cs); > > > + int id =3D ppc_get_vcpu_dt_id(cpu); > > > + void *fdt; > > > + int offset, fdt_size; > > > + char *nodename; > > > + > > > + fdt =3D create_device_tree(&fdt_size); > > > + nodename =3D g_strdup_printf("%s@%x", dc->fw_name, id); > > > + offset =3D fdt_add_subnode(fdt, 0, nodename); > > > + > > > + spapr_populate_cpu_dt(cs, fdt, offset); > > > + g_free(nodename); > > > + > > > + *fdt_offset =3D offset; > > > + return fdt; > > > +} > > > + > > > +static void spapr_cpu_plug(HotplugHandler *hotplug_dev, DeviceState = *dev, > > > + Error **errp) > > > +{ > > > + CPUState *cs =3D CPU(dev); > > > + PowerPCCPU *cpu =3D POWERPC_CPU(cs); > > > + int id =3D ppc_get_vcpu_dt_id(cpu); > > > + sPAPRDRConnector *drc =3D > > > + spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id); > > > + sPAPRDRConnectorClass *drck; > > > + int smt =3D kvmppc_smt_threads(); > > > + Error *local_err =3D NULL; > > > + void *fdt =3D NULL; > > > + int i, fdt_offset =3D 0; > > > + > > > + /* Set NUMA node for the added CPUs */ > > > + for (i =3D 0; i < nb_numa_nodes; i++) { > > > + if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) { > > > + cs->numa_node =3D i; > > > + break; > > > + } > > > + } > > > + > > > + /* > > > + * SMT threads return from here, only main thread (core) will > > > + * continue and signal hotplug event to the guest. > > > + */ > > > + if ((id % smt) !=3D 0) { > > > + return; > > > + } > >=20 > > Couldn't you avoid this by attaching this call to the core device, > > rather than the individual vcpu thread objects? >=20 > Adding a socket device will result in realize call for that device. > Socket realizefn will call core realizefn and core realizefn will call > thread (or CPU) realizefn. device_set_realized() will call ->plug handler > for all these devices (socket, cores and threads) and that's how we > end up here even for threads. >=20 > This will be same when I get rid of socket abstraction and hot plug > cores for the same reason as above. >=20 > And calling ->plug handler in the context of threads is required > to initialize board specific CPU bits for the CPU thread that is being > realized. Right, but can't you do the thread specific initialization from the thread hotplug path, and the core specific initialization (basically everything below this if) from the core hotplug path? >=20 > >=20 > >=20 > > > + if (!spapr->dr_cpu_enabled) { > > > + /* > > > + * This is a cold plugged CPU but the machine doesn't support > > > + * DR. So skip the hotplug path ensuring that the CPU is bro= ught > > > + * up online with out an associated DR connector. > > > + */ > > > + return; > > > + } > > > + > > > + g_assert(drc); > > > + > > > + /* > > > + * Setup CPU DT entries only for hotplugged CPUs. For boot time = or > > > + * coldplugged CPUs DT entries are setup in spapr_finalize_fdt(). > > > + */ > > > + if (dev->hotplugged) { > > > + fdt =3D spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset); > > > + } > > > + > > > + drck =3D SPAPR_DR_CONNECTOR_GET_CLASS(drc); > > > + drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local= _err); > > > + if (local_err) { > > > + g_free(fdt); > > > + error_propagate(errp, local_err); > > > + return; > > > + } > > > + > > > + /* > > > + * We send hotplug notification interrupt to the guest only in c= ase > > > + * of hotplugged CPUs. > > > + */ > > > + if (dev->hotplugged) { > > > + spapr_hotplug_req_add_event(drc); > > > + } else { > > > + /* > > > + * HACK to support removal of hotplugged CPU after VM migrat= ion: > > > + * > > > + * Since we want to be able to hot-remove those coldplugged = CPUs > > > + * started at boot time using -device option at the target V= M, we set > > > + * the right allocation_state and isolation_state for them, = which for > > > + * the hotplugged CPUs would be set via RTAS calls done from= the > > > + * guest during hotplug. > > > + * > > > + * This allows the coldplugged CPUs started using -device op= tion to > > > + * have the right isolation and allocation states as expecte= d by the > > > + * CPU hot removal code. > > > + * > > > + * This hack will be removed once we have DRC states migrate= d as part > > > + * of VM migration. > > > + */ > > > + drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_US= ABLE); > > > + drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNIS= OLATED); > > > + } > > > + > > > + return; > > > +} > > > + > > > static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, > > > DeviceState *dev, Error **errp) > > > { > > > @@ -1858,6 +1978,15 @@ static void spapr_machine_device_plug(HotplugH= andler *hotplug_dev, > > > PowerPCCPU *cpu =3D POWERPC_CPU(cs); > > > =20 > > > spapr_cpu_init(cpu); > > > + spapr_cpu_reset(cpu); > >=20 > > I'm a little surprised these get called here, rather than in the > > creation / realize path of the core qdev. >=20 > These two routines (spapr_cpu_init and spapr_cpu_reset) are sPAPR or > board specific and I can't even have them as part of ppc_cpu_realizefn. Oh, yes of course. Sorry. > We had discussed this earlier at > http://lists.gnu.org/archive/html/qemu-devel/2015-02/msg04399.html >=20 > Regards, > Bharata. >=20 --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --+JUInw4efm7IfTNU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVSrnzAAoJEGw4ysog2bOSQz4QALPTlwppahDizDB26UJAWtNd BhW/pv0jGJEwA7cemOY92e1yCCqX/kQd+YZK2h173J8vqpiHFME/R2Z5oY9RmJz4 CIOa0xN2bD6+nfYhsEZUC3lohFUhAkQXlXLSHHILtxoEbiQsGnDQNFpuf+tM6D4g pxTQqsbx5hUbecEeTCLJs0VtF9ZgJQ3vDokwYWAudj/scolwd4uj7mwlE3Qr+fV3 tbwMF+6wuj2zHeU93OwlPGLhq3zcVTajT27Le/i6rRxy1ZLk3geqNRcYUT9Qpvef SRwSFcJJoqbawt+OrGHUsKUw/DaGXG8zhyUMcaGEGD1nxLdh5rQWzGZ0/gk620wo 3sPF074ELXKwNhUNXC/LHM217Xqze6nZR9Tz1sVq7IT+d/tBkTD7smCuMykmSGWH ZAumT5JlC5Pngqve37spW0b9w5uTg/4pY1Uu2gqmWW+BHmIXC35bXRddZIhYjgJg /o8hDZ9ODH3sTfoyWYpRqqZmTH+hsEiaqGxT7prMP+vnugJ9VXBJrO0rsrsjVm6H LY7IVy3hhn0+2EopUNB0Lzu8/K9Chmx9jny7ljKwgXAyFJ3Axpe74xKNepDiYCFu DIW0pSRRp2Ummf345U2oL5inwX/FaE2bh3geBkYnVmmPMlpT24hB1HuKZyTuo7CF oEcCWzjLgTn/p/Liei7y =dt41 -----END PGP SIGNATURE----- --+JUInw4efm7IfTNU--