From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56257) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ad9GI-0006tR-VY for qemu-devel@nongnu.org; Mon, 07 Mar 2016 23:28:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ad9GF-0007oz-Lb for qemu-devel@nongnu.org; Mon, 07 Mar 2016 23:28:46 -0500 Date: Tue, 8 Mar 2016 15:27:39 +1100 From: David Gibson Message-ID: <20160308042739.GW22546@voom.fritz.box> References: <1457074461-14285-1-git-send-email-bharata@linux.vnet.ibm.com> <1457074461-14285-9-git-send-email-bharata@linux.vnet.ibm.com> <20160307034906.GF22546@voom.fritz.box> <20160307062942.GF5054@in.ibm.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UTjKcilERHWBCdCp" Content-Disposition: inline In-Reply-To: <20160307062942.GF5054@in.ibm.com> Subject: Re: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Bharata B Rao Cc: mjrosato@linux.vnet.ibm.com, agraf@suse.de, thuth@redhat.com, pkrempa@redhat.com, ehabkost@redhat.com, aik@ozlabs.ru, qemu-devel@nongnu.org, armbru@redhat.com, borntraeger@de.ibm.com, qemu-ppc@nongnu.org, pbonzini@redhat.com, imammedo@redhat.com, afaerber@suse.de, mdroth@linux.vnet.ibm.com --UTjKcilERHWBCdCp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 07, 2016 at 11:59:42AM +0530, Bharata B Rao wrote: > On Mon, Mar 07, 2016 at 02:49:06PM +1100, David Gibson wrote: > > On Fri, Mar 04, 2016 at 12:24:19PM +0530, Bharata B Rao wrote: > > > Set up device tree entries for the hotplugged CPU core and use the > > > exising EPOW event infrastructure to send CPU hotplug notification to > > > the guest. > > >=20 > > > Signed-off-by: Bharata B Rao > > > --- > > > hw/ppc/spapr.c | 73 +++++++++++++++++++++++++++++++= +++++++++- > > > hw/ppc/spapr_cpu_core.c | 60 +++++++++++++++++++++++++++++++= ++ > > > hw/ppc/spapr_events.c | 3 ++ > > > hw/ppc/spapr_rtas.c | 24 ++++++++++++++ > > > include/hw/ppc/spapr.h | 4 +++ > > > include/hw/ppc/spapr_cpu_core.h | 2 ++ > > > 6 files changed, 165 insertions(+), 1 deletion(-) > > >=20 > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > > > index 5acb612..6c4ac50 100644 > > > --- a/hw/ppc/spapr.c > > > +++ b/hw/ppc/spapr.c > > > @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, = void *fdt, int offset, > > > size_t page_sizes_prop_size; > > > uint32_t vcpus_per_socket =3D smp_threads * smp_cores; > > > uint32_t pft_size_prop[] =3D {0, cpu_to_be32(spapr->htab_shift)}; > > > + sPAPRMachineClass *smc =3D SPAPR_MACHINE_GET_CLASS(qdev_get_mach= ine()); > > > + sPAPRDRConnector *drc; > > > + sPAPRDRConnectorClass *drck; > > > + int drc_index; > > > + > > > + if (smc->dr_cpu_enabled) { > > > + drc =3D spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU= , index); > > > + g_assert(drc); > > > + drck =3D SPAPR_DR_CONNECTOR_GET_CLASS(drc); > > > + drc_index =3D drck->get_index(drc); > > > + _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_= index))); > > > + } > > > =20 > > > /* Note: we keep CI large pages off for now because a 64K capabl= e guest > > > * provisioned with large pages might otherwise try to map a qemu > > > @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState= *spapr, > > > _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_= TYPE_LMB)); > > > } > > > =20 > > > + if (smc->dr_cpu_enabled) { > > > + int offset =3D fdt_path_offset(fdt, "/cpus"); > > > + ret =3D spapr_drc_populate_dt(fdt, offset, NULL, > > > + SPAPR_DR_CONNECTOR_TYPE_CPU); > > > + if (ret < 0) { > > > + error_report("Couldn't set up CPU DR device tree propert= ies"); > > > + exit(1); > > > + } > > > + } > > > + > > > _FDT((fdt_pack(fdt))); > > > =20 > > > if (fdt_totalsize(fdt) > FDT_MAX_SIZE) { > > > @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void) > > > =20 > > > } > > > =20 > > > -static void spapr_cpu_reset(void *opaque) > > > +void spapr_cpu_reset(void *opaque) > > > { > > > sPAPRMachineState *spapr =3D SPAPR_MACHINE(qdev_get_machine()); > > > PowerPCCPU *cpu =3D opaque; > > > @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const = char *boot_device, > > > void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error= **errp) > > > { > > > CPUPPCState *env =3D &cpu->env; > > > + CPUState *cs =3D CPU(cpu); > > > + int i; > > > =20 > > > /* Set time-base frequency to 512 MHz */ > > > cpu_ppc_tb_init(env, TIMEBASE_FREQ); > > > @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, = PowerPCCPU *cpu, Error **errp) > > > } > > > } > > > =20 > > > + /* Set NUMA node for the added CPUs */ > > > + for (i =3D 0; i < nb_numa_nodes; i++) { > > > + if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) { > > > + cs->numa_node =3D i; > > > + break; > > > + } > > > + } > > > + > >=20 > > This hunk seems like it belongs in a different patch. >=20 > It appears that this would be needed by other archs also to set the > NUMA node for the hot-plugged CPU. How about make an API out of this > and use this something like below ? Igor ? Is there a way we could put this in the the CPU thread initialization itself? Rather than requiring every platform to call a helper. > ------------------------------------------------------------------- > diff --git a/hw/i386/pc.c b/hw/i386/pc.c > index 0aeefd2..8347234 100644 > --- a/hw/i386/pc.c > +++ b/hw/i386/pc.c > @@ -1112,6 +1112,7 @@ void pc_hot_add_cpu(const int64_t id, Error **errp) > error_propagate(errp, local_err); > return; > } > + numa_set_cpu(CPU(cpu)); > object_unref(OBJECT(cpu)); > } > =20 > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > index a42f8c0..f2b3b67 100644 > --- a/hw/ppc/spapr.c > +++ b/hw/ppc/spapr.c > @@ -1645,7 +1645,6 @@ void spapr_cpu_init(sPAPRMachineState *spapr, Power= PCCPU *cpu, Error **errp) > { > CPUPPCState *env =3D &cpu->env; > CPUState *cs =3D CPU(cpu); > - int i; > =20 > /* Set time-base frequency to 512 MHz */ > cpu_ppc_tb_init(env, TIMEBASE_FREQ); > @@ -1671,12 +1670,7 @@ void spapr_cpu_init(sPAPRMachineState *spapr, Powe= rPCCPU *cpu, Error **errp) > } > =20 > /* Set NUMA node for the added CPUs */ > - for (i =3D 0; i < nb_numa_nodes; i++) { > - if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) { > - cs->numa_node =3D i; > - break; > - } > - } > + numa_set_cpu(cs); > =20 > xics_cpu_setup(spapr->icp, cpu); > qemu_register_reset(spapr_cpu_reset, cpu); > diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h > index bb184c9..648d68b 100644 > --- a/include/sysemu/numa.h > +++ b/include/sysemu/numa.h > @@ -31,5 +31,6 @@ extern QemuOptsList qemu_numa_opts; > void numa_set_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t node); > void numa_unset_mem_node_id(ram_addr_t addr, uint64_t size, uint32_t nod= e); > uint32_t numa_get_node(ram_addr_t addr, Error **errp); > +void numa_set_cpu(CPUState *cpu); > =20 > #endif > diff --git a/numa.c b/numa.c > index 4c4f7f5..1b47c15 100644 > --- a/numa.c > +++ b/numa.c > @@ -396,20 +396,32 @@ void parse_numa_opts(MachineClass *mc) > } > } > =20 > +static void numa_set_cpu_numa_node(CPUState *cpu) > +{ > + int i; > + > + for (i =3D 0; i < nb_numa_nodes; i++) { > + if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) { > + cpu->numa_node =3D i; > + break; > + } > + } > +} > + > void numa_post_machine_init(void) > { > CPUState *cpu; > - int i; > =20 > CPU_FOREACH(cpu) { > - for (i =3D 0; i < nb_numa_nodes; i++) { > - if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) { > - cpu->numa_node =3D i; > - } > - } > + numa_set_cpu_numa_node(cpu); > } > } > =20 > +void numa_set_cpu(CPUState *cpu) > +{ > + numa_set_cpu_numa_node(cpu); > +} > + > static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *own= er, > const char *name, > uint64_t ram_size) >=20 > ------------------------------------------------------------------- > >=20 > > > xics_cpu_setup(spapr->icp, cpu); > > > qemu_register_reset(spapr_cpu_reset, cpu); > > > } > > > @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machin= e) > > > char *filename; > > > int spapr_cores =3D smp_cpus / smp_threads; > > > int spapr_max_cores =3D max_cpus / smp_threads; > > > + int smt =3D kvmppc_smt_threads(); > > > =20 > > > if (smp_cpus % smp_threads) { > > > error_report("smp_cpus (%u) must be multiple of threads (%u)= ", > > > @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machi= ne) > > > spapr_validate_node_memory(machine, &error_fatal); > > > } > > > =20 > > > + if (smc->dr_cpu_enabled) { > > > + for (i =3D 0; i < spapr_max_cores; i++) { > > > + sPAPRDRConnector *drc =3D > > > + spapr_dr_connector_new(OBJECT(spapr), > > > + SPAPR_DR_CONNECTOR_TYPE_CPU, = i * smt); > > > + qemu_register_reset(spapr_drc_reset, drc); > > > + } > > > + } > > > + > >=20 > > Nit: would this be cleaner to include in the same loop that constructs > > the (empty) links and boot-time cpu cores? >=20 > Seems possible, will change. >=20 > >=20 > > > /* init CPUs */ > > > if (machine->cpu_model =3D=3D NULL) { > > > machine->cpu_model =3D kvm_enabled() ? "host" : "POWER7"; > > > @@ -2267,6 +2309,27 @@ out: > > > error_propagate(errp, local_err); > > > } > > > =20 > > > +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs, > > > + int *fdt_offset, sPAPRMachineSta= te *spapr) > > > +{ > > > + PowerPCCPU *cpu =3D POWERPC_CPU(cs); > > > + DeviceClass *dc =3D DEVICE_GET_CLASS(cs); > > > + int id =3D ppc_get_vcpu_dt_id(cpu); > > > + void *fdt; > > > + int offset, fdt_size; > > > + char *nodename; > > > + > > > + fdt =3D create_device_tree(&fdt_size); > > > + nodename =3D g_strdup_printf("%s@%x", dc->fw_name, id); > > > + offset =3D fdt_add_subnode(fdt, 0, nodename); > > > + > > > + spapr_populate_cpu_dt(cs, fdt, offset, spapr); > > > + g_free(nodename); > > > + > > > + *fdt_offset =3D offset; > > > + return fdt; > > > +} > > > + > > > static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, > > > DeviceState *dev, Error **errp) > > > { > > > @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugH= andler *hotplug_dev, > > > } > > > =20 > > > spapr_memory_plug(hotplug_dev, dev, node, errp); > > > + } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)= ) { > > > + if (!smc->dr_cpu_enabled && dev->hotplugged) { > > > + error_setg(errp, "CPU hotplug not supported for this mac= hine"); > > > + return; > > > + } > > > + spapr_core_plug(hotplug_dev, dev, errp); > > > } > > > } > > > =20 > > > @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClas= s *oc, void *data) > > > mc->cpu_index_to_socket_id =3D spapr_cpu_index_to_socket_id; > > > =20 > > > smc->dr_lmb_enabled =3D true; > > > + smc->dr_cpu_enabled =3D true; > > > fwc->get_dev_path =3D spapr_get_fw_dev_path; > > > nc->nmi_monitor_handler =3D spapr_nmi; > > > } > > > @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(Mac= hineClass *mc) > > > =20 > > > spapr_machine_2_6_class_options(mc); > > > smc->use_ohci_by_default =3D true; > > > + smc->dr_cpu_enabled =3D false; > > > SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5); > > > } > > > =20 > > > diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c > > > index 9ddf3ce..4c233d7 100644 > > > --- a/hw/ppc/spapr_cpu_core.c > > > +++ b/hw/ppc/spapr_cpu_core.c > > > @@ -14,6 +14,65 @@ > > > #include "qapi/visitor.h" > > > #include > > > =20 > > > +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, > > > + Error **errp) > > > +{ > > > + sPAPRMachineClass *smc =3D SPAPR_MACHINE_GET_CLASS(qdev_get_mach= ine()); > > > + sPAPRMachineState *ms =3D SPAPR_MACHINE(qdev_get_machine()); > > > + sPAPRCPUCore *core =3D SPAPR_CPU_CORE(OBJECT(dev)); > > > + PowerPCCPU *cpu =3D &core->threads[0]; > > > + CPUState *cs =3D CPU(cpu); > > > + int id =3D ppc_get_vcpu_dt_id(cpu); > > > + sPAPRDRConnector *drc =3D > > > + spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id); > > > + sPAPRDRConnectorClass *drck; > > > + Error *local_err =3D NULL; > > > + void *fdt =3D NULL; > > > + int fdt_offset =3D 0; > > > + > > > + if (!smc->dr_cpu_enabled) { > > > + /* > > > + * This is a cold plugged CPU core but the machine doesn't s= upport > > > + * DR. So skip the hotplug path ensuring that the core is br= ought > > > + * up online with out an associated DR connector. > > > + */ > > > + return; > > > + } > > > + > > > + g_assert(drc); > > > + > > > + /* > > > + * Setup CPU DT entries only for hotplugged CPUs. For boot time = or > > > + * coldplugged CPUs DT entries are setup in spapr_finalize_fdt(). > > > + */ > > > + if (dev->hotplugged) { > > > + fdt =3D spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, = ms); > > > + dev->hotplugged =3D true; > > > + } > > > + > > > + drck =3D SPAPR_DR_CONNECTOR_GET_CLASS(drc); > > > + drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local= _err); > > > + if (local_err) { > > > + g_free(fdt); > > > + error_propagate(errp, local_err); > > > + return; > > > + } > > > + > > > + if (dev->hotplugged) { > > > + /* > > > + * Send hotplug notification interrupt to the guest only in = case > > > + * of hotplugged CPUs. > > > + */ > > > + spapr_hotplug_req_add_by_index(drc); > > > + } else { > > > + /* > > > + * Set the right DRC states for cold plugged CPU. > > > + */ > > > + drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_US= ABLE); > > > + drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNIS= OLATED); > > > + } > > > +} > > > + > > > static int spapr_cpu_core_realize_child(Object *child, void *opaque) > > > { > > > Error **errp =3D opaque; > > > @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *chi= ld, void *opaque) > > > if (*errp) { > > > return 1; > > > } > > > + spapr_cpu_reset(cpu); > >=20 > > This also looks like it belongs in a different patch. >=20 > You mean a separate patch for this or push this around to an existing > patch of the series ? >=20 > Regards, > Bharata. >=20 --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --UTjKcilERHWBCdCp Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJW3lS7AAoJEGw4ysog2bOSqIEP/20sWpGdFiFSwgYu+eVQzhWh VUBfIi5AIBHrHKy3izfcdMPLNzGer/HH1xLsF7BGdhMqhTV6rXOymCl/LLdPMeWE IAaXlYsFoxTxlldInsqaJXYun+dd1Tub8QEBnqdDP41/2KjkHuu30wGz0CPhcZE2 22igV4cFS7Z8X1U53e+NRtyuAFN6MO3bmJBMkhloDxI0j4wgRWRphwGSW1OKejc0 FgERlfLtmzmC7gl+z5dKx4T1+FMJrSfnhoh3gakEP4HyqhK/R+1JLhCBcXwdBfzW qB1aRKuEDnpPuWwga4yFYus7sslrkMwYFmo0ieNb1/UBdTdByUDR1/Q+OoFP6jVP Ma0V/llXCnfC9+LqxjePAPveDgfrWmW3QF2VOde2kWt6bbmu283irQj3cBkj+i4x O5R/eKzC4TDYNut8+z4dspGZazS3dgbmyGiHm55YnE9bNYzjZjQIOR7Q9/c+Dqz1 SF6y8ljGBEQFg7gBlSHhkzKG3viOOHuafBXzwWfBmohlKqfM+HEtcnHp0qKeoEAd X4hZtspyNvJfoqYdfgQ0zPaZNeZulowYmUxsCoB5qe0jsrzniJ2yUIaXW2PIZ2at 5DYTQ7xVQJE1IpqCtq2QZEMlqpFo+HvS7c3VrWplNXMWMpQ02FnzSkyfXv3J0GWv Fv1oHl1OF49sQSiqX3Ve =v+HK -----END PGP SIGNATURE----- --UTjKcilERHWBCdCp--