From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33378) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZwKLK-000212-Bb for qemu-devel@nongnu.org; Tue, 10 Nov 2015 20:37:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZwKLF-0006iH-TC for qemu-devel@nongnu.org; Tue, 10 Nov 2015 20:36:58 -0500 Date: Wed, 11 Nov 2015 12:36:30 +1100 From: David Gibson Message-ID: <20151111013630.GC5852@voom.redhat.com> References: <1445853185-22518-1-git-send-email-bharata@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0lnxQi9hkpPO77W3" Content-Disposition: inline In-Reply-To: <1445853185-22518-1-git-send-email-bharata@linux.vnet.ibm.com> Subject: Re: [Qemu-devel] [Qemu-ppc] [RFC PATCH v1] spapr: Memory hot-unplug support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Bharata B Rao Cc: qemu-devel@nongnu.org, mdroth@linux.vnet.ibm.com, qemu-ppc@nongnu.org, tyreld@linux.vnet.ibm.com, imammedo@redhat.com, nfont@linux.vnet.ibm.com --0lnxQi9hkpPO77W3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 26, 2015 at 03:23:05PM +0530, Bharata B Rao wrote: > Add support to hot remove pc-dimm memory devices. Sorry it's taken me so long to look at this. > TODO: In response to memory hot removal operation on a DIMM device, > guest kernel might refuse to offline a few LMBs that are part of that dev= ice. > In such cases, we will have a DIMM device that has some LMBs online and s= ome > LMBs offline. To avoid this situation, drmgr could be enhanced to support > a command line option that results in removal of all the LMBs or none. Hm.. what would be the end result of such a situation? We want to handle it as gracefully as we can, even if the guest has old tools. Is there some way we can detect this failure condition, and re-connect the DIMM? It does highlight the fact that the PAPR hotplug interface and the pc-dimm model don't work together terribly well. I think we have to try to support it for the sake of management layers, but I do wonder if we ought to thinkg about an alternative "lmb-pool" backend, where the precise location of memory blocks isn't so important. With some thought such a backend might also be useful for paravirt x86. Which also makes me think, I wonder if it would be possible to wire up a PAPR compatible interface to qemu's balloon backend, since in some ways the PAPR memory hotplug model acts more like a balloon (in that the guest physical address of removed LMBs isn't usually important to the host). Still, we need to get the dimm backed model working first, I guess. Apart from those overall considerations, the patch looks good. > Signed-off-by: Bharata B Rao > --- > Changes in v1: > - Got rid of the patch that introduced a field in PCDIMMDevice to track > DIMM marked for removal since we can track that using within DRC > object. > - Removed the patch that added return value to rtas_set_indicator() > since the required changes are already pushed by Michael Roth. >=20 > v0: >=20 > hw/ppc/spapr.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++= +++++- > hw/ppc/spapr_drc.c | 18 +++++++++++ > 2 files changed, 107 insertions(+), 1 deletion(-) >=20 > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > index e1202ce..f5b1ac2 100644 > --- a/hw/ppc/spapr.c > +++ b/hw/ppc/spapr.c > @@ -2174,6 +2174,85 @@ out: > error_propagate(errp, local_err); > } > =20 > +typedef struct sPAPRDIMMState { > + uint32_t nr_lmbs; > +} sPAPRDIMMState; > + > +static void spapr_lmb_release(DeviceState *dev, void *opaque) > +{ > + sPAPRDIMMState *ds =3D (sPAPRDIMMState *)opaque; > + HotplugHandler *hotplug_ctrl =3D NULL; > + Error *local_err =3D NULL; > + > + if (--ds->nr_lmbs) { > + return; > + } > + > + g_free(ds); > + > + /* > + * Now that all the LMBs have been removed by the guest, call the > + * pc-dimm unplug handler to cleanup up the pc-dimm device. > + */ > + hotplug_ctrl =3D qdev_get_hotplug_handler(dev); > + hotplug_handler_unplug(hotplug_ctrl, dev, &local_err); > +} > + > +static void spapr_del_lmbs(DeviceState *dev, uint64_t addr, uint64_t siz= e, > + Error **errp) > +{ > + sPAPRDRConnector *drc; > + sPAPRDRConnectorClass *drck; > + uint32_t nr_lmbs =3D size/SPAPR_MEMORY_BLOCK_SIZE; > + Error *local_err =3D NULL; > + int i; > + sPAPRDIMMState *ds =3D g_malloc0(sizeof(sPAPRDIMMState)); > + > + ds->nr_lmbs =3D nr_lmbs; > + for (i =3D 0; i < nr_lmbs; i++) { > + drc =3D spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_LMB, > + addr/SPAPR_MEMORY_BLOCK_SIZE); > + g_assert(drc); > + > + drck =3D SPAPR_DR_CONNECTOR_GET_CLASS(drc); > + drck->detach(drc, dev, spapr_lmb_release, ds, &local_err); > + addr +=3D SPAPR_MEMORY_BLOCK_SIZE; > + } > + spapr_hotplug_req_remove_by_count(SPAPR_DR_CONNECTOR_TYPE_LMB, nr_lm= bs); > +} > + > +static void spapr_memory_unplug(HotplugHandler *hotplug_dev, DeviceState= *dev, > + Error **errp) > +{ > + sPAPRMachineState *ms =3D SPAPR_MACHINE(hotplug_dev); > + PCDIMMDevice *dimm =3D PC_DIMM(dev); > + PCDIMMDeviceClass *ddc =3D PC_DIMM_GET_CLASS(dimm); > + MemoryRegion *mr =3D ddc->get_memory_region(dimm); > + > + pc_dimm_memory_unplug(dev, &ms->hotplug_memory, mr); > + object_unparent(OBJECT(dev)); > +} > + > +static void spapr_memory_unplug_request(HotplugHandler *hotplug_dev, > + DeviceState *dev, Error **errp) > +{ > + Error *local_err =3D NULL; > + PCDIMMDevice *dimm =3D PC_DIMM(dev); > + PCDIMMDeviceClass *ddc =3D PC_DIMM_GET_CLASS(dimm); > + MemoryRegion *mr =3D ddc->get_memory_region(dimm); > + uint64_t size =3D memory_region_size(mr); > + uint64_t addr; > + > + addr =3D object_property_get_int(OBJECT(dimm), PC_DIMM_ADDR_PROP, &l= ocal_err); > + if (local_err) { > + goto out; > + } > + > + spapr_del_lmbs(dev, addr, size, &local_err); > +out: > + error_propagate(errp, local_err); > +} > + > static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, > DeviceState *dev, Error **errp) > { > @@ -2221,7 +2300,15 @@ static void spapr_machine_device_unplug(HotplugHan= dler *hotplug_dev, > DeviceState *dev, Error **errp) > { > if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) { > - error_setg(errp, "Memory hot unplug not supported by sPAPR"); > + spapr_memory_unplug(hotplug_dev, dev, errp); > + } > +} > + > +static void spapr_machine_device_unplug_request(HotplugHandler *hotplug_= dev, > + DeviceState *dev, Error = **errp) > +{ > + if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) { > + spapr_memory_unplug_request(hotplug_dev, dev, errp); > } > } > =20 > @@ -2263,6 +2350,7 @@ static void spapr_machine_class_init(ObjectClass *o= c, void *data) > hc->plug =3D spapr_machine_device_plug; > hc->unplug =3D spapr_machine_device_unplug; > mc->cpu_index_to_socket_id =3D spapr_cpu_index_to_socket_id; > + hc->unplug_request =3D spapr_machine_device_unplug_request; > =20 > smc->dr_lmb_enabled =3D false; > fwc->get_dev_path =3D spapr_get_fw_dev_path; > diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c > index 5d6ea7c..59b6ea9 100644 > --- a/hw/ppc/spapr_drc.c > +++ b/hw/ppc/spapr_drc.c > @@ -11,6 +11,7 @@ > */ > =20 > #include "hw/ppc/spapr_drc.h" > +#include "hw/ppc/spapr.h" > #include "qom/object.h" > #include "hw/qdev.h" > #include "qapi/visitor.h" > @@ -77,6 +78,23 @@ static uint32_t set_isolation_state(sPAPRDRConnector *= drc, > } > } > =20 > + /* > + * Fail any request to ISOLATE the LMB DRC if this LMB doesn't > + * belong to a DIMM device that is marked for removal. > + * > + * Currently the guest userspace tool drmgr that drives the memory > + * hotplug/unplug will just try to remove a set of 'removable' LMBs > + * in response to a hot unplug request that is based on drc-count. > + * If the LMB being removed doesn't belong to a DIMM device that is > + * actually being unplugged, fail the isolation request here. > + */ > + if (drc->type =3D=3D SPAPR_DR_CONNECTOR_TYPE_LMB) { > + if ((state =3D=3D SPAPR_DR_ISOLATION_STATE_ISOLATED) && > + !drc->awaiting_release) { > + return RTAS_OUT_HW_ERROR; > + } > + } > + > drc->isolation_state =3D state; > =20 > if (drc->isolation_state =3D=3D SPAPR_DR_ISOLATION_STATE_ISOLATED) { --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --0lnxQi9hkpPO77W3 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJWQpueAAoJEGw4ysog2bOSJCoQAIuMONzzaoCcgOiw2v5YIspI 4HpRXL3yz2sjmX+IXjGY8ObqXm68JUjezkSUpVTokRZvAASU1jFfW3FMTZ+f5yu3 Uv2Fqd/QK8FUNc/GlHcEDqJX0TvOtZdkhVA2i+BdjJSW9tC6B0CfbxE2Pqk4lyzb Qo6iCBHx5IY0WZOqj0VAd9a1wRT9SiKdQfpOUv9kYMu2FMPy4b6dUF9GnEAlScIe KLr76WU8rRBLHDK00z5burHXeSb/kxQ+RW8aBQ/ekEpbMEmX73M+e6CA80Cco4Yj vH8j33L7o4kpTAlH+GBXk/SoQ2/LoP2yQSgU8VZCgGg2M7M5GORrU6ZqZIzCaB0X ac7XBWZgdtqLgMJM2sKkMb3YalHmhQOLeZ+CMyHqN5C4nXuk8u32A3DMd8VFfXu2 mdzLn9or542N/KgUO9PeiiFpnnQofgv5uFQ+oew5pysrj9XIHeQv1rw7zjqJ6qBx tLD2bia1Bwsy6RJ1TdP0K5Be7qgD40XLAfCpAWWzFbG4X1cqsyp23w7wuT6QmuOj lcb6pgA+deSLDOGu1z7Jct5BjfVwcISx3ct8DwNeqikzcRdzslrZD9pkD2eXt7AV 9dX1sJx9jOem9OoWKWryKuVr7MrlrO7pAxtZHZ5QfrMMyRbv6gdIJqrwUp61urvp w/suBZNqTfezyOA3jiAj =KfwV -----END PGP SIGNATURE----- --0lnxQi9hkpPO77W3--