From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49761) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c7mGf-0001Hf-PC for qemu-devel@nongnu.org; Fri, 18 Nov 2016 11:44:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c7mGc-0006BG-Kv for qemu-devel@nongnu.org; Fri, 18 Nov 2016 11:44:01 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:33841 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1c7mGc-0006Ax-EV for qemu-devel@nongnu.org; Fri, 18 Nov 2016 11:43:58 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id uAIGhupV106418 for ; Fri, 18 Nov 2016 11:43:57 -0500 Received: from e36.co.us.ibm.com (e36.co.us.ibm.com [32.97.110.154]) by mx0a-001b2d01.pphosted.com with ESMTP id 26t2tv046x-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Fri, 18 Nov 2016 11:43:57 -0500 Received: from localhost by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 18 Nov 2016 09:43:56 -0700 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable From: Michael Roth In-Reply-To: <20161118054505.GD31640@umbus.fritz.box> References: <1479433227-29238-1-git-send-email-mdroth@linux.vnet.ibm.com> <20161118054505.GD31640@umbus.fritz.box> Date: Fri, 18 Nov 2016 10:39:49 -0600 Message-Id: <20161118163949.3756.44682@loki> Subject: Re: [Qemu-devel] [PATCH for-2.8 0/3] spapr: fix breakage of memory unplug after migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: David Gibson Cc: qemu-devel@nongnu.org, qemu-ppc@nongnu.org, duanj@linux.vnet.ibm.com, bharata@linux.vnet.ibm.com, dgilbert@redhat.com, quintela@redhat.com, amit.shah@redhat.com Quoting David Gibson (2016-11-17 23:45:05) > On Thu, Nov 17, 2016 at 07:40:24PM -0600, Michael Roth wrote: > > These patches are based on David's ppc-for-2.8 tree, and are also > > available from: > > = > > https://github.com/mdroth/qemu/commits/spapr-cas-migration > > = > > Currently, memory hotplugged to a pseries guest cannot be removed after > > the guest has been migrated. This is due to 2 issues: > > = > > 1) The coldplugged state of memory on the target side is one where the > > corresponding DRC's allocation state is: > > = > > allocation_state =3D=3D unallocated, > > awaiting_allocation =3D=3D true, > > = > > When the guest attempts to unplug memory on the target side, it first > > checks that allocation_state =3D=3D allocated. If we fix this, the g= uest > > can successfully notify QEMU of completion on it's end, but then the > > DRC code sees that awaiting_allocation =3D=3D true, so it defers the > > finalizing of the LMB and corresponding DIMM since it assumes that > > the DIMM must have been previously allocated before it can be remove= d. > > = > > To address this, we pull in patches 1-2 from Jian Jun's DRC migration > > series: > > = > > https://lists.gnu.org/archive/html/qemu-ppc/2016-10/msg00048.html > > = > > with some minor changes relating to prior review comments, and > > the addition of migrating the DRC's awaiting_allocation value, which > > wasn't part of the original patch. This doesn't address the full sco= pe > > of the issues Jian Jun was looking at (involving synchronizing state > > when migration occurs during fairly small race windows), just this > > particular case, which is more user visible since the time window is > > indefinite. > > = > > 2) The ability to unplug memory is gated on the QEMU side by a check as > > to whether or not support for newer-style hotplug events was negotia= ted > > via CAS during boot. The check is performed by checking the correspo= nding > > entry in the sPAPROptionVector structure. However, since this value = isn't > > migrated currently, we are unable to unplug until after the guest re= boots. > > = > > We address that here by adding migration support for sPAPROptionVect= ors, > > and including the CAS-negotiated vector as part of the migration str= eam > > for any cases where we advertise newer-style hotplug event support to > > the guest. > > = > > David, > > = > > These fixes ended up going out much later than planned. I'm not sure > > if you're planning another pull for 2.8 or not, and realize there are > > some patches here not specifically pseries-related so it's > > understandable if we opt to pursue these for 2.9/2.8.1 instead. But if > > possible I'm hoping to get these in so that the memory unplug > > support is fully functional for 2.8. > = > Yeah, I'm still expecting to push a few bugfixes in before 2.8. So, > I've merged these patches into ppc-for-2.8 (fixing a couple of trivial > style nits along the way). I have a couple of comments that I'll make > on the patches, but they're not important enough to stop these going > in ASAP. > = > Unfortunately, of course, this is not the only migration breakage we > have at the moment. I'm presently wrestling with both breakage due to > changes in the insns_flags masks, and due to the reworking of the mmio > windows for the PHB. Ok, thanks for the heads up. FYI I'm still hoping to get the insns_flags fix in for 2.7.1 (which is a bit behind at this point, should have schedule and initial tree posted next week though), so I will keep an eye out for those. > = > > = > > = > > hw/core/qdev.c | 6 +++++- > > hw/ppc/spapr.c | 68 +++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++ > > hw/ppc/spapr_drc.c | 70 +++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++ > > hw/ppc/spapr_ovec.c | 12 ++++++++++++ > > hw/ppc/spapr_pci.c | 22 ++++++++++++++++++++++ > > include/hw/ppc/spapr_drc.h | 9 +++++++++ > > include/hw/ppc/spapr_ovec.h | 4 ++++ > > include/hw/qdev-core.h | 9 +++++++++ > > migration/savevm.c | 4 ++-- > > 9 files changed, 201 insertions(+), 3 deletions(-) > > = > = > -- = > David Gibson | I'll have my music baroque, and my code > david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _othe= r_ > | _way_ _around_! > http://www.ozlabs.org/~dgibson