From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39548) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btTcD-00074i-FT for qemu-devel@nongnu.org; Mon, 10 Oct 2016 01:59:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1btTc9-0004Rq-7c for qemu-devel@nongnu.org; Mon, 10 Oct 2016 01:59:08 -0400 Date: Mon, 10 Oct 2016 16:31:20 +1100 From: David Gibson Message-ID: <20161010053120.GG22498@umbus.fritz.box> References: <1475519097-27611-1-git-send-email-duanj@linux.vnet.ibm.com> <1475519097-27611-2-git-send-email-duanj@linux.vnet.ibm.com> <20161005101254.GD2036@work-vm> <9dc23ca2-9ca6-9b2e-9c22-257d11e234d1@linux.vnet.ibm.com> <20161007025442.GK18490@umbus.fritz.box> <20161007080749.GA2040@work-vm> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="3607uds81ZQvwCD0" Content-Disposition: inline In-Reply-To: <20161007080749.GA2040@work-vm> Subject: Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: Jianjun Duan , qemu-devel@nongnu.org, qemu-ppc@nongnu.org, dmitry@daynix.com, peter.maydell@linaro.org, kraxel@redhat.com, mst@redhat.com, pbonzini@redhat.com, veroniabahaa@gmail.com, quintela@redhat.com, amit.shah@redhat.com, mreitz@redhat.com, kwolf@redhat.com, rth@twiddle.net, aurelien@aurel32.net, leon.alrae@imgtec.com, blauwirbel@gmail.com, mark.cave-ayland@ilande.co.uk, mdroth@linux.vnet.ibm.com --3607uds81ZQvwCD0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 07, 2016 at 09:07:49AM +0100, Dr. David Alan Gilbert wrote: > * David Gibson (david@gibson.dropbear.id.au) wrote: > > On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote: > > > Please see comments below: > > >=20 > > > On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote: > > > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote: > > > >> In QOM(QEMU Object Model) migrated objects are identified with ins= tance_id > > > >> which is calculated automatically using their path in the QOM comp= osition > > > >> tree. For some objects, this path could change from source to targ= et in > > > >> migration. To migrate such objects, we need to make sure the insta= nce_id does > > > >> not change from source to target. We add a hook in DeviceClass to = do customized > > > >> instance_id calculation in such cases. > > > >=20 > > > > Can you explain a bit about why the path changes from source to des= tination; > > > > the path here should be a feature of the guest state not the host, = and so I > > > > don't understand why it changes. > > > Please see the discussion with David in the previous versions: > > > http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html > >=20 > > Um.. your description above really isn't an accurate summary of that > > discussion. > >=20 > > The point is not that the qom path will vary from source to > > destination for some arbitrary reason, but rather that we anticipate > > future changes in the QOM structure. Specifically we're considering > > eliminating the DRC objects, and folding their (limited) state into an > > array in the parent object (either the machine or a PCI host bridge). > >=20 > > That would change the qom paths, and hence the auto-generated instance > > ids, which would break migration between qemu versions before and > > after the restructure. > >=20 > > I'm not sure that changing the instance ids is enough though, anyway, > > since we're talking about eliminating the object entirely, the > > class/type information in the migration stream also wouldn't match. > >=20 > > Dave, if you have ideas on how to deal with that, I'd love to hear > > them >=20 > Eliminating the object entirely would be tricky to deal with; > allowing the structure to change would work if instead of a custom instan= ce_id > you had a custom idstr. Sorry, two misunderstandings here. * When I said "structure" I meant in the high-level sense of what objects exist and are responsible for what things, not in the 'struct WhateverState' sense. * In fact right now eliminating the objects would be ok, since they have no migration state (which causes the problems this series is trying to address). But applying this series which adds migration state would make it difficult to eliminate the objects in future. That's an awkward constraint given that we've already got some hints that these objects are not a good idea. > However, the question then becomes why is the structure changing so much; > ideally things in the migration stream should represent some tangible obj= ect > that corresponds to something real, but (again ideally) the contents > of the stream should reflect the state of those objects not the current > implementation - so if you want to change the implementation the stream > doesn't change. Is it your implementation, or the understanding of what > the objects actually represent that's in flux? So, the objects in question are DRCs - "Dynamic Re-configuration Connector"; silly IBM talk for "a port into which something can be hotplugged", bascially. These aren't "real" devices, but rather a firmware/hypervisor abstraction which are used to describe a hotplug point. Each DRC allows either one CPU core, one PCI device, or one LMB (256MiB chunk of RAM) to be hotplugged (or removed). The PCI DRCs are "owned" by the PCI host bridge to which the device would be connected, the CPU and memory DRCs are owned by the machine. The state variables which Jianjun Duan is adding to migration are values defined in the PAPR (hypervisor interface) spec, and so are tangible enough to be sensible to migrate. At the moment, each LMB is represented by a discrete QOM object, but I've been thinking for a while that this may be a mistake. In particular it's a problem for the LMB DRCs - because each LMB is only 256MiB of memory, we end up with thousands, maybe tens of thousands of DRC objects on a guest with with large maxmem (even if initial memory is small). The QOM infrastructure doesn't deal terribly well that many object child properties. The construction of those DRCs used to be an N^3 algorithm, which as you'd imagine caused horrible startup times with large maxmem (minutes to hours above ~512GiB). We fixed it to be only N^2, so now the startup delays are merely bad (at least once you get to ~2TiB+ maxmem). We could fix that by changing QOM to use a hash or tree for object children, instead of a plain list, but I'm not sure if it makes sense to do that with only this use case. So, for some time, I've been considering rewriting the DRC stuff to be implemented via QOM interface on the "owner" object, rather than separate objects for every single connector. So the difficulty is that if we add the state migration to the DRC objects as they stand, we prevent ourselves from doing a refactor that's probably a good idea. But no-one's yet had time to actually thrash out such a refactor. :/ --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --3607uds81ZQvwCD0 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJX+yelAAoJEGw4ysog2bOSZzIP/i6U/61eWqsUU7yb30aEau/T jcEeyTDS52o8f1llqy8IxKq7j3W0rpc60qa4K5yOLba+U8C2RA47utXiG5ZJgQid pM51NaY7HdNLEfqf3b5PD8Iric5SjZe3NHMjanjzjLxEn7KawPxMJxl5cSWpihaj i1T1l9n56oLjmU9Qd3TeH9NxB4LgitcJw8L0OwsA66qmFogydRBWD7MskGoSRVfK MP46tuM+bX3VOaK384nfLsfuuf6M0DfjDMam1NEqtc4zfhT0a3zlCmON8Gvd1aV5 pgVpsNhSODjVVAGCNgmaM1gw83MnxX+7OhAu5iO3LSLef1hPf0JzkEQqThyeHfQ9 oPbqH3CeVLfkmLXUubmUi+LHF1pjEdcrFWtZQGZtgUwssrP3OC6MfauSJT2ibdkL QvTNUc7Iqrj4yG44WenSnuFYmCR6idlThNyOQifUeScvE8I0q1nxo74A7coPvfnM hsnEvY0kzhCW25JdiGLtEziSjm1oG7TLXSRasP8cCh+RmgP6rHxDdZDlGRga5PQZ qoFlWBXYciUmyq3nqY467w7o/qWq3m0ifzc926yBCpY840LwOiIpXX9JXxFVi1Hp n8bICiAC2VncR73p3kpMqH6BA/OSv3hqO9NGIM5XSg02LF67f4PzbElX3b6jF6Ms 3V8MkwS4pHU/iH+V/v4P =dHMI -----END PGP SIGNATURE----- --3607uds81ZQvwCD0--