From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34423) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c4JpR-0002Ac-F5 for qemu-devel@nongnu.org; Tue, 08 Nov 2016 22:45:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c4JpO-0004ZE-CC for qemu-devel@nongnu.org; Tue, 08 Nov 2016 22:45:37 -0500 Date: Wed, 9 Nov 2016 14:42:35 +1100 From: David Gibson Message-ID: <20161109034235.GC427@umbus.fritz.box> References: <1476672219-8836-1-git-send-email-david@gibson.dropbear.id.au> <1476672219-8836-16-git-send-email-david@gibson.dropbear.id.au> <20161108011645.GC28688@umbus.fritz.box> <4424b1f9-669e-9d6b-022d-e9c3c86c117e@ozlabs.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="E13BgyNx05feLLmH" Content-Disposition: inline In-Reply-To: <4424b1f9-669e-9d6b-022d-e9c3c86c117e@ozlabs.ru> Subject: Re: [Qemu-devel] [PULL 15/16] spapr_pci: Add a 64-bit MMIO window List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: peter.maydell@linaro.org, agraf@suse.de, qemu-ppc@nongnu.org, qemu-devel@nongnu.org --E13BgyNx05feLLmH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 08, 2016 at 02:59:30PM +1100, Alexey Kardashevskiy wrote: > On 08/11/16 12:16, David Gibson wrote: > > On Fri, Nov 04, 2016 at 04:03:31PM +1100, Alexey Kardashevskiy wrote: > >> On 17/10/16 13:43, David Gibson wrote: > >>> On real hardware, and under pHyp, the PCI host bridges on Power machi= nes > >>> typically advertise two outbound MMIO windows from the guest's physic= al > >>> memory space to PCI memory space: > >>> - A 32-bit window which maps onto 2GiB..4GiB in the PCI address spa= ce > >>> - A 64-bit window which maps onto a large region somewhere high in = PCI > >>> address space (traditionally this used an identity mapping from g= uest > >>> physical address to PCI address, but that's not always the case) > >>> > >>> The qemu implementation in spapr-pci-host-bridge, however, only suppo= rts a > >>> single outbound MMIO window, however. At least some Linux versions e= xpect > >>> the two windows however, so we arranged this window to map onto the P= CI > >>> memory space from 2 GiB..~64 GiB, then advertised it as two contiguous > >>> windows, the "32-bit" window from 2G..4G and the "64-bit" window from > >>> 4G..~64G. > >>> > >>> This approach means, however, that the 64G window is not naturally al= igned. > >>> In turn this limits the size of the largest BAR we can map (which doe= s have > >>> to be naturally aligned) to roughly half of the total window. With s= ome > >>> large nVidia GPGPU cards which have huge memory BARs, this is startin= g to > >>> be a problem. > >>> > >>> This patch adds true support for separate 32-bit and 64-bit outbound = MMIO > >>> windows to the spapr-pci-host-bridge implementation, each of which can > >>> be independently configured. The 32-bit window always maps to 2G.. i= n PCI > >>> space, but the PCI address of the 64-bit window can be configured (it > >>> defaults to the same as the guest physical address). > >>> > >>> So as not to break possible existing configurations, as long as a 64-= bit > >>> window is not specified, a large single window can be specified. This > >>> will appear the same way to the guest as the old approach, although i= t's > >>> now implemented by two contiguous memory regions rather than a single= one. > >>> > >>> For now, this only adds the possibility of 64-bit windows. The defau= lt > >>> configuration still uses the legacy mode. > >> > >> > >> This breaks migration to QEMU v2.7, the destination reports: > >> > >> 22901@1478235261.799031:vmstate_load spapr_pci, spapr_pci > >> 22901@1478235261.799040:vmstate_load_field_error field "mem_win_size" = load > >> failed, ret =3D -22 > >> qemu-hostos1: error while loading state for instance 0x0 of device 'sp= apr_pci' > >> 22901@1478235261.801324:migrate_set_state new state 7 > >> qemu-hostos1: load of migration failed: Invalid argument > >> > >> > >> mem_win_size decreased from 0xf80000000 to 0x80000000. > >> > >> I'd think it should be allowed to migrate like this. > >=20 > > AIUI, we don't generally care (upstream) about migration from newer to > > older qemu, only from older to newer.=20 >=20 > Older (v2.7.0) to newer (current upstream with -machine pseries-2.7) does > not work either with the exact same symptom. Drat. Ok.. I see why. I was converting the old style property into new-style meaning during property parsing, but the "raw" property value was still being sent and compared in the migration stream. It would be possible to fix it both ways, by keeping around the "raw" mem_window_size parameter and having some pre_save / post_load logic to shuffle the various possibilities. But that's going to leave ugly cruft around indefinitely. I think it's preferable to just bump the version number and drop those more-trouble-than-they're-worth VMSTATE_EQUAL fields. Patch coming shortly. --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --E13BgyNx05feLLmH Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYIpspAAoJEGw4ysog2bOScEsP/A3RZzxNd/c7UQrJNzAnxWjN VzaKEAAhNO4dIhe25+joloMlsXJVoydpMUcojIq+i6naTzSf68eQ6+UusigC+FpJ MGNMiw9Z8Q07rHiTo0NGQYP71fOTNv0yBzbCdBDiBVorjuHoJ0pgI8EP+GOQfezJ OQyje2f4u55c/0pyfB3p3Led+OcIQdrtDuo9bdIPdNaMP046GYlVrr2JYNBNU4bO FtEvJiOfZ2/cVafj6Yb7w439rqilD7OEUmo5CYf06kYIMcUdfRXzotDwTTajIioq 4L0+qZWsbO3Qm0OPWGgWqvhUsL8BuCVXFKyah87jOHGdKr/XNTSmYgMWfZ/tDpBK HO6frOpUEh/wYeXQoFQfvhSlWycHZdi2AoshLBj7bpAVVk/DNnc+92n/jGQf83Vb wTGb4McaW73TzUSePsDO7IkC51clxbsK3fb0lHnNGk0iSTF1dvUdogy3ooXaHDRO zdekETRjxbmsqRBtGe8FrsN3wcE6zdfgA2ln2hqq7JwISMnzAEr/9oD3xP+9YdsW LjkDGWc6mMxELh0FPQ/PTA76EiklTriYNWO5MQPu/lBiDUrfGU7+KcTYJMDBiK2i ixAajVl5UrsB3JN19tqObZUYVqdCVDlygqRdD92s+2hGhreaR5jbTB/qmgugcv8P jB7rXFiyhhIWt3umekZK =Segu -----END PGP SIGNATURE----- --E13BgyNx05feLLmH--