From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41823) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YJKlR-00017d-HG for qemu-devel@nongnu.org; Thu, 05 Feb 2015 06:38:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YJKlN-0004v0-46 for qemu-devel@nongnu.org; Thu, 05 Feb 2015 06:38:29 -0500 Date: Thu, 5 Feb 2015 22:30:07 +1100 From: David Gibson Message-ID: <20150205113007.GT25675@voom.fritz.box> References: <1422943851-25836-1-git-send-email-david@gibson.dropbear.id.au> <20150203211906.GA13992@iris.ozlabs.ibm.com> <20150204013211.GU28703@voom.fritz.box> <54D23872.90007@suse.de> <20150205004812.GD25675@voom.fritz.box> <54D2BF4F.1030609@suse.de> <20150205025556.GH25675@voom.fritz.box> <6EFB0F0E-BB1D-4DAE-8BA4-367B05E88553@suse.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="RlgFhasKO3bfRJ5j" Content-Disposition: inline In-Reply-To: <6EFB0F0E-BB1D-4DAE-8BA4-367B05E88553@suse.de> Subject: Re: [Qemu-devel] [RFC] pseries: Enable in-kernel H_LOGICAL_CI_{LOAD, STORE} implementations List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexander Graf Cc: "aik@ozlabs.ru" , "qemu-ppc@nongnu.org" , Paul Mackerras , "qemu-devel@nongnu.org" , "mdroth@us.ibm.com" --RlgFhasKO3bfRJ5j Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 05, 2015 at 11:22:13AM +0100, Alexander Graf wrote: >=20 >=20 >=20 > > Am 05.02.2015 um 03:55 schrieb David Gibson : > >=20 > >> On Thu, Feb 05, 2015 at 01:54:39AM +0100, Alexander Graf wrote: > >>=20 > >>=20 > >>> On 05.02.15 01:48, David Gibson wrote: > >>>> On Wed, Feb 04, 2015 at 04:19:14PM +0100, Alexander Graf wrote: > >>>>=20 > >>>>=20 > >>>>> On 04.02.15 02:32, David Gibson wrote: > >>>>>> On Wed, Feb 04, 2015 at 08:19:06AM +1100, Paul Mackerras wrote: > >>>>>>> On Tue, Feb 03, 2015 at 05:10:51PM +1100, David Gibson wrote: > >>>>>>> qemu currently implements the hypercalls H_LOGICAL_CI_LOAD and > >>>>>>> H_LOGICAL_CI_STORE as PAPR extensions. These are used by the SLO= F firmware > >>>>>>> for IO, because performing cache inhibited MMIO accesses with the= MMU off > >>>>>>> (real mode) is very awkward on POWER. > >>>>>>>=20 > >>>>>>> This approach breaks when SLOF needs to access IO devices impleme= nted > >>>>>>> within KVM instead of in qemu. The simplest example would be vir= tio-blk > >>>>>>> using an iothread, because the iothread / dataplane mechanism rel= ies on > >>>>>>> an in-kernel implementation of the virtio queue notification MMIO. > >>>>>>>=20 > >>>>>>> To fix this, an in-kernel implementation of these hypercalls has = been made, > >>>>>>> however, the hypercalls still need to be enabled from qemu. This= performs > >>>>>>> the necessary calls to do so. > >>>>>>>=20 > >>>>>>> Signed-off-by: David Gibson > >>>>>>=20 > >>>>>> [snip] > >>>>>>=20 > >>>>>>> + ret1 =3D kvmppc_enable_hcall(kvm_state, H_LOGICAL_CI_LOAD); > >>>>>>> + if (ret1 !=3D 0) { > >>>>>>> + fprintf(stderr, "Warning: error enabling H_LOGICAL_CI_LO= AD in KVM:" > >>>>>>> + " %s\n", strerror(errno)); > >>>>>>> + } > >>>>>>> + > >>>>>>> + ret2 =3D kvmppc_enable_hcall(kvm_state, H_LOGICAL_CI_STORE); > >>>>>>> + if (ret2 !=3D 0) { > >>>>>>> + fprintf(stderr, "Warning: error enabling H_LOGICAL_CI_ST= ORE in KVM:" > >>>>>>> + " %s\n", strerror(errno)); > >>>>>>> + } > >>>>>>> + > >>>>>>> + if ((ret1 !=3D 0) || (ret2 !=3D 0)) { > >>>>>>> + fprintf(stderr, "Warning: Couldn't enable H_LOGICAL_CI_*= in KVM, SLOF" > >>>>>>> + " may be unable to operate devices with in-kerne= l emulation\n"); > >>>>>>> + } > >>>>>>=20 > >>>>>> You'll always get these warnings if you're running on an old (mean= ing > >>>>>> current upstream) kernel, which could be annoying. > >>>>>=20 > >>>>> True. > >>>>>=20 > >>>>>> Is there any way > >>>>>> to tell whether you have configured any devices which need the > >>>>>> in-kernel MMIO emulation and only warn if you have? > >>>>>=20 > >>>>> In theory, I guess so. In practice I can't see how you'd enumerate > >>>>> all devices that might require kernel intervention without something > >>>>> horribly invasive. > >>>>=20 > >>>> We could WARN_ONCE in QEMU if we emulate such a hypercall, but its > >>>> handler is io_mem_unassigned (or we add another minimum priority huge > >>>> memory region on all 64bits of address space that reports the breaka= ge). > >>>=20 > >>> Would that work for the virtio+iothread case? I had the impression > >>> the kernel handled notification region was layered over the qemu > >>> emulated region in that case. > >>=20 > >> IIRC we don't have a way to call back into kvm saying "please write to > >> this in-kernel device". But we could at least defer the warning to a > >> point where we know that we actually hit it. > >=20 > > Right, but I'm saying we might miss the warning in cases where we want > > it, because the KVM device is shadowed by a qemu device, so qemu won't > > see the IO as unassigned or unhandled. > >=20 > > In particular, I think that will happen in the case of virtio-blk with > > iothread, which is the simplest case in which to observe the problem. > > The virtio-blk device exists in qemu and is functional, but we rely on > > KVM catching the queue notification MMIO before it reaches the qemu > > implementation of the rest of the device's IO space. >=20 > But in that case the VM stays functional and will merely see a > performance hit when using virtio in SLOF, no? I don't think that's > a problem worth worrying users about. Alas, no. The iothread stuff *relies* on the in-kernel notification, so it will not work if the IO gets punted to qemu. This is the whole reason for the in-kernel hcall implementation. --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --RlgFhasKO3bfRJ5j Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJU01Q/AAoJEGw4ysog2bOSgoEP/jOgpwfFIEyzoEtRXVeewD1s k13+L4QWxehowVfeakDR1b3SUXd9OzU9ooTOJ8ESyw1EB7pnUJ4zTV+b4AhXVUD7 Kj6VF1VsLxEKsX43dNzShX3TM9Ykqq5+3S6VwPFTh+F+05J642zqvbAOMzX+WwFh 7PNWYSajT6Nn1soY4YyH46u5BFhSCdHyvnhh1+YCr6dd0T7y9hJJG5XmfoVaqsUV UEkLx+msDdOPb1sbs92a+zDNFYN5ukmMw7n72BlvXTkN3OssdJIkWYXdLkgE07jN y4PyIRRIwX+VEd2s88h3yVcrsZcnYhHbrUhvPG3V27ibzkeqKucMHpucb7Xn17hT KBEi+IchDmWDTM3k8ckJEUs9VbUN/vVbblIsREBG5jZtqd46dMNmibW9JcI5s4Fs yluM5ieOz6RS8GEPXyeYraCewr13zlDR5XDkqmCE8bghFUUY9gawgnFqMfM59WRt NQun0WBe5hqq/cQ0uNOdwCv6Iegs7xvoOTOYwuHox2kIF+4AYnXiZWiEzbCj1RpP Ps5XzVteIn2k0+k2SYVv26E939K0b3+srAm3/K27doQHZI70JihPCrCQrqaDijp4 UKae5cz13ekUgYBJMmQTYnvggTGhJyRHgS4+YSNKBt6PlLe0njXWEbapaVyRmfZs 1irI2Dh4y6wZJKeTce94 =i/HQ -----END PGP SIGNATURE----- --RlgFhasKO3bfRJ5j--