From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45258) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YJL2J-00013n-QX for qemu-devel@nongnu.org; Thu, 05 Feb 2015 06:55:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YJL2D-0002Ap-6H for qemu-devel@nongnu.org; Thu, 05 Feb 2015 06:55:55 -0500 Message-ID: <54D35A41.6020907@suse.de> Date: Thu, 05 Feb 2015 12:55:45 +0100 From: Alexander Graf MIME-Version: 1.0 References: <1422943851-25836-1-git-send-email-david@gibson.dropbear.id.au> <20150203211906.GA13992@iris.ozlabs.ibm.com> <20150204013211.GU28703@voom.fritz.box> <54D23872.90007@suse.de> <20150205004812.GD25675@voom.fritz.box> <54D2BF4F.1030609@suse.de> <20150205025556.GH25675@voom.fritz.box> <6EFB0F0E-BB1D-4DAE-8BA4-367B05E88553@suse.de> <20150205113007.GT25675@voom.fritz.box> In-Reply-To: <20150205113007.GT25675@voom.fritz.box> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] pseries: Enable in-kernel H_LOGICAL_CI_{LOAD, STORE} implementations List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: David Gibson Cc: "aik@ozlabs.ru" , "qemu-ppc@nongnu.org" , Paul Mackerras , "qemu-devel@nongnu.org" , "mdroth@us.ibm.com" On 05.02.15 12:30, David Gibson wrote: > On Thu, Feb 05, 2015 at 11:22:13AM +0100, Alexander Graf wrote: >> >> >> >>> Am 05.02.2015 um 03:55 schrieb David Gibson : >>> >>>> On Thu, Feb 05, 2015 at 01:54:39AM +0100, Alexander Graf wrote: >>>> >>>> >>>>> On 05.02.15 01:48, David Gibson wrote: >>>>>> On Wed, Feb 04, 2015 at 04:19:14PM +0100, Alexander Graf wrote: >>>>>> >>>>>> >>>>>>> On 04.02.15 02:32, David Gibson wrote: >>>>>>>> On Wed, Feb 04, 2015 at 08:19:06AM +1100, Paul Mackerras wrote: >>>>>>>>> On Tue, Feb 03, 2015 at 05:10:51PM +1100, David Gibson wrote: >>>>>>>>> qemu currently implements the hypercalls H_LOGICAL_CI_LOAD and >>>>>>>>> H_LOGICAL_CI_STORE as PAPR extensions. These are used by the SLOF firmware >>>>>>>>> for IO, because performing cache inhibited MMIO accesses with the MMU off >>>>>>>>> (real mode) is very awkward on POWER. >>>>>>>>> >>>>>>>>> This approach breaks when SLOF needs to access IO devices implemented >>>>>>>>> within KVM instead of in qemu. The simplest example would be virtio-blk >>>>>>>>> using an iothread, because the iothread / dataplane mechanism relies on >>>>>>>>> an in-kernel implementation of the virtio queue notification MMIO. >>>>>>>>> >>>>>>>>> To fix this, an in-kernel implementation of these hypercalls has been made, >>>>>>>>> however, the hypercalls still need to be enabled from qemu. This performs >>>>>>>>> the necessary calls to do so. >>>>>>>>> >>>>>>>>> Signed-off-by: David Gibson >>>>>>>> >>>>>>>> [snip] >>>>>>>> >>>>>>>>> + ret1 = kvmppc_enable_hcall(kvm_state, H_LOGICAL_CI_LOAD); >>>>>>>>> + if (ret1 != 0) { >>>>>>>>> + fprintf(stderr, "Warning: error enabling H_LOGICAL_CI_LOAD in KVM:" >>>>>>>>> + " %s\n", strerror(errno)); >>>>>>>>> + } >>>>>>>>> + >>>>>>>>> + ret2 = kvmppc_enable_hcall(kvm_state, H_LOGICAL_CI_STORE); >>>>>>>>> + if (ret2 != 0) { >>>>>>>>> + fprintf(stderr, "Warning: error enabling H_LOGICAL_CI_STORE in KVM:" >>>>>>>>> + " %s\n", strerror(errno)); >>>>>>>>> + } >>>>>>>>> + >>>>>>>>> + if ((ret1 != 0) || (ret2 != 0)) { >>>>>>>>> + fprintf(stderr, "Warning: Couldn't enable H_LOGICAL_CI_* in KVM, SLOF" >>>>>>>>> + " may be unable to operate devices with in-kernel emulation\n"); >>>>>>>>> + } >>>>>>>> >>>>>>>> You'll always get these warnings if you're running on an old (meaning >>>>>>>> current upstream) kernel, which could be annoying. >>>>>>> >>>>>>> True. >>>>>>> >>>>>>>> Is there any way >>>>>>>> to tell whether you have configured any devices which need the >>>>>>>> in-kernel MMIO emulation and only warn if you have? >>>>>>> >>>>>>> In theory, I guess so. In practice I can't see how you'd enumerate >>>>>>> all devices that might require kernel intervention without something >>>>>>> horribly invasive. >>>>>> >>>>>> We could WARN_ONCE in QEMU if we emulate such a hypercall, but its >>>>>> handler is io_mem_unassigned (or we add another minimum priority huge >>>>>> memory region on all 64bits of address space that reports the breakage). >>>>> >>>>> Would that work for the virtio+iothread case? I had the impression >>>>> the kernel handled notification region was layered over the qemu >>>>> emulated region in that case. >>>> >>>> IIRC we don't have a way to call back into kvm saying "please write to >>>> this in-kernel device". But we could at least defer the warning to a >>>> point where we know that we actually hit it. >>> >>> Right, but I'm saying we might miss the warning in cases where we want >>> it, because the KVM device is shadowed by a qemu device, so qemu won't >>> see the IO as unassigned or unhandled. >>> >>> In particular, I think that will happen in the case of virtio-blk with >>> iothread, which is the simplest case in which to observe the problem. >>> The virtio-blk device exists in qemu and is functional, but we rely on >>> KVM catching the queue notification MMIO before it reaches the qemu >>> implementation of the rest of the device's IO space. >> >> But in that case the VM stays functional and will merely see a >> performance hit when using virtio in SLOF, no? I don't think that's >> a problem worth worrying users about. > > Alas, no. The iothread stuff *relies* on the in-kernel notification, > so it will not work if the IO gets punted to qemu. This is the whole > reason for the in-kernel hcall implementation. So at least with vhost-net the in-kernel trapping is optional. If we happen to get MMIO into QEMU, we'll just handle it there. Enlighten me why the iothread stuff can't handle it that way too. Alex