From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gleb Natapov Subject: Re: guest assigned device MMIO maps with WC: does this work correctly? Date: Thu, 21 Nov 2013 10:25:52 +0200 Message-ID: <20131121082552.GA19747@redhat.com> References: <20131120195815.GA11239@redhat.com> <20131121071855.GD2108@redhat.com> <20131121080207.GA19812@redhat.com> <20131121080126.GG2108@redhat.com> <20131121081627.GA19951@redhat.com> <20131121081834.GA20041@redhat.com> <20131121082133.GB20041@redhat.com> <20131121082534.GC20041@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=cp1255 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org To: "Michael S. Tsirkin" Return-path: Received: from mx1.redhat.com ([209.132.183.28]:43191 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751154Ab3KUIZ5 convert rfc822-to-8bit (ORCPT ); Thu, 21 Nov 2013 03:25:57 -0500 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id rAL8PraF032069 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 21 Nov 2013 03:25:53 -0500 Content-Disposition: inline In-Reply-To: <20131121082534.GC20041@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Thu, Nov 21, 2013 at 10:25:34AM +0200, Michael S. Tsirkin wrote: > On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote: > > On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote: > > > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrot= e: > > > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote: > > > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin = wrote: > > > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrot= e: > > > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsir= kin wrote: > > > > > > > > I see this in kvm: > > > > > > > >=20 > > > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t= gfn, bool > > > > > > > > is_mmio) > > > > > > > > { =20 > > > > > > > > u64 ret; > > > > > > > > =20 > > > > > > > > /* For VT-d and EPT combination > > > > > > > > * 1. MMIO: always map as UC > > > > > > > > * 2. EPT with VT-d: > > > > > > > > * a. VT-d without snooping control feature: = can't guarantee > > > > > > > > * the > > > > > > > > * result, try to trust guest. > > > > > > > > * b. VT-d with snooping control feature: sno= oping control > > > > > > > > * feature of > > > > > > > > * VT-d engine can guarantee the cache cor= rectness. Just > > > > > > > > * set it > > > > > > > > * to WB to keep consistent with host. So = the same as item > > > > > > > > * 3. > > > > > > > > * 3. EPT without VT-d: always map as WB and se= t IPAT=3D1 to keep > > > > > > > > * consistent with host MTRR=20 > > > > > > > > */ > > > > > > > > if (is_mmio) > > > > > > > > ret =3D MTRR_TYPE_UNCACHABLE << VMX_EPT= _MT_EPTE_SHIFT; > > > > > > > >=20 > > > > > > > > ... > > > > > > > > } > > > > > > > >=20 > > > > > > > >=20 > > > > > > > > does this mean that even if guest maps BAR for an assig= ned device > > > > > > > > as write combined (or configures such using an MTRR), > > > > > > > > host will override this and use uncacheable in practice= ? > > > > > > > >=20 > > > > > > > No, it does not mean that. I already answered this once (= my previous > > > > > > > answer included below): effective memory type is a combin= ation of MTRR > > > > > > > (EPT MT bits in case of a guest) and PAT bits. See sectio= n 11.5.2.2 > > > > > > > in SDM > > > > > >=20 > > > > > >=20 > > > > > > Can you quote chapter name please? > > > > > > My SDM has > > > > > > 11.5.2.2 Denormal-Operand Exception (#D) > > > > > >=20 > > > > > Either your or mine is out of date: > > > > > Selecting Memory Types for Pentium III and More Recent Proces= sor Families > > > >=20 > > > > OK this one I'm familiar with, it describes how PAT > > > > interacts with MTRR. But how does this interact with EPT? > > > > do you remember where's that described? > > >=20 > > > Found it 28.2.4 EPT and Memory Typing > > > sorry about the noise. > >=20 > >=20 > > OK and that's explicit: > > If CR0.CD =3D 0, the effective memory type depends upon the value o= f bit > > 6 of the last EPT paging-structure entry: > > =97 If the value is 0, the effective memory type is the combination= of the > > EPT memory type and the PAT memory type specified in Table 11-7 in = Section > > 11.5.2.2, using the EPT memory type in place of the MTRR memory typ= e. > >=20 > > =97 If the value is 1, the memory type used for the access is the E= PT > > memory > > type. The PAT memory type is ignored. > >=20 > > If CR0.CD =3D 1, the effective memory type is UC. > >=20 > > So it's simple. EPT replaces guest's MTRR. >=20 >=20 > And that in turn means that since we set UC in EPT, > VCPU will always work as if it's UC except for > guests using WC - WC takes precedence. >=20 That's what I wrote below, no? :) >=20 > Thanks! > > >=20 > > > > > > > on how effective memory type is calculated. > > > > > > >=20 > > > > > > > Since MTRR UC + PAT WC =3D WC, if guest maps MMIO as WC = in a page table > > > > > > > (that what ioremap_wc does), everything works as it shou= ld. If guest maps > > > > > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (lik= e any MMIO will > > > > > > > be by default) combined memory type will be UC, so also = fine. If guest > > > > > > > maps MMIO range as WB and fixes mtrr for this region to = be WB then memory > > > > > > > type will be incorrect in a guest, > > > > > >=20 > > > > > > Meaning MTRR in guest is ignored in this case? > > > > > Yes. > > > > >=20 > > > > > >=20 > > > > > > > but I found only one place that does > > > > > > > it in Linux: drivers/video/vesafb.c. All other uses of i= oremap_cache > > > > > > > either remap RAM or used to get whatever memory type con= figured in MTRR. > > > > > > >=20 > > > > > > > -- > > > > > > > Gleb. > > > > >=20 > > > > > -- > > > > > Gleb. -- Gleb.