From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: guest assigned device MMIO maps with WC: does this work correctly? Date: Thu, 21 Nov 2013 10:34:10 +0200 Message-ID: <20131121083410.GA20102@redhat.com> References: <20131120195815.GA11239@redhat.com> <20131121071855.GD2108@redhat.com> <20131121080207.GA19812@redhat.com> <20131121080126.GG2108@redhat.com> <20131121081627.GA19951@redhat.com> <20131121081834.GA20041@redhat.com> <20131121082133.GB20041@redhat.com> <20131121082534.GC20041@redhat.com> <20131121082552.GA19747@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org To: Gleb Natapov Return-path: Received: from mx1.redhat.com ([209.132.183.28]:47624 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750941Ab3KUIa5 (ORCPT ); Thu, 21 Nov 2013 03:30:57 -0500 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id rAL8Utbp029785 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 21 Nov 2013 03:30:56 -0500 Content-Disposition: inline In-Reply-To: <20131121082552.GA19747@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Thu, Nov 21, 2013 at 10:25:52AM +0200, Gleb Natapov wrote: > On Thu, Nov 21, 2013 at 10:25:34AM +0200, Michael S. Tsirkin wrote: > > On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote: > > > On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrot= e: > > > > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wr= ote: > > > > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote: > > > > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirki= n wrote: > > > > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wr= ote: > > > > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Ts= irkin wrote: > > > > > > > > > I see this in kvm: > > > > > > > > >=20 > > > > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn= _t gfn, bool > > > > > > > > > is_mmio) > > > > > > > > > { =20 > > > > > > > > > u64 ret; > > > > > > > > > =20 > > > > > > > > > /* For VT-d and EPT combination > > > > > > > > > * 1. MMIO: always map as UC > > > > > > > > > * 2. EPT with VT-d: > > > > > > > > > * a. VT-d without snooping control feature= : can't guarantee > > > > > > > > > * the > > > > > > > > > * result, try to trust guest. > > > > > > > > > * b. VT-d with snooping control feature: s= nooping control > > > > > > > > > * feature of > > > > > > > > > * VT-d engine can guarantee the cache c= orrectness. Just > > > > > > > > > * set it > > > > > > > > > * to WB to keep consistent with host. S= o the same as item > > > > > > > > > * 3. > > > > > > > > > * 3. EPT without VT-d: always map as WB and = set IPAT=3D1 to keep > > > > > > > > > * consistent with host MTRR=20 > > > > > > > > > */ > > > > > > > > > if (is_mmio) > > > > > > > > > ret =3D MTRR_TYPE_UNCACHABLE << VMX_E= PT_MT_EPTE_SHIFT; > > > > > > > > >=20 > > > > > > > > > ... > > > > > > > > > } > > > > > > > > >=20 > > > > > > > > >=20 > > > > > > > > > does this mean that even if guest maps BAR for an ass= igned device > > > > > > > > > as write combined (or configures such using an MTRR), > > > > > > > > > host will override this and use uncacheable in practi= ce? > > > > > > > > >=20 > > > > > > > > No, it does not mean that. I already answered this once= (my previous > > > > > > > > answer included below): effective memory type is a comb= ination of MTRR > > > > > > > > (EPT MT bits in case of a guest) and PAT bits. See sect= ion 11.5.2.2 > > > > > > > > in SDM > > > > > > >=20 > > > > > > >=20 > > > > > > > Can you quote chapter name please? > > > > > > > My SDM has > > > > > > > 11.5.2.2 Denormal-Operand Exception (#D) > > > > > > >=20 > > > > > > Either your or mine is out of date: > > > > > > Selecting Memory Types for Pentium III and More Recent Proc= essor Families > > > > >=20 > > > > > OK this one I'm familiar with, it describes how PAT > > > > > interacts with MTRR. But how does this interact with EPT? > > > > > do you remember where's that described? > > > >=20 > > > > Found it 28.2.4 EPT and Memory Typing > > > > sorry about the noise. > > >=20 > > >=20 > > > OK and that's explicit: > > > If CR0.CD =3D 0, the effective memory type depends upon the value= of bit > > > 6 of the last EPT paging-structure entry: > > > =E2=80=94 If the value is 0, the effective memory type is the com= bination of the > > > EPT memory type and the PAT memory type specified in Table 11-7 i= n Section > > > 11.5.2.2, using the EPT memory type in place of the MTRR memory t= ype. > > >=20 > > > =E2=80=94 If the value is 1, the memory type used for the access = is the EPT > > > memory > > > type. The PAT memory type is ignored. > > >=20 > > > If CR0.CD =3D 1, the effective memory type is UC. > > >=20 > > > So it's simple. EPT replaces guest's MTRR. > >=20 > >=20 > > And that in turn means that since we set UC in EPT, > > VCPU will always work as if it's UC except for > > guests using WC - WC takes precedence. > >=20 > That's what I wrote below, no? :) Yes I just added that this applies to other memory types like WT (that probably nobody uses?). > >=20 > > Thanks! > > > >=20 > > > > > > > > on how effective memory type is calculated. > > > > > > > >=20 > > > > > > > > Since MTRR UC + PAT WC =3D WC, if guest maps MMIO as W= C in a page table > > > > > > > > (that what ioremap_wc does), everything works as it sh= ould. If guest maps > > > > > > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (l= ike any MMIO will > > > > > > > > be by default) combined memory type will be UC, so als= o fine. If guest > > > > > > > > maps MMIO range as WB and fixes mtrr for this region t= o be WB then memory > > > > > > > > type will be incorrect in a guest, > > > > > > >=20 > > > > > > > Meaning MTRR in guest is ignored in this case? > > > > > > Yes. > > > > > >=20 > > > > > > >=20 > > > > > > > > but I found only one place that does > > > > > > > > it in Linux: drivers/video/vesafb.c. All other uses of= ioremap_cache > > > > > > > > either remap RAM or used to get whatever memory type c= onfigured in MTRR. > > > > > > > >=20 > > > > > > > > -- > > > > > > > > Gleb. > > > > > >=20 > > > > > > -- > > > > > > Gleb. >=20 > -- > Gleb.