Re: guest assigned device MMIO maps with WC: does this work correctly?

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > I see this in kvm:
> > > > > > > 
> > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > is_mmio)
> > > > > > > {        
> > > > > > >         u64 ret;
> > > > > > >          
> > > > > > >         /* For VT-d and EPT combination
> > > > > > >          * 1. MMIO: always map as UC
> > > > > > >          * 2. EPT with VT-d:
> > > > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > > > >          *   the
> > > > > > >          *      result, try to trust guest.
> > > > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > > > >          *   feature of
> > > > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > > > >          *      set it
> > > > > > >          *      to WB to keep consistent with host. So the same as item
> > > > > > >          *      3.
> > > > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > >          *    consistent with host MTRR 
> > > > > > >          */
> > > > > > >         if (is_mmio)
> > > > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > > 
> > > > > > > ...
> > > > > > > }
> > > > > > > 
> > > > > > > 
> > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > host will override this and use uncacheable in practice?
> > > > > > > 
> > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > in SDM
> > > > > 
> > > > > 
> > > > > Can you quote chapter name please?
> > > > > My SDM has
> > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > > 
> > > > Either your or mine is out of date:
> > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > > 
> > > OK this one I'm familiar with, it describes how PAT
> > > interacts with MTRR. But how does this interact with EPT?
> > > do you remember where's that described?
> > 
> > Found it 28.2.4 EPT and Memory Typing
> > sorry about the noise.
> 
> 
> OK and that's explicit:
> If CR0.CD = 0, the effective memory type depends upon the value of bit
> 6 of the last EPT paging-structure entry:
> — If the value is 0, the effective memory type is the combination of the
> EPT memory type and the PAT memory type specified in Table 11-7 in Section
> 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
> 
> — If the value is 1, the memory type used for the access is the EPT
> memory
> type. The PAT memory type is ignored.
> 
> If CR0.CD = 1, the effective memory type is UC.
> 
> So it's simple. EPT replaces guest's MTRR.


And that in turn means that since we set UC in EPT,
VCPU will always work as if it's UC except for
guests using WC - WC takes precedence.


Thanks!
> > 
> > > > > > on how effective memory type is calculated.
> > > > > > 
> > > > > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > >  be by default) combined memory type will be UC, so also fine. If guest
> > > > > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > >  type will be incorrect in a guest,
> > > > > 
> > > > > Meaning  MTRR in guest is ignored in this case?
> > > > Yes.
> > > > 
> > > > > 
> > > > > > but I found only one place that does
> > > > > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > > 
> > > > > > --
> > > > > > 			Gleb.
> > > > 
> > > > --
> > > > 			Gleb.