guest assigned device MMIO maps with WC: does this work correctly?

guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

I see this in kvm:

static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
is_mmio)
{        
        u64 ret;
         
        /* For VT-d and EPT combination
         * 1. MMIO: always map as UC
         * 2. EPT with VT-d:
         *   a. VT-d without snooping control feature: can't guarantee
         *   the
         *      result, try to trust guest.
         *   b. VT-d with snooping control feature: snooping control
         *   feature of
         *      VT-d engine can guarantee the cache correctness. Just
         *      set it
         *      to WB to keep consistent with host. So the same as item
         *      3.
         * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
         *    consistent with host MTRR 
         */
        if (is_mmio)
                ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;

...
}


does this mean that even if guest maps BAR for an assigned device
as write combined (or configures such using an MTRR),
host will override this and use uncacheable in practice?

I think there are some drivers that map MMIO as write-combined
for performance (but don't have such hardware) so
I'd like to figure out from code whether this will work correctly.

-- 
MST

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Gleb Natapov]

On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> I see this in kvm:
> 
> static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> is_mmio)
> {        
>         u64 ret;
>          
>         /* For VT-d and EPT combination
>          * 1. MMIO: always map as UC
>          * 2. EPT with VT-d:
>          *   a. VT-d without snooping control feature: can't guarantee
>          *   the
>          *      result, try to trust guest.
>          *   b. VT-d with snooping control feature: snooping control
>          *   feature of
>          *      VT-d engine can guarantee the cache correctness. Just
>          *      set it
>          *      to WB to keep consistent with host. So the same as item
>          *      3.
>          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
>          *    consistent with host MTRR 
>          */
>         if (is_mmio)
>                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> 
> ...
> }
> 
> 
> does this mean that even if guest maps BAR for an assigned device
> as write combined (or configures such using an MTRR),
> host will override this and use uncacheable in practice?
> 
No, it does not mean that. I already answered this once (my previous
answer included below): effective memory type is a combination of MTRR
(EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
in SDM on how effective memory type is calculated.

 Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
 (that what ioremap_wc does), everything works as it should. If guest maps
 MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
 be by default) combined memory type will be UC, so also fine. If guest
 maps MMIO range as WB and fixes mtrr for this region to be WB then memory
 type will be incorrect in a guest, but I found only one place that does
 it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
 either remap RAM or used to get whatever memory type configured in MTRR.

--
			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > I see this in kvm:
> > 
> > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > is_mmio)
> > {        
> >         u64 ret;
> >          
> >         /* For VT-d and EPT combination
> >          * 1. MMIO: always map as UC
> >          * 2. EPT with VT-d:
> >          *   a. VT-d without snooping control feature: can't guarantee
> >          *   the
> >          *      result, try to trust guest.
> >          *   b. VT-d with snooping control feature: snooping control
> >          *   feature of
> >          *      VT-d engine can guarantee the cache correctness. Just
> >          *      set it
> >          *      to WB to keep consistent with host. So the same as item
> >          *      3.
> >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> >          *    consistent with host MTRR 
> >          */
> >         if (is_mmio)
> >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > 
> > ...
> > }
> > 
> > 
> > does this mean that even if guest maps BAR for an assigned device
> > as write combined (or configures such using an MTRR),
> > host will override this and use uncacheable in practice?
> > 
> No, it does not mean that. I already answered this once (my previous
> answer included below): effective memory type is a combination of MTRR
> (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> in SDM


Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)

> on how effective memory type is calculated.
> 
>  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
>  (that what ioremap_wc does), everything works as it should. If guest maps
>  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
>  be by default) combined memory type will be UC, so also fine. If guest
>  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
>  type will be incorrect in a guest,

Meaning  MTRR in guest is ignored in this case?

> but I found only one place that does
>  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
>  either remap RAM or used to get whatever memory type configured in MTRR.
> 
> --
> 			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Gleb Natapov]

On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > I see this in kvm:
> > > 
> > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > is_mmio)
> > > {        
> > >         u64 ret;
> > >          
> > >         /* For VT-d and EPT combination
> > >          * 1. MMIO: always map as UC
> > >          * 2. EPT with VT-d:
> > >          *   a. VT-d without snooping control feature: can't guarantee
> > >          *   the
> > >          *      result, try to trust guest.
> > >          *   b. VT-d with snooping control feature: snooping control
> > >          *   feature of
> > >          *      VT-d engine can guarantee the cache correctness. Just
> > >          *      set it
> > >          *      to WB to keep consistent with host. So the same as item
> > >          *      3.
> > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > >          *    consistent with host MTRR 
> > >          */
> > >         if (is_mmio)
> > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > 
> > > ...
> > > }
> > > 
> > > 
> > > does this mean that even if guest maps BAR for an assigned device
> > > as write combined (or configures such using an MTRR),
> > > host will override this and use uncacheable in practice?
> > > 
> > No, it does not mean that. I already answered this once (my previous
> > answer included below): effective memory type is a combination of MTRR
> > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > in SDM
> 
> 
> Can you quote chapter name please?
> My SDM has
> 11.5.2.2 Denormal-Operand Exception (#D)
> 
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families

> > on how effective memory type is calculated.
> > 
> >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> >  (that what ioremap_wc does), everything works as it should. If guest maps
> >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> >  be by default) combined memory type will be UC, so also fine. If guest
> >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> >  type will be incorrect in a guest,
> 
> Meaning  MTRR in guest is ignored in this case?
Yes.

> 
> > but I found only one place that does
> >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> >  either remap RAM or used to get whatever memory type configured in MTRR.
> > 
> > --
> > 			Gleb.

--
			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > I see this in kvm:
> > > > 
> > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > is_mmio)
> > > > {        
> > > >         u64 ret;
> > > >          
> > > >         /* For VT-d and EPT combination
> > > >          * 1. MMIO: always map as UC
> > > >          * 2. EPT with VT-d:
> > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > >          *   the
> > > >          *      result, try to trust guest.
> > > >          *   b. VT-d with snooping control feature: snooping control
> > > >          *   feature of
> > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > >          *      set it
> > > >          *      to WB to keep consistent with host. So the same as item
> > > >          *      3.
> > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > >          *    consistent with host MTRR 
> > > >          */
> > > >         if (is_mmio)
> > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > 
> > > > ...
> > > > }
> > > > 
> > > > 
> > > > does this mean that even if guest maps BAR for an assigned device
> > > > as write combined (or configures such using an MTRR),
> > > > host will override this and use uncacheable in practice?
> > > > 
> > > No, it does not mean that. I already answered this once (my previous
> > > answer included below): effective memory type is a combination of MTRR
> > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > in SDM
> > 
> > 
> > Can you quote chapter name please?
> > My SDM has
> > 11.5.2.2 Denormal-Operand Exception (#D)
> > 
> Either your or mine is out of date:
> Selecting Memory Types for Pentium III and More Recent Processor Families

OK this one I'm familiar with, it describes how PAT
interacts with MTRR. But how does this interact with EPT?
do you remember where's that described?

> > > on how effective memory type is calculated.
> > > 
> > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > >  be by default) combined memory type will be UC, so also fine. If guest
> > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > >  type will be incorrect in a guest,
> > 
> > Meaning  MTRR in guest is ignored in this case?
> Yes.
> 
> > 
> > > but I found only one place that does
> > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > 
> > > --
> > > 			Gleb.
> 
> --
> 			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Gleb Natapov]

On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > I see this in kvm:
> > > > > 
> > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > is_mmio)
> > > > > {        
> > > > >         u64 ret;
> > > > >          
> > > > >         /* For VT-d and EPT combination
> > > > >          * 1. MMIO: always map as UC
> > > > >          * 2. EPT with VT-d:
> > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > >          *   the
> > > > >          *      result, try to trust guest.
> > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > >          *   feature of
> > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > >          *      set it
> > > > >          *      to WB to keep consistent with host. So the same as item
> > > > >          *      3.
> > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > >          *    consistent with host MTRR 
> > > > >          */
> > > > >         if (is_mmio)
> > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > 
> > > > > ...
> > > > > }
> > > > > 
> > > > > 
> > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > as write combined (or configures such using an MTRR),
> > > > > host will override this and use uncacheable in practice?
> > > > > 
> > > > No, it does not mean that. I already answered this once (my previous
> > > > answer included below): effective memory type is a combination of MTRR
> > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > in SDM
> > > 
> > > 
> > > Can you quote chapter name please?
> > > My SDM has
> > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > 
> > Either your or mine is out of date:
> > Selecting Memory Types for Pentium III and More Recent Processor Families
> 
> OK this one I'm familiar with, it describes how PAT
> interacts with MTRR. But how does this interact with EPT?
> do you remember where's that described?
> 
28.2.5.2 Memory Type Used for Translated Guest-Physical Addresses
Substitute MTRR with EPT MT and go to the same table.

--
			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > I see this in kvm:
> > > > > 
> > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > is_mmio)
> > > > > {        
> > > > >         u64 ret;
> > > > >          
> > > > >         /* For VT-d and EPT combination
> > > > >          * 1. MMIO: always map as UC
> > > > >          * 2. EPT with VT-d:
> > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > >          *   the
> > > > >          *      result, try to trust guest.
> > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > >          *   feature of
> > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > >          *      set it
> > > > >          *      to WB to keep consistent with host. So the same as item
> > > > >          *      3.
> > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > >          *    consistent with host MTRR 
> > > > >          */
> > > > >         if (is_mmio)
> > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > 
> > > > > ...
> > > > > }
> > > > > 
> > > > > 
> > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > as write combined (or configures such using an MTRR),
> > > > > host will override this and use uncacheable in practice?
> > > > > 
> > > > No, it does not mean that. I already answered this once (my previous
> > > > answer included below): effective memory type is a combination of MTRR
> > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > in SDM
> > > 
> > > 
> > > Can you quote chapter name please?
> > > My SDM has
> > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > 
> > Either your or mine is out of date:
> > Selecting Memory Types for Pentium III and More Recent Processor Families
> 
> OK this one I'm familiar with, it describes how PAT
> interacts with MTRR. But how does this interact with EPT?
> do you remember where's that described?

Found it 28.2.4 EPT and Memory Typing
sorry about the noise.


> > > > on how effective memory type is calculated.
> > > > 
> > > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > >  be by default) combined memory type will be UC, so also fine. If guest
> > > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > >  type will be incorrect in a guest,
> > > 
> > > Meaning  MTRR in guest is ignored in this case?
> > Yes.
> > 
> > > 
> > > > but I found only one place that does
> > > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > > 
> > > > --
> > > > 			Gleb.
> > 
> > --
> > 			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > I see this in kvm:
> > > > > > 
> > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > is_mmio)
> > > > > > {        
> > > > > >         u64 ret;
> > > > > >          
> > > > > >         /* For VT-d and EPT combination
> > > > > >          * 1. MMIO: always map as UC
> > > > > >          * 2. EPT with VT-d:
> > > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > > >          *   the
> > > > > >          *      result, try to trust guest.
> > > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > > >          *   feature of
> > > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > > >          *      set it
> > > > > >          *      to WB to keep consistent with host. So the same as item
> > > > > >          *      3.
> > > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > >          *    consistent with host MTRR 
> > > > > >          */
> > > > > >         if (is_mmio)
> > > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > 
> > > > > > ...
> > > > > > }
> > > > > > 
> > > > > > 
> > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > as write combined (or configures such using an MTRR),
> > > > > > host will override this and use uncacheable in practice?
> > > > > > 
> > > > > No, it does not mean that. I already answered this once (my previous
> > > > > answer included below): effective memory type is a combination of MTRR
> > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > in SDM
> > > > 
> > > > 
> > > > Can you quote chapter name please?
> > > > My SDM has
> > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > 
> > > Either your or mine is out of date:
> > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > 
> > OK this one I'm familiar with, it describes how PAT
> > interacts with MTRR. But how does this interact with EPT?
> > do you remember where's that described?
> 
> Found it 28.2.4 EPT and Memory Typing
> sorry about the noise.


OK and that's explicit:
If CR0.CD = 0, the effective memory type depends upon the value of bit
6 of the last EPT paging-structure entry:
— If the value is 0, the effective memory type is the combination of the
EPT memory type and the PAT memory type specified in Table 11-7 in Section
11.5.2.2, using the EPT memory type in place of the MTRR memory type.

— If the value is 1, the memory type used for the access is the EPT
memory
type. The PAT memory type is ignored.

If CR0.CD = 1, the effective memory type is UC.

So it's simple. EPT replaces guest's MTRR.

> 
> > > > > on how effective memory type is calculated.
> > > > > 
> > > > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > > > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > >  be by default) combined memory type will be UC, so also fine. If guest
> > > > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > >  type will be incorrect in a guest,
> > > > 
> > > > Meaning  MTRR in guest is ignored in this case?
> > > Yes.
> > > 
> > > > 
> > > > > but I found only one place that does
> > > > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > 
> > > > > --
> > > > > 			Gleb.
> > > 
> > > --
> > > 			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > I see this in kvm:
> > > > > > > 
> > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > is_mmio)
> > > > > > > {        
> > > > > > >         u64 ret;
> > > > > > >          
> > > > > > >         /* For VT-d and EPT combination
> > > > > > >          * 1. MMIO: always map as UC
> > > > > > >          * 2. EPT with VT-d:
> > > > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > > > >          *   the
> > > > > > >          *      result, try to trust guest.
> > > > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > > > >          *   feature of
> > > > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > > > >          *      set it
> > > > > > >          *      to WB to keep consistent with host. So the same as item
> > > > > > >          *      3.
> > > > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > >          *    consistent with host MTRR 
> > > > > > >          */
> > > > > > >         if (is_mmio)
> > > > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > > 
> > > > > > > ...
> > > > > > > }
> > > > > > > 
> > > > > > > 
> > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > host will override this and use uncacheable in practice?
> > > > > > > 
> > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > in SDM
> > > > > 
> > > > > 
> > > > > Can you quote chapter name please?
> > > > > My SDM has
> > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > > 
> > > > Either your or mine is out of date:
> > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > > 
> > > OK this one I'm familiar with, it describes how PAT
> > > interacts with MTRR. But how does this interact with EPT?
> > > do you remember where's that described?
> > 
> > Found it 28.2.4 EPT and Memory Typing
> > sorry about the noise.
> 
> 
> OK and that's explicit:
> If CR0.CD = 0, the effective memory type depends upon the value of bit
> 6 of the last EPT paging-structure entry:
> — If the value is 0, the effective memory type is the combination of the
> EPT memory type and the PAT memory type specified in Table 11-7 in Section
> 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
> 
> — If the value is 1, the memory type used for the access is the EPT
> memory
> type. The PAT memory type is ignored.
> 
> If CR0.CD = 1, the effective memory type is UC.
> 
> So it's simple. EPT replaces guest's MTRR.


And that in turn means that since we set UC in EPT,
VCPU will always work as if it's UC except for
guests using WC - WC takes precedence.


Thanks!
> > 
> > > > > > on how effective memory type is calculated.
> > > > > > 
> > > > > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > >  be by default) combined memory type will be UC, so also fine. If guest
> > > > > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > >  type will be incorrect in a guest,
> > > > > 
> > > > > Meaning  MTRR in guest is ignored in this case?
> > > > Yes.
> > > > 
> > > > > 
> > > > > > but I found only one place that does
> > > > > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > > 
> > > > > > --
> > > > > > 			Gleb.
> > > > 
> > > > --
> > > > 			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Gleb Natapov]

On Thu, Nov 21, 2013 at 10:25:34AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > > I see this in kvm:
> > > > > > > > 
> > > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > > is_mmio)
> > > > > > > > {        
> > > > > > > >         u64 ret;
> > > > > > > >          
> > > > > > > >         /* For VT-d and EPT combination
> > > > > > > >          * 1. MMIO: always map as UC
> > > > > > > >          * 2. EPT with VT-d:
> > > > > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > > > > >          *   the
> > > > > > > >          *      result, try to trust guest.
> > > > > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > > > > >          *   feature of
> > > > > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > > > > >          *      set it
> > > > > > > >          *      to WB to keep consistent with host. So the same as item
> > > > > > > >          *      3.
> > > > > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > > >          *    consistent with host MTRR 
> > > > > > > >          */
> > > > > > > >         if (is_mmio)
> > > > > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > > > 
> > > > > > > > ...
> > > > > > > > }
> > > > > > > > 
> > > > > > > > 
> > > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > > host will override this and use uncacheable in practice?
> > > > > > > > 
> > > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > > in SDM
> > > > > > 
> > > > > > 
> > > > > > Can you quote chapter name please?
> > > > > > My SDM has
> > > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > > > 
> > > > > Either your or mine is out of date:
> > > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > > > 
> > > > OK this one I'm familiar with, it describes how PAT
> > > > interacts with MTRR. But how does this interact with EPT?
> > > > do you remember where's that described?
> > > 
> > > Found it 28.2.4 EPT and Memory Typing
> > > sorry about the noise.
> > 
> > 
> > OK and that's explicit:
> > If CR0.CD = 0, the effective memory type depends upon the value of bit
> > 6 of the last EPT paging-structure entry:
> > — If the value is 0, the effective memory type is the combination of the
> > EPT memory type and the PAT memory type specified in Table 11-7 in Section
> > 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
> > 
> > — If the value is 1, the memory type used for the access is the EPT
> > memory
> > type. The PAT memory type is ignored.
> > 
> > If CR0.CD = 1, the effective memory type is UC.
> > 
> > So it's simple. EPT replaces guest's MTRR.
> 
> 
> And that in turn means that since we set UC in EPT,
> VCPU will always work as if it's UC except for
> guests using WC - WC takes precedence.
> 
That's what I wrote below, no? :)

> 
> Thanks!
> > > 
> > > > > > > on how effective memory type is calculated.
> > > > > > > 
> > > > > > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > > >  be by default) combined memory type will be UC, so also fine. If guest
> > > > > > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > > >  type will be incorrect in a guest,
> > > > > > 
> > > > > > Meaning  MTRR in guest is ignored in this case?
> > > > > Yes.
> > > > > 
> > > > > > 
> > > > > > > but I found only one place that does
> > > > > > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > > > 
> > > > > > > --
> > > > > > > 			Gleb.
> > > > > 
> > > > > --
> > > > > 			Gleb.

--
			Gleb.

Re: guest assigned device MMIO maps with WC: does this work correctly?

[Michael S. Tsirkin]

On Thu, Nov 21, 2013 at 10:25:52AM +0200, Gleb Natapov wrote:
> On Thu, Nov 21, 2013 at 10:25:34AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > > > I see this in kvm:
> > > > > > > > > 
> > > > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > > > is_mmio)
> > > > > > > > > {        
> > > > > > > > >         u64 ret;
> > > > > > > > >          
> > > > > > > > >         /* For VT-d and EPT combination
> > > > > > > > >          * 1. MMIO: always map as UC
> > > > > > > > >          * 2. EPT with VT-d:
> > > > > > > > >          *   a. VT-d without snooping control feature: can't guarantee
> > > > > > > > >          *   the
> > > > > > > > >          *      result, try to trust guest.
> > > > > > > > >          *   b. VT-d with snooping control feature: snooping control
> > > > > > > > >          *   feature of
> > > > > > > > >          *      VT-d engine can guarantee the cache correctness. Just
> > > > > > > > >          *      set it
> > > > > > > > >          *      to WB to keep consistent with host. So the same as item
> > > > > > > > >          *      3.
> > > > > > > > >          * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > > > >          *    consistent with host MTRR 
> > > > > > > > >          */
> > > > > > > > >         if (is_mmio)
> > > > > > > > >                 ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > > > > 
> > > > > > > > > ...
> > > > > > > > > }
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > > > host will override this and use uncacheable in practice?
> > > > > > > > > 
> > > > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > > > in SDM
> > > > > > > 
> > > > > > > 
> > > > > > > Can you quote chapter name please?
> > > > > > > My SDM has
> > > > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > > > > 
> > > > > > Either your or mine is out of date:
> > > > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > > > > 
> > > > > OK this one I'm familiar with, it describes how PAT
> > > > > interacts with MTRR. But how does this interact with EPT?
> > > > > do you remember where's that described?
> > > > 
> > > > Found it 28.2.4 EPT and Memory Typing
> > > > sorry about the noise.
> > > 
> > > 
> > > OK and that's explicit:
> > > If CR0.CD = 0, the effective memory type depends upon the value of bit
> > > 6 of the last EPT paging-structure entry:
> > > — If the value is 0, the effective memory type is the combination of the
> > > EPT memory type and the PAT memory type specified in Table 11-7 in Section
> > > 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
> > > 
> > > — If the value is 1, the memory type used for the access is the EPT
> > > memory
> > > type. The PAT memory type is ignored.
> > > 
> > > If CR0.CD = 1, the effective memory type is UC.
> > > 
> > > So it's simple. EPT replaces guest's MTRR.
> > 
> > 
> > And that in turn means that since we set UC in EPT,
> > VCPU will always work as if it's UC except for
> > guests using WC - WC takes precedence.
> > 
> That's what I wrote below, no? :)

Yes I just added that this applies to other memory types
like WT (that probably nobody uses?).

> > 
> > Thanks!
> > > > 
> > > > > > > > on how effective memory type is calculated.
> > > > > > > > 
> > > > > > > >  Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > > > >  (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > > > >  MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > > > >  be by default) combined memory type will be UC, so also fine. If guest
> > > > > > > >  maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > > > >  type will be incorrect in a guest,
> > > > > > > 
> > > > > > > Meaning  MTRR in guest is ignored in this case?
> > > > > > Yes.
> > > > > > 
> > > > > > > 
> > > > > > > > but I found only one place that does
> > > > > > > >  it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > > > >  either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > > > > 
> > > > > > > > --
> > > > > > > > 			Gleb.
> > > > > > 
> > > > > > --
> > > > > > 			Gleb.
> 
> --
> 			Gleb.