* guest assigned device MMIO maps with WC: does this work correctly?
@ 2013-11-20 19:58 Michael S. Tsirkin
2013-11-21 7:18 ` Gleb Natapov
0 siblings, 1 reply; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-20 19:58 UTC (permalink / raw)
To: kvm, gleb
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
does this mean that even if guest maps BAR for an assigned device
as write combined (or configures such using an MTRR),
host will override this and use uncacheable in practice?
I think there are some drivers that map MMIO as write-combined
for performance (but don't have such hardware) so
I'd like to figure out from code whether this will work correctly.
--
MST
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-20 19:58 guest assigned device MMIO maps with WC: does this work correctly? Michael S. Tsirkin
@ 2013-11-21 7:18 ` Gleb Natapov
2013-11-21 8:02 ` Michael S. Tsirkin
0 siblings, 1 reply; 11+ messages in thread
From: Gleb Natapov @ 2013-11-21 7:18 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: kvm
On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> I see this in kvm:
>
> static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> is_mmio)
> {
> u64 ret;
>
> /* For VT-d and EPT combination
> * 1. MMIO: always map as UC
> * 2. EPT with VT-d:
> * a. VT-d without snooping control feature: can't guarantee
> * the
> * result, try to trust guest.
> * b. VT-d with snooping control feature: snooping control
> * feature of
> * VT-d engine can guarantee the cache correctness. Just
> * set it
> * to WB to keep consistent with host. So the same as item
> * 3.
> * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> * consistent with host MTRR
> */
> if (is_mmio)
> ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
>
> ...
> }
>
>
> does this mean that even if guest maps BAR for an assigned device
> as write combined (or configures such using an MTRR),
> host will override this and use uncacheable in practice?
>
No, it does not mean that. I already answered this once (my previous
answer included below): effective memory type is a combination of MTRR
(EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
in SDM on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
(that what ioremap_wc does), everything works as it should. If guest maps
MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
be by default) combined memory type will be UC, so also fine. If guest
maps MMIO range as WB and fixes mtrr for this region to be WB then memory
type will be incorrect in a guest, but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:02 ` Michael S. Tsirkin
@ 2013-11-21 8:01 ` Gleb Natapov
2013-11-21 8:16 ` Michael S. Tsirkin
0 siblings, 1 reply; 11+ messages in thread
From: Gleb Natapov @ 2013-11-21 8:01 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: kvm
On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > I see this in kvm:
> > >
> > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > is_mmio)
> > > {
> > > u64 ret;
> > >
> > > /* For VT-d and EPT combination
> > > * 1. MMIO: always map as UC
> > > * 2. EPT with VT-d:
> > > * a. VT-d without snooping control feature: can't guarantee
> > > * the
> > > * result, try to trust guest.
> > > * b. VT-d with snooping control feature: snooping control
> > > * feature of
> > > * VT-d engine can guarantee the cache correctness. Just
> > > * set it
> > > * to WB to keep consistent with host. So the same as item
> > > * 3.
> > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > * consistent with host MTRR
> > > */
> > > if (is_mmio)
> > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > >
> > > ...
> > > }
> > >
> > >
> > > does this mean that even if guest maps BAR for an assigned device
> > > as write combined (or configures such using an MTRR),
> > > host will override this and use uncacheable in practice?
> > >
> > No, it does not mean that. I already answered this once (my previous
> > answer included below): effective memory type is a combination of MTRR
> > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > in SDM
>
>
> Can you quote chapter name please?
> My SDM has
> 11.5.2.2 Denormal-Operand Exception (#D)
>
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families
> > on how effective memory type is calculated.
> >
> > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > (that what ioremap_wc does), everything works as it should. If guest maps
> > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > be by default) combined memory type will be UC, so also fine. If guest
> > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > type will be incorrect in a guest,
>
> Meaning MTRR in guest is ignored in this case?
Yes.
>
> > but I found only one place that does
> > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > either remap RAM or used to get whatever memory type configured in MTRR.
> >
> > --
> > Gleb.
--
Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 7:18 ` Gleb Natapov
@ 2013-11-21 8:02 ` Michael S. Tsirkin
2013-11-21 8:01 ` Gleb Natapov
0 siblings, 1 reply; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-21 8:02 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > I see this in kvm:
> >
> > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > is_mmio)
> > {
> > u64 ret;
> >
> > /* For VT-d and EPT combination
> > * 1. MMIO: always map as UC
> > * 2. EPT with VT-d:
> > * a. VT-d without snooping control feature: can't guarantee
> > * the
> > * result, try to trust guest.
> > * b. VT-d with snooping control feature: snooping control
> > * feature of
> > * VT-d engine can guarantee the cache correctness. Just
> > * set it
> > * to WB to keep consistent with host. So the same as item
> > * 3.
> > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > * consistent with host MTRR
> > */
> > if (is_mmio)
> > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> >
> > ...
> > }
> >
> >
> > does this mean that even if guest maps BAR for an assigned device
> > as write combined (or configures such using an MTRR),
> > host will override this and use uncacheable in practice?
> >
> No, it does not mean that. I already answered this once (my previous
> answer included below): effective memory type is a combination of MTRR
> (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> in SDM
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)
> on how effective memory type is calculated.
>
> Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> (that what ioremap_wc does), everything works as it should. If guest maps
> MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> be by default) combined memory type will be UC, so also fine. If guest
> maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
> but I found only one place that does
> it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> either remap RAM or used to get whatever memory type configured in MTRR.
>
> --
> Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:01 ` Gleb Natapov
@ 2013-11-21 8:16 ` Michael S. Tsirkin
2013-11-21 8:16 ` Gleb Natapov
2013-11-21 8:18 ` Michael S. Tsirkin
0 siblings, 2 replies; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-21 8:16 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > I see this in kvm:
> > > >
> > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > is_mmio)
> > > > {
> > > > u64 ret;
> > > >
> > > > /* For VT-d and EPT combination
> > > > * 1. MMIO: always map as UC
> > > > * 2. EPT with VT-d:
> > > > * a. VT-d without snooping control feature: can't guarantee
> > > > * the
> > > > * result, try to trust guest.
> > > > * b. VT-d with snooping control feature: snooping control
> > > > * feature of
> > > > * VT-d engine can guarantee the cache correctness. Just
> > > > * set it
> > > > * to WB to keep consistent with host. So the same as item
> > > > * 3.
> > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > * consistent with host MTRR
> > > > */
> > > > if (is_mmio)
> > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > >
> > > > ...
> > > > }
> > > >
> > > >
> > > > does this mean that even if guest maps BAR for an assigned device
> > > > as write combined (or configures such using an MTRR),
> > > > host will override this and use uncacheable in practice?
> > > >
> > > No, it does not mean that. I already answered this once (my previous
> > > answer included below): effective memory type is a combination of MTRR
> > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > in SDM
> >
> >
> > Can you quote chapter name please?
> > My SDM has
> > 11.5.2.2 Denormal-Operand Exception (#D)
> >
> Either your or mine is out of date:
> Selecting Memory Types for Pentium III and More Recent Processor Families
OK this one I'm familiar with, it describes how PAT
interacts with MTRR. But how does this interact with EPT?
do you remember where's that described?
> > > on how effective memory type is calculated.
> > >
> > > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > (that what ioremap_wc does), everything works as it should. If guest maps
> > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > be by default) combined memory type will be UC, so also fine. If guest
> > > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > type will be incorrect in a guest,
> >
> > Meaning MTRR in guest is ignored in this case?
> Yes.
>
> >
> > > but I found only one place that does
> > > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > either remap RAM or used to get whatever memory type configured in MTRR.
> > >
> > > --
> > > Gleb.
>
> --
> Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:16 ` Michael S. Tsirkin
@ 2013-11-21 8:16 ` Gleb Natapov
2013-11-21 8:18 ` Michael S. Tsirkin
1 sibling, 0 replies; 11+ messages in thread
From: Gleb Natapov @ 2013-11-21 8:16 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: kvm
On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > I see this in kvm:
> > > > >
> > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > is_mmio)
> > > > > {
> > > > > u64 ret;
> > > > >
> > > > > /* For VT-d and EPT combination
> > > > > * 1. MMIO: always map as UC
> > > > > * 2. EPT with VT-d:
> > > > > * a. VT-d without snooping control feature: can't guarantee
> > > > > * the
> > > > > * result, try to trust guest.
> > > > > * b. VT-d with snooping control feature: snooping control
> > > > > * feature of
> > > > > * VT-d engine can guarantee the cache correctness. Just
> > > > > * set it
> > > > > * to WB to keep consistent with host. So the same as item
> > > > > * 3.
> > > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > * consistent with host MTRR
> > > > > */
> > > > > if (is_mmio)
> > > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > >
> > > > > ...
> > > > > }
> > > > >
> > > > >
> > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > as write combined (or configures such using an MTRR),
> > > > > host will override this and use uncacheable in practice?
> > > > >
> > > > No, it does not mean that. I already answered this once (my previous
> > > > answer included below): effective memory type is a combination of MTRR
> > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > in SDM
> > >
> > >
> > > Can you quote chapter name please?
> > > My SDM has
> > > 11.5.2.2 Denormal-Operand Exception (#D)
> > >
> > Either your or mine is out of date:
> > Selecting Memory Types for Pentium III and More Recent Processor Families
>
> OK this one I'm familiar with, it describes how PAT
> interacts with MTRR. But how does this interact with EPT?
> do you remember where's that described?
>
28.2.5.2 Memory Type Used for Translated Guest-Physical Addresses
Substitute MTRR with EPT MT and go to the same table.
--
Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:16 ` Michael S. Tsirkin
2013-11-21 8:16 ` Gleb Natapov
@ 2013-11-21 8:18 ` Michael S. Tsirkin
2013-11-21 8:21 ` Michael S. Tsirkin
1 sibling, 1 reply; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-21 8:18 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > I see this in kvm:
> > > > >
> > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > is_mmio)
> > > > > {
> > > > > u64 ret;
> > > > >
> > > > > /* For VT-d and EPT combination
> > > > > * 1. MMIO: always map as UC
> > > > > * 2. EPT with VT-d:
> > > > > * a. VT-d without snooping control feature: can't guarantee
> > > > > * the
> > > > > * result, try to trust guest.
> > > > > * b. VT-d with snooping control feature: snooping control
> > > > > * feature of
> > > > > * VT-d engine can guarantee the cache correctness. Just
> > > > > * set it
> > > > > * to WB to keep consistent with host. So the same as item
> > > > > * 3.
> > > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > * consistent with host MTRR
> > > > > */
> > > > > if (is_mmio)
> > > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > >
> > > > > ...
> > > > > }
> > > > >
> > > > >
> > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > as write combined (or configures such using an MTRR),
> > > > > host will override this and use uncacheable in practice?
> > > > >
> > > > No, it does not mean that. I already answered this once (my previous
> > > > answer included below): effective memory type is a combination of MTRR
> > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > in SDM
> > >
> > >
> > > Can you quote chapter name please?
> > > My SDM has
> > > 11.5.2.2 Denormal-Operand Exception (#D)
> > >
> > Either your or mine is out of date:
> > Selecting Memory Types for Pentium III and More Recent Processor Families
>
> OK this one I'm familiar with, it describes how PAT
> interacts with MTRR. But how does this interact with EPT?
> do you remember where's that described?
Found it 28.2.4 EPT and Memory Typing
sorry about the noise.
> > > > on how effective memory type is calculated.
> > > >
> > > > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > (that what ioremap_wc does), everything works as it should. If guest maps
> > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > be by default) combined memory type will be UC, so also fine. If guest
> > > > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > type will be incorrect in a guest,
> > >
> > > Meaning MTRR in guest is ignored in this case?
> > Yes.
> >
> > >
> > > > but I found only one place that does
> > > > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > either remap RAM or used to get whatever memory type configured in MTRR.
> > > >
> > > > --
> > > > Gleb.
> >
> > --
> > Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:18 ` Michael S. Tsirkin
@ 2013-11-21 8:21 ` Michael S. Tsirkin
2013-11-21 8:25 ` Michael S. Tsirkin
0 siblings, 1 reply; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-21 8:21 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > I see this in kvm:
> > > > > >
> > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > is_mmio)
> > > > > > {
> > > > > > u64 ret;
> > > > > >
> > > > > > /* For VT-d and EPT combination
> > > > > > * 1. MMIO: always map as UC
> > > > > > * 2. EPT with VT-d:
> > > > > > * a. VT-d without snooping control feature: can't guarantee
> > > > > > * the
> > > > > > * result, try to trust guest.
> > > > > > * b. VT-d with snooping control feature: snooping control
> > > > > > * feature of
> > > > > > * VT-d engine can guarantee the cache correctness. Just
> > > > > > * set it
> > > > > > * to WB to keep consistent with host. So the same as item
> > > > > > * 3.
> > > > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > * consistent with host MTRR
> > > > > > */
> > > > > > if (is_mmio)
> > > > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > >
> > > > > > ...
> > > > > > }
> > > > > >
> > > > > >
> > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > as write combined (or configures such using an MTRR),
> > > > > > host will override this and use uncacheable in practice?
> > > > > >
> > > > > No, it does not mean that. I already answered this once (my previous
> > > > > answer included below): effective memory type is a combination of MTRR
> > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > in SDM
> > > >
> > > >
> > > > Can you quote chapter name please?
> > > > My SDM has
> > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > >
> > > Either your or mine is out of date:
> > > Selecting Memory Types for Pentium III and More Recent Processor Families
> >
> > OK this one I'm familiar with, it describes how PAT
> > interacts with MTRR. But how does this interact with EPT?
> > do you remember where's that described?
>
> Found it 28.2.4 EPT and Memory Typing
> sorry about the noise.
OK and that's explicit:
If CR0.CD = 0, the effective memory type depends upon the value of bit
6 of the last EPT paging-structure entry:
— If the value is 0, the effective memory type is the combination of the
EPT memory type and the PAT memory type specified in Table 11-7 in Section
11.5.2.2, using the EPT memory type in place of the MTRR memory type.
— If the value is 1, the memory type used for the access is the EPT
memory
type. The PAT memory type is ignored.
If CR0.CD = 1, the effective memory type is UC.
So it's simple. EPT replaces guest's MTRR.
>
> > > > > on how effective memory type is calculated.
> > > > >
> > > > > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > be by default) combined memory type will be UC, so also fine. If guest
> > > > > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > type will be incorrect in a guest,
> > > >
> > > > Meaning MTRR in guest is ignored in this case?
> > > Yes.
> > >
> > > >
> > > > > but I found only one place that does
> > > > > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > either remap RAM or used to get whatever memory type configured in MTRR.
> > > > >
> > > > > --
> > > > > Gleb.
> > >
> > > --
> > > Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:21 ` Michael S. Tsirkin
@ 2013-11-21 8:25 ` Michael S. Tsirkin
2013-11-21 8:25 ` Gleb Natapov
0 siblings, 1 reply; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-21 8:25 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > I see this in kvm:
> > > > > > >
> > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > is_mmio)
> > > > > > > {
> > > > > > > u64 ret;
> > > > > > >
> > > > > > > /* For VT-d and EPT combination
> > > > > > > * 1. MMIO: always map as UC
> > > > > > > * 2. EPT with VT-d:
> > > > > > > * a. VT-d without snooping control feature: can't guarantee
> > > > > > > * the
> > > > > > > * result, try to trust guest.
> > > > > > > * b. VT-d with snooping control feature: snooping control
> > > > > > > * feature of
> > > > > > > * VT-d engine can guarantee the cache correctness. Just
> > > > > > > * set it
> > > > > > > * to WB to keep consistent with host. So the same as item
> > > > > > > * 3.
> > > > > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > > * consistent with host MTRR
> > > > > > > */
> > > > > > > if (is_mmio)
> > > > > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > >
> > > > > > > ...
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > host will override this and use uncacheable in practice?
> > > > > > >
> > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > in SDM
> > > > >
> > > > >
> > > > > Can you quote chapter name please?
> > > > > My SDM has
> > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > >
> > > > Either your or mine is out of date:
> > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > >
> > > OK this one I'm familiar with, it describes how PAT
> > > interacts with MTRR. But how does this interact with EPT?
> > > do you remember where's that described?
> >
> > Found it 28.2.4 EPT and Memory Typing
> > sorry about the noise.
>
>
> OK and that's explicit:
> If CR0.CD = 0, the effective memory type depends upon the value of bit
> 6 of the last EPT paging-structure entry:
> — If the value is 0, the effective memory type is the combination of the
> EPT memory type and the PAT memory type specified in Table 11-7 in Section
> 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
>
> — If the value is 1, the memory type used for the access is the EPT
> memory
> type. The PAT memory type is ignored.
>
> If CR0.CD = 1, the effective memory type is UC.
>
> So it's simple. EPT replaces guest's MTRR.
And that in turn means that since we set UC in EPT,
VCPU will always work as if it's UC except for
guests using WC - WC takes precedence.
Thanks!
> >
> > > > > > on how effective memory type is calculated.
> > > > > >
> > > > > > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > > (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > > be by default) combined memory type will be UC, so also fine. If guest
> > > > > > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > > type will be incorrect in a guest,
> > > > >
> > > > > Meaning MTRR in guest is ignored in this case?
> > > > Yes.
> > > >
> > > > >
> > > > > > but I found only one place that does
> > > > > > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > > either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > >
> > > > > > --
> > > > > > Gleb.
> > > >
> > > > --
> > > > Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:25 ` Michael S. Tsirkin
@ 2013-11-21 8:25 ` Gleb Natapov
2013-11-21 8:34 ` Michael S. Tsirkin
0 siblings, 1 reply; 11+ messages in thread
From: Gleb Natapov @ 2013-11-21 8:25 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: kvm
On Thu, Nov 21, 2013 at 10:25:34AM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > > I see this in kvm:
> > > > > > > >
> > > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > > is_mmio)
> > > > > > > > {
> > > > > > > > u64 ret;
> > > > > > > >
> > > > > > > > /* For VT-d and EPT combination
> > > > > > > > * 1. MMIO: always map as UC
> > > > > > > > * 2. EPT with VT-d:
> > > > > > > > * a. VT-d without snooping control feature: can't guarantee
> > > > > > > > * the
> > > > > > > > * result, try to trust guest.
> > > > > > > > * b. VT-d with snooping control feature: snooping control
> > > > > > > > * feature of
> > > > > > > > * VT-d engine can guarantee the cache correctness. Just
> > > > > > > > * set it
> > > > > > > > * to WB to keep consistent with host. So the same as item
> > > > > > > > * 3.
> > > > > > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > > > * consistent with host MTRR
> > > > > > > > */
> > > > > > > > if (is_mmio)
> > > > > > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > > >
> > > > > > > > ...
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > > host will override this and use uncacheable in practice?
> > > > > > > >
> > > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > > in SDM
> > > > > >
> > > > > >
> > > > > > Can you quote chapter name please?
> > > > > > My SDM has
> > > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > > >
> > > > > Either your or mine is out of date:
> > > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > > >
> > > > OK this one I'm familiar with, it describes how PAT
> > > > interacts with MTRR. But how does this interact with EPT?
> > > > do you remember where's that described?
> > >
> > > Found it 28.2.4 EPT and Memory Typing
> > > sorry about the noise.
> >
> >
> > OK and that's explicit:
> > If CR0.CD = 0, the effective memory type depends upon the value of bit
> > 6 of the last EPT paging-structure entry:
> > — If the value is 0, the effective memory type is the combination of the
> > EPT memory type and the PAT memory type specified in Table 11-7 in Section
> > 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
> >
> > — If the value is 1, the memory type used for the access is the EPT
> > memory
> > type. The PAT memory type is ignored.
> >
> > If CR0.CD = 1, the effective memory type is UC.
> >
> > So it's simple. EPT replaces guest's MTRR.
>
>
> And that in turn means that since we set UC in EPT,
> VCPU will always work as if it's UC except for
> guests using WC - WC takes precedence.
>
That's what I wrote below, no? :)
>
> Thanks!
> > >
> > > > > > > on how effective memory type is calculated.
> > > > > > >
> > > > > > > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > > > (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > > > be by default) combined memory type will be UC, so also fine. If guest
> > > > > > > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > > > type will be incorrect in a guest,
> > > > > >
> > > > > > Meaning MTRR in guest is ignored in this case?
> > > > > Yes.
> > > > >
> > > > > >
> > > > > > > but I found only one place that does
> > > > > > > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > > > either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > > >
> > > > > > > --
> > > > > > > Gleb.
> > > > >
> > > > > --
> > > > > Gleb.
--
Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guest assigned device MMIO maps with WC: does this work correctly?
2013-11-21 8:25 ` Gleb Natapov
@ 2013-11-21 8:34 ` Michael S. Tsirkin
0 siblings, 0 replies; 11+ messages in thread
From: Michael S. Tsirkin @ 2013-11-21 8:34 UTC (permalink / raw)
To: Gleb Natapov; +Cc: kvm
On Thu, Nov 21, 2013 at 10:25:52AM +0200, Gleb Natapov wrote:
> On Thu, Nov 21, 2013 at 10:25:34AM +0200, Michael S. Tsirkin wrote:
> > On Thu, Nov 21, 2013 at 10:21:33AM +0200, Michael S. Tsirkin wrote:
> > > On Thu, Nov 21, 2013 at 10:18:34AM +0200, Michael S. Tsirkin wrote:
> > > > On Thu, Nov 21, 2013 at 10:16:27AM +0200, Michael S. Tsirkin wrote:
> > > > > On Thu, Nov 21, 2013 at 10:01:27AM +0200, Gleb Natapov wrote:
> > > > > > On Thu, Nov 21, 2013 at 10:02:07AM +0200, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Nov 21, 2013 at 09:18:55AM +0200, Gleb Natapov wrote:
> > > > > > > > On Wed, Nov 20, 2013 at 09:58:15PM +0200, Michael S. Tsirkin wrote:
> > > > > > > > > I see this in kvm:
> > > > > > > > >
> > > > > > > > > static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool
> > > > > > > > > is_mmio)
> > > > > > > > > {
> > > > > > > > > u64 ret;
> > > > > > > > >
> > > > > > > > > /* For VT-d and EPT combination
> > > > > > > > > * 1. MMIO: always map as UC
> > > > > > > > > * 2. EPT with VT-d:
> > > > > > > > > * a. VT-d without snooping control feature: can't guarantee
> > > > > > > > > * the
> > > > > > > > > * result, try to trust guest.
> > > > > > > > > * b. VT-d with snooping control feature: snooping control
> > > > > > > > > * feature of
> > > > > > > > > * VT-d engine can guarantee the cache correctness. Just
> > > > > > > > > * set it
> > > > > > > > > * to WB to keep consistent with host. So the same as item
> > > > > > > > > * 3.
> > > > > > > > > * 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
> > > > > > > > > * consistent with host MTRR
> > > > > > > > > */
> > > > > > > > > if (is_mmio)
> > > > > > > > > ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
> > > > > > > > >
> > > > > > > > > ...
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > does this mean that even if guest maps BAR for an assigned device
> > > > > > > > > as write combined (or configures such using an MTRR),
> > > > > > > > > host will override this and use uncacheable in practice?
> > > > > > > > >
> > > > > > > > No, it does not mean that. I already answered this once (my previous
> > > > > > > > answer included below): effective memory type is a combination of MTRR
> > > > > > > > (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2
> > > > > > > > in SDM
> > > > > > >
> > > > > > >
> > > > > > > Can you quote chapter name please?
> > > > > > > My SDM has
> > > > > > > 11.5.2.2 Denormal-Operand Exception (#D)
> > > > > > >
> > > > > > Either your or mine is out of date:
> > > > > > Selecting Memory Types for Pentium III and More Recent Processor Families
> > > > >
> > > > > OK this one I'm familiar with, it describes how PAT
> > > > > interacts with MTRR. But how does this interact with EPT?
> > > > > do you remember where's that described?
> > > >
> > > > Found it 28.2.4 EPT and Memory Typing
> > > > sorry about the noise.
> > >
> > >
> > > OK and that's explicit:
> > > If CR0.CD = 0, the effective memory type depends upon the value of bit
> > > 6 of the last EPT paging-structure entry:
> > > — If the value is 0, the effective memory type is the combination of the
> > > EPT memory type and the PAT memory type specified in Table 11-7 in Section
> > > 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
> > >
> > > — If the value is 1, the memory type used for the access is the EPT
> > > memory
> > > type. The PAT memory type is ignored.
> > >
> > > If CR0.CD = 1, the effective memory type is UC.
> > >
> > > So it's simple. EPT replaces guest's MTRR.
> >
> >
> > And that in turn means that since we set UC in EPT,
> > VCPU will always work as if it's UC except for
> > guests using WC - WC takes precedence.
> >
> That's what I wrote below, no? :)
Yes I just added that this applies to other memory types
like WT (that probably nobody uses?).
> >
> > Thanks!
> > > >
> > > > > > > > on how effective memory type is calculated.
> > > > > > > >
> > > > > > > > Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table
> > > > > > > > (that what ioremap_wc does), everything works as it should. If guest maps
> > > > > > > > MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will
> > > > > > > > be by default) combined memory type will be UC, so also fine. If guest
> > > > > > > > maps MMIO range as WB and fixes mtrr for this region to be WB then memory
> > > > > > > > type will be incorrect in a guest,
> > > > > > >
> > > > > > > Meaning MTRR in guest is ignored in this case?
> > > > > > Yes.
> > > > > >
> > > > > > >
> > > > > > > > but I found only one place that does
> > > > > > > > it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache
> > > > > > > > either remap RAM or used to get whatever memory type configured in MTRR.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Gleb.
> > > > > >
> > > > > > --
> > > > > > Gleb.
>
> --
> Gleb.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2013-11-21 8:30 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-11-20 19:58 guest assigned device MMIO maps with WC: does this work correctly? Michael S. Tsirkin
2013-11-21 7:18 ` Gleb Natapov
2013-11-21 8:02 ` Michael S. Tsirkin
2013-11-21 8:01 ` Gleb Natapov
2013-11-21 8:16 ` Michael S. Tsirkin
2013-11-21 8:16 ` Gleb Natapov
2013-11-21 8:18 ` Michael S. Tsirkin
2013-11-21 8:21 ` Michael S. Tsirkin
2013-11-21 8:25 ` Michael S. Tsirkin
2013-11-21 8:25 ` Gleb Natapov
2013-11-21 8:34 ` Michael S. Tsirkin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox