Re: Xen's use of PAT and PV guests

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Mar 29, 2010 at 05:35:57PM -0700, Jeremy Fitzhardinge wrote:
> I'm looking again at what it will take to reconcile Xen's PAT setup with  
> the standard Linux one so that we can enable PAT use in pvops kernels.
>
> Just for reference, this is the Linux vs Xen vs default PAT setups:

And this LKML is good a primer:

http://www.linuxsymposium.org/archives/OLS/Reprints-2008/pallipadi-reprint.pdf

>
> Index	PTE flags	Linux	Xen	Default
> 0			WB	WB	WB
> 1	        PWT	WC	WT	WT
> 2	    PCD	   	UC-	UC-	UC-
> 3	    PCD PWT	UC	UC	UC
> 4	PAT        	WB	WC	WB
> 5	PAT     PWT	WC	WP	WT
> 6	PAT PCD	   	UC-	UC	UC-
> 7	PAT PCD PWT	UC	UC	UC
>
>
> Originally I was thinking of a moderately complex scheme in which an ELF  
> node on the dom0 kernel could determine the system-wide Xen PAT MSR, and  
> then the kernel ELF notes on subsequent domains would determine whether  
> the PAT CPU feature flag is enabled or not.
>
> However this has several problems:
>
>   1. it is fairly complex
>   2. if dom0 sets the PAT configuration to something strange, it may
>      completely break other PV guests entirely (since it might
>      effectively change the meaning of PCD+PWT globally)

How does this work on pages shared across domains? Say Guest A makes the
page WC,Dom0 makes it WB and Xen puts it in WC, and Dom0 reads does a
Write/Read/Write, but in actuallity it is a Read/Write/Write. Or is
there no danger there since the grant table pages have UC set on them?

>   3. disabling the PAT CPU feature flag is meaningless, as its only
>      effect is to say "there's no PAT, so PCD/PWT have their default
>      behaviours", which is definitely not true in general
>
> Linux only uses the first 4 PAT entries, and repeats it, effectively  
> making the PAT pte flag a don't-care.  In those 4 entries, the Linux,  
> Xen and Default configurations are identical aside from Linux using WC  
> rather than WT.
>
> It therefore seems to me that if I make Linux:
>
>   1. never set the PAT flag (which it won't anyway),
>   2. check that the value written to IA32_PAT is as expected, but
>      otherwise ignore it, and
>   3. use WT rather than WC

That would make all writes synchronous. Why not write back?
>
> then it all should just work.  I'm not completely confident in the third  
> point though, since I'm not quite sure about the full set of differences  
> between WT and WC, and their respective interactions with the MTRR, and  
> whether that would break anything.  At first glance it seems pretty safe  
> though...

The graphics cards (and the XServer) are the ones that come in my mind
as heavy users of having this "just right".  But in most (all?) cases
they want it to be UC or better UC- so that shouldn't affect this.

http://lkml.indiana.edu/hypermail/linux/kernel/9904.1/0306.html

Ah, but of course, there is an exception:

(i915_dma.c): gtt = ioremap_wc(pci_resource_start(dev->pdev, gtt_bar) + gtt_offset, gtt_size);

and then 'ttm_bo_ioremap' which does the ioremap_wc if  TTM_PL_FLAG_WC is set.

And it looks to be is set by the Radeon (if card is an AGP) and nouveau
on their first memory BAR.  Also the vmwgfx (VMWare) driver sets this, but we don't have
to worry about that.


So I think setting it to WC->WT would mean that graphics performance
would go down the drain. But then, there are some lingering issues with the
TTM/DRM infrastructure that need to tracked down and I believe 
Arvind and Michael are actively looking at that.