Re: [PATCH v2 1/5] arm/xen: define xen_remap as ioremap_cached

[Catalin Marinas <catalin.marinas@arm.com>]

On Tue, Jun 04, 2013 at 10:20:50AM +0100, Ian Campbell wrote:
> On Mon, 2013-06-03 at 16:33 +0100, Stefano Stabellini wrote:
> > Define xen_remap as ioremap_cache (MT_MEMORY and MT_DEVICE_CACHED end up
> > having the same AttrIndx encoding).
> 
> The entries in static struct mem_type mem_types[] look entirely
> different to me.  What am I missing?
> 	[MT_DEVICE_CACHED] = {	  /* ioremap_cached */
> 		.prot_pte	= PROT_PTE_DEVICE | L_PTE_MT_DEV_CACHED,
> 		.prot_l1	= PMD_TYPE_TABLE,
> 		.prot_sect	= PROT_SECT_DEVICE | PMD_SECT_WB,
> 		.domain		= DOMAIN_IO,
> 	},
> 	[MT_MEMORY] = {
> 		.prot_pte  = L_PTE_PRESENT | L_PTE_YOUNG | L_PTE_DIRTY,
> 		.prot_l1   = PMD_TYPE_TABLE,
> 		.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE,
> 		.domain    = DOMAIN_KERNEL,
> 	},
> 
> I can see in pgtable-3level.h how L_PTE_MT_DEV_CACHED and
> L_PTE_MT_WRITEBACK are the same but not where the MT_WRITEBACK comes
> from for MT_MEMORY. Things are less clear in pgtable-2level.h, where one
> is 0x3 and the other is 0xb. I can see that the entries are the same in
> armv6_mt_table but how that would apply to a v7 processor?

PROT_PTE_DEVICE and PROT_SECT_DEVICE above don't contain any memory type
information, just attributes/permission - present, young, dirty and XN:

#define PROT_PTE_DEVICE		L_PTE_PRESENT|L_PTE_YOUNG|L_PTE_DIRTY|L_PTE_XN
#define PROT_SECT_DEVICE	PMD_TYPE_SECT|PMD_SECT_AP_WRITE

The memory type is given by the L_PTE_MT_DEV_CACHED and PMD_SECT_WB
macros. Let's take prot_sect first as it's simpler. For MT_DEVICE_CACHED
we have:

.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_SECT_WB

For MT_MEMORY we have:

.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE

The cache policy is added later to MT_MEMORY which is either WB or WBWA
(based on SMP, no particular reason as these are just processor hints;
for some historical reasons we enabled WBWA for ARM11MPCore but we could
leave it on all the time).

Similarly for prot_pte, present, young, dirty are the same.

Regarding the type, on ARMv7 (with or without LPAE) we use TEX remapping
and L_PTE_MT_DEVICE has the same index (3-bit TEX[0], C, B for NMRR/PRRR
or TEX[2:0] for MAIR0/MAIR1 registers) as Normal Cacheable Writeback
memory (there is no such thing as Device memory with cacheability
attributes, only Normal Cacheable memory).

We have XN in addition for MT_DEVICE_CACHED to prevent speculative
instruction fetches. However, you still get speculative D-cache line
fills since the memory is Normal Cacheable.

> Anyhow, even if I'm prepared to believe that MT_MEMORY and
> MT_DEVICE_CACHED end up being the same thing (which TBH I am) it seems
> that the level of abstraction involved makes us vulnerable to future
> changes subtly breaking things for us. What about:
> 
>         /* Device shared memory */
>         #define ioremap_shm(cookie,size)		__arm_ioremap((cookie), (size), MT_MEMORY)

For my understanding, what is Xen doing with such mapping? I would
rather make ioremap_cached() use MT_MEMORY on ARMv6/v7 (or make it
closer to that, I'm not sure about the implications on ARMv5 and earlier
but for newer architectures I don't see the point in pretending to have
Cacheable Device memory). That's however for Russell to comment.

-- 
Catalin