Cache alias handling in copy_user_page / copy_user_highpage

Cache alias handling in copy_user_page / copy_user_highpage

[Ralf Baechle]

The problem:

There is a dcache aliasing problem on preempt kernel (or SMP kernel,
perhaps) when a multi-threaded program calls fork().

1. Now there is a process containing two thread (T1 and T2).  The
   thread T1 call fork().  dup_mmap() function called on T1 context.

static inline int dup_mmap(struct mm_struct * mm, struct mm_struct * oldmm)
{
	...
	flush_cache_mm(current->mm);
	/* A */
	...
	(write-protect all Copy-On-Write pages)
	...
	/* B */
	flush_tlb_mm(current->mm);
	...
}

2. When preemption happens between A and B (or on SMP kernel), the
   thread T2 can run and modify data on COW pages without page fault
   (modified data will stay in cache).

3. Some time after fork() completed, the thread T2 may cause page
   fault by write-protect on COW pages .

4. Then data of the COW page will be copied to newly allocated
   physical page in cow_user_page() which calls copy_user_highpage which
   in turn calls copy_user_page.  copy_user_highpage is mapping source and
   destination page using kmap_atomic(), then calls copy_user_page to
   perform the actual copy operation.
   The kernel mapping can have a different 'color' than the user space
   mapping of thread T2, resulting in dcache aliasing.  Therefore
   copy_cow_page() will copy stale data.  Then the modified data in cache
   will be lost.

A common implementation of copy_user_page performs a copy_page and a
cacheflush operation and this implementation is suspect to above problem.
I know MIPS and PARISC suffer from the problem.  ARM and m68k look suspect
to me at which point I stopped looking at further architectures.

The only sane solution I can see is setting up temporary kernel mappings
for source and destination pages with virtual addresses choosen to be
congruent to userspace addresses.  cachetlb.txt's wording describes this
as if it was an optional fancy implementation.

Now that I've described the problem, the last thing that is left to
explain is why copy_user_page is an insufficient interface, that's the
most trivial one.  copy_user_highpage() uses kmap_atomic which will
create atomic mappings without consideration for cache aliasing.  So
copy_user_page() will be invoked with the virtual address of these
mappings and the struct page pointer for the destination page.  on a
machine with aliases copy_user_page has no choice but to ignore the
destination address and map the pag itself again - wasted time.  And it
doesn't receive a struct page * for the source page ...

So here are two alterantive fixes for the API:

1) Move copy_user_highpage into the arch code.  This will allow the arch code
to create any sort of crazy mapping it wants.  The old copy_user_page as
kernel API is only called from copy_user_highpage and would therefore be
obsoleted.

2) Introduce a new kmap_atomic variant to create a congruent mapping.
copy_user_highpage would change into something like this:

static inline void copy_user_highpage(struct page *to, struct page *from,
	unsigned long vaddr)
{
        char *vfrom, *vto;

        vfrom = kmap_coherent(from, KM_USER0, vaddr);
        vto = kmap_coherent(to, KM_USER1, vaddr);
        copy_page(vto, vfrom);
        kunmap_coherent(vfrom, KM_USER0, vaddr);
        kunmap_coherent(vto, KM_USER1, vaddr);
        /* Make sure this page is cleared on other CPU's too before using it */
        smp_wmb();
}

I lean toward 2) since kmap_coherent is a more versatile interface.

clear_user_page / clear_user_highpage are not affected by the same kind of
bug but they should be changed similarly for consistency.

  Ralf

Re: Cache alias handling in copy_user_page / copy_user_highpage

[Russell King]

On Sat, Aug 19, 2006 at 01:25:20PM +0100, Ralf Baechle wrote:
> A common implementation of copy_user_page performs a copy_page and a
> cacheflush operation and this implementation is suspect to above problem.
> I know MIPS and PARISC suffer from the problem.  ARM and m68k look suspect
> to me at which point I stopped looking at further architectures.
> 
> The only sane solution I can see is setting up temporary kernel mappings
> for source and destination pages with virtual addresses choosen to be
> congruent to userspace addresses.  cachetlb.txt's wording describes this
> as if it was an optional fancy implementation.

Even this won't work for VIVT caches.  Luckily, VIVT don't feature on ARM
SMP machines that much, so the only case we have to worry about there is
the preempt one.  The immediately obvious solution to that is to disable
preemption over the critical area.  This solves the preemption race, but
I suspect the preempt people will be very unhappy because it'll increase
latency.

For ARM SMP, VIVT isn't that much of an issue - the production ARM SMP
machines are VIPT, and we already map pages in our copy_user_page function.
The kmap interface is entirely wrong for this sort of thing, and if we
ever get ARM SMP+highmem, the kmap interface would need to be changed.

In fact, on such a platform, the kmap step would be entirely redundant
given this implementation, and would only serve to waste CPU cycles.

> 1) Move copy_user_highpage into the arch code.  This will allow the arch code
> to create any sort of crazy mapping it wants.  The old copy_user_page as
> kernel API is only called from copy_user_highpage and would therefore be
> obsoleted.
> 
> 2) Introduce a new kmap_atomic variant to create a congruent mapping.
> copy_user_highpage would change into something like this:
> 
> static inline void copy_user_highpage(struct page *to, struct page *from,
> 	unsigned long vaddr)
> {
>         char *vfrom, *vto;
> 
>         vfrom = kmap_coherent(from, KM_USER0, vaddr);
>         vto = kmap_coherent(to, KM_USER1, vaddr);
>         copy_page(vto, vfrom);
>         kunmap_coherent(vfrom, KM_USER0, vaddr);
>         kunmap_coherent(vto, KM_USER1, vaddr);
>         /* Make sure this page is cleared on other CPU's too before using it */
>         smp_wmb();
> }
> 
> I lean toward 2) since kmap_coherent is a more versatile interface.

This gives me cause for concern - if we move the VIPT remapping of pages
into kmap_coherent(), it's wrong for VIVT caches, and we already build
kernels which run-time configure themselves depending on what they find.

So I'm in favour of (1), where the architecture specific details can be
properly handled by the architecture specific code, perhaps by indirecting
copy_user_highpage() via a function pointer when necessary.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: Cache alias handling in copy_user_page / copy_user_highpage

[James Bottomley]

On Sat, 2006-08-26 at 09:20 +0100, Russell King wrote:
> > I lean toward 2) since kmap_coherent is a more versatile interface.
> 
> This gives me cause for concern - if we move the VIPT remapping of pages
> into kmap_coherent(), it's wrong for VIVT caches, and we already build
> kernels which run-time configure themselves depending on what they find.
> 
> So I'm in favour of (1), where the architecture specific details can be
> properly handled by the architecture specific code, perhaps by indirecting
> copy_user_highpage() via a function pointer when necessary.

Actually, I think I favour (1) as well.  We have all the machinery in
place on parisc for copying coherently via the tmpalias space, I just
need to enable it.  While kmap_coherent would get us out of having to
take special action on kunmap (the patch I sent in previously), we just
don't have enough spare kernel mappings on 32 bit machines to implement
it.

The reason for choosing (1) over (2) is simply that we already take
special action to restore coherency on kmap/kunmap, but it would be
faster to do a coherent copy, so we have to hijack the whole interface,
because we don't want the kmap overhead if we're going to copy
coherently.

James

Re: Cache alias handling in copy_user_page / copy_user_highpage

[David Miller]

From: Ralf Baechle <ralf@linux-mips.org>
Date: Sat, 19 Aug 2006 13:25:20 +0100

> So here are two alterantive fixes for the API:
> 
> 1) Move copy_user_highpage into the arch code.  This will allow the arch code
> to create any sort of crazy mapping it wants.  The old copy_user_page as
> kernel API is only called from copy_user_highpage and would therefore be
> obsoleted.
> 
> 2) Introduce a new kmap_atomic variant to create a congruent mapping.

I think you will find that copy_user_highpage() and
clear_user_highpage() are performance critical enough
that you will want to implement it totally in hand-crafted
assembler which disables preemption, loads up the TLB
entries by hand, does the copy, and flushes out the
temp mappings.

This is essentially what the code in arch/sparc64/lib/copy_page.S and
clear_page.S is all about.

So I'd lean towards #1.  That's the reason copy_user_page() gets
a virtual address in the first place.  It's just HIGHMEM configs
that make it difficult and try to do the mapping for you.

You're definitely right that if you try to use a cache flush to
implement clear_user_page() and copy_user_page(), you will risk
data corruption.

Re: Cache alias handling in copy_user_page / copy_user_highpage

[Ralf Baechle]

Time to bring this to a closure.  Suggested patch including generic and
MIPS bits is below.

On Mon, Aug 28, 2006 at 11:36:56PM -0700, David Miller wrote:

> > 1) Move copy_user_highpage into the arch code.  This will allow the arch code
> > to create any sort of crazy mapping it wants.  The old copy_user_page as
> > kernel API is only called from copy_user_highpage and would therefore be
> > obsoleted.
> > 
> > 2) Introduce a new kmap_atomic variant to create a congruent mapping.
> 
> I think you will find that copy_user_highpage() and
> clear_user_highpage() are performance critical enough
> that you will want to implement it totally in hand-crafted
> assembler which disables preemption, loads up the TLB
> entries by hand, does the copy, and flushes out the
> temp mappings.

I found it to be much more maintainable to deal with this in C due to a
large number of processor variations to be supported.

> This is essentially what the code in arch/sparc64/lib/copy_page.S and
> clear_page.S is all about.
> 
> So I'd lean towards #1.  That's the reason copy_user_page() gets
> a virtual address in the first place.  It's just HIGHMEM configs
> that make it difficult and try to do the mapping for you.

Nobody liked #2.  James said on 32-bit PARISC setting up coherent mappings
would waste prohibitive amounts of address space.

> You're definitely right that if you try to use a cache flush to
> implement clear_user_page() and copy_user_page(), you will risk
> data corruption.

  Ralf

From: Atsushi Nemoto <anemo@mba.ocn.ne.jp>

[MIPS] Fix COW D-cache aliasing on fork

Problem:

1. There is a process containing two thread (T1 and T2).  The
   thread T1 calls fork().  Then dup_mmap() function called on T1 context.

static inline int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
	...
	flush_cache_mm(current->mm);
	...	/* A */
	(write-protect all Copy-On-Write pages)
	...	/* B */
	flush_tlb_mm(current->mm);
	...

2. When preemption happens between A and B (or on SMP kernel), the
   thread T2 can run and modify data on COW pages without page fault
   (modified data will stay in cache).

3. Some time after fork() completed, the thread T2 may cause a page
   fault by write-protect on a COW page.

4. Then data of the COW page will be copied to newly allocated
   physical page (copy_cow_page()).  It reads data via kernel mapping.
   The kernel mapping can have different 'color' with user space
   mapping of the thread T2 (dcache aliasing).  Therefore
   copy_cow_page() will copy stale data.  Then the modified data in
   cache will be lost.

This patch fixes above problems using custom copy_user_highpage().  It
uses kmap_coherent() to map an user page for kernel with same color.
Also copy_to_user_page() and copy_from_user_page() are rewritten using
the kmap_coherent() to avoid extra cache flushing.

To make copy_user_highpage() more effective, the argument "vma" is
added to the function and cow_user_page().

The main part of this patch was originally written by Ralf Baechle.

Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -30,11 +30,39 @@ #include <asm/bootinfo.h>
 #include <asm/cachectl.h>
 #include <asm/cpu.h>
 #include <asm/dma.h>
+#include <asm/kmap_types.h>
 #include <asm/mmu_context.h>
 #include <asm/sections.h>
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>
 #include <asm/tlb.h>
+#include <asm/fixmap.h>
+
+/* CP0 hazard avoidance. */
+#define BARRIER __asm__ __volatile__(".set noreorder\n\t" \
+				     "nop; nop; nop; nop; nop; nop;\n\t" \
+				     ".set reorder\n\t")
+
+/* Atomicity and interruptability */
+#ifdef CONFIG_MIPS_MT_SMTC
+
+#include <asm/mipsmtregs.h>
+
+#define ENTER_CRITICAL(flags) \
+	{ \
+	unsigned int mvpflags; \
+	local_irq_save(flags);\
+	mvpflags = dvpe()
+#define EXIT_CRITICAL(flags) \
+	evpe(mvpflags); \
+	local_irq_restore(flags); \
+	}
+#else
+
+#define ENTER_CRITICAL(flags) local_irq_save(flags)
+#define EXIT_CRITICAL(flags) local_irq_restore(flags)
+
+#endif /* CONFIG_MIPS_MT_SMTC */
 
 DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
 
@@ -80,13 +108,183 @@ unsigned long setup_zero_pages(void)
 	return 1UL << order;
 }
 
-#ifdef CONFIG_HIGHMEM
-pte_t *kmap_pte;
-pgprot_t kmap_prot;
+/*
+ * These are almost like kmap_atomic / kunmap_atmic except they take an
+ * additional address argument as the hint.
+ */
 
 #define kmap_get_fixmap_pte(vaddr)					\
 	pte_offset_kernel(pmd_offset(pud_offset(pgd_offset_k(vaddr), (vaddr)), (vaddr)), (vaddr))
 
+#ifdef CONFIG_MIPS_MT_SMTC
+static pte_t *kmap_coherent_pte;
+static void __init kmap_coherent_init(void)
+{
+	unsigned long vaddr;
+
+	/* cache the first coherent kmap pte */
+	vaddr = __fix_to_virt(FIX_CMAP_BEGIN);
+	kmap_coherent_pte = kmap_get_fixmap_pte(vaddr);
+}
+#else
+static inline void kmap_coherent_init(void) {}
+#endif
+
+static inline void *kmap_coherent(struct page *page, unsigned long addr)
+{
+	enum fixed_addresses idx;
+	unsigned long vaddr, flags, entrylo;
+	unsigned long old_ctx;
+	pte_t pte;
+	unsigned int tlbidx;
+
+	inc_preempt_count();
+	idx = (addr >> PAGE_SHIFT) & (FIX_N_COLOURS - 1);
+#ifdef CONFIG_MIPS_MT_SMTC
+	idx += FIX_N_COLOURS * smp_processor_id();
+#endif
+	vaddr = __fix_to_virt(FIX_CMAP_END - idx);
+	pte = mk_pte(page, PAGE_KERNEL);
+#if defined(CONFIG_64BIT_PHYS_ADDR) && defined(CONFIG_CPU_MIPS32_R1)
+	entrylo = pte.pte_high;
+#else
+	entrylo = pte_val(pte) >> 6;
+#endif
+
+	ENTER_CRITICAL(flags);
+	old_ctx = read_c0_entryhi();
+	write_c0_entryhi(vaddr & (PAGE_MASK << 1));
+	write_c0_entrylo0(entrylo);
+	write_c0_entrylo1(entrylo);
+#ifdef CONFIG_MIPS_MT_SMTC
+	set_pte(kmap_coherent_pte - (FIX_CMAP_END - idx), pte);
+	/* preload TLB instead of local_flush_tlb_one() */
+	mtc0_tlbw_hazard();
+	tlb_probe();
+	BARRIER;
+	tlbidx = read_c0_index();
+	mtc0_tlbw_hazard();
+	if (tlbidx < 0)
+		tlb_write_random();
+	else
+		tlb_write_indexed();
+#else
+	tlbidx = read_c0_wired();
+	write_c0_wired(tlbidx + 1);
+	write_c0_index(tlbidx);
+	mtc0_tlbw_hazard();
+	tlb_write_indexed();
+#endif
+	tlbw_use_hazard();
+	write_c0_entryhi(old_ctx);
+	EXIT_CRITICAL(flags);
+
+	return (void*) vaddr;
+}
+
+#define UNIQUE_ENTRYHI(idx) (CKSEG0 + ((idx) << (PAGE_SHIFT + 1)))
+
+static inline void kunmap_coherent(struct page *page)
+{
+#ifndef CONFIG_MIPS_MT_SMTC
+	unsigned int wired;
+	unsigned long flags, old_ctx;
+
+	ENTER_CRITICAL(flags);
+	old_ctx = read_c0_entryhi();
+	wired = read_c0_wired() - 1;
+	write_c0_wired(wired);
+	write_c0_index(wired);
+	write_c0_entryhi(UNIQUE_ENTRYHI(wired));
+	write_c0_entrylo0(0);
+	write_c0_entrylo1(0);
+	mtc0_tlbw_hazard();
+	tlb_write_indexed();
+	write_c0_entryhi(old_ctx);
+	EXIT_CRITICAL(flags);
+#endif
+	dec_preempt_count();
+	preempt_check_resched();
+}
+
+void copy_user_highpage(struct page *to, struct page *from,
+	unsigned long vaddr, struct vm_area_struct *vma)
+{
+	void *vfrom, *vto;
+
+	vto = kmap_atomic(to, KM_USER1);
+	if (cpu_has_dc_aliases) {
+		vfrom = kmap_coherent(from, vaddr);
+		copy_page(vto, vfrom);
+		kunmap_coherent(from);
+	} else {
+		vfrom = kmap_atomic(from, KM_USER0);
+		copy_page(vto, vfrom);
+		kunmap_atomic(vfrom, KM_USER0);
+	}
+	if (((vma->vm_flags & VM_EXEC) && !cpu_has_ic_fills_f_dc) ||
+	    pages_do_alias((unsigned long)vto, vaddr & PAGE_MASK))
+		flush_data_cache_page((unsigned long)vto);
+	kunmap_atomic(vto, KM_USER1);
+	/* Make sure this page is cleared on other CPU's too before using it */
+	smp_wmb();
+}
+
+EXPORT_SYMBOL(copy_user_highpage);
+
+void copy_user_page(void *vto, void *vfrom, unsigned long vaddr,
+	struct page *to)
+{
+	if (cpu_has_dc_aliases) {
+		struct page *from = virt_to_page(vfrom);
+		vfrom = kmap_coherent(from, vaddr);
+		copy_page(vto, vfrom);
+		kunmap_coherent(from);
+	} else
+		copy_page(vto, vfrom);
+	if (!cpu_has_ic_fills_f_dc ||
+	    pages_do_alias((unsigned long)vto, vaddr & PAGE_MASK))
+		flush_data_cache_page((unsigned long)vto);
+}
+
+EXPORT_SYMBOL(copy_user_page);
+
+void copy_to_user_page(struct vm_area_struct *vma,
+	struct page *page, unsigned long vaddr, void *dst, const void *src,
+	unsigned long len)
+{
+	if (cpu_has_dc_aliases) {
+		void *vto = kmap_coherent(page, vaddr) + (vaddr & ~PAGE_MASK);
+		memcpy(vto, src, len);
+		kunmap_coherent(page);
+	} else
+		memcpy(dst, src, len);
+	if ((vma->vm_flags & VM_EXEC) && !cpu_has_ic_fills_f_dc)
+		flush_cache_page(vma, vaddr, page_to_pfn(page));
+}
+
+EXPORT_SYMBOL(copy_to_user_page);
+
+void copy_from_user_page(struct vm_area_struct *vma,
+	struct page *page, unsigned long vaddr, void *dst, const void *src,
+	unsigned long len)
+{
+	if (cpu_has_dc_aliases) {
+		void *vfrom =
+			kmap_coherent(page, vaddr) + (vaddr & ~PAGE_MASK);
+		memcpy(dst, vfrom, len);
+		kunmap_coherent(page);
+	} else
+		memcpy(dst, src, len);
+}
+
+EXPORT_SYMBOL(copy_from_user_page);
+
+
+#ifdef CONFIG_HIGHMEM
+pte_t *kmap_pte;
+pgprot_t kmap_prot;
+
 static void __init kmap_init(void)
 {
 	unsigned long kmap_vstart;
@@ -97,11 +295,12 @@ static void __init kmap_init(void)
 
 	kmap_prot = PAGE_KERNEL;
 }
+#endif /* CONFIG_HIGHMEM */
 
-#ifdef CONFIG_32BIT
 void __init fixrange_init(unsigned long start, unsigned long end,
 	pgd_t *pgd_base)
 {
+#if defined(CONFIG_HIGHMEM) || defined(CONFIG_MIPS_MT_SMTC)
 	pgd_t *pgd;
 	pud_t *pud;
 	pmd_t *pmd;
@@ -122,7 +321,7 @@ void __init fixrange_init(unsigned long 
 			for (; (k < PTRS_PER_PMD) && (vaddr != end); pmd++, k++) {
 				if (pmd_none(*pmd)) {
 					pte = (pte_t *) alloc_bootmem_low_pages(PAGE_SIZE);
-					set_pmd(pmd, __pmd(pte));
+					set_pmd(pmd, __pmd((unsigned long)pte));
 					if (pte != pte_offset_kernel(pmd, 0))
 						BUG();
 				}
@@ -132,9 +331,8 @@ void __init fixrange_init(unsigned long 
 		}
 		j = 0;
 	}
+#endif
 }
-#endif /* CONFIG_32BIT */
-#endif /* CONFIG_HIGHMEM */
 
 #ifndef CONFIG_NEED_MULTIPLE_NODES
 extern void pagetable_init(void);
@@ -175,6 +373,7 @@ #endif
 #ifdef CONFIG_HIGHMEM
 	kmap_init();
 #endif
+	kmap_coherent_init();
 
 	max_dma = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT;
 	low = max_low_pfn;
diff --git a/arch/mips/mm/pgtable-32.c b/arch/mips/mm/pgtable-32.c
index 4bdaa05..4a61e62 100644
--- a/arch/mips/mm/pgtable-32.c
+++ b/arch/mips/mm/pgtable-32.c
@@ -31,9 +31,10 @@ void pgd_init(unsigned long page)
 
 void __init pagetable_init(void)
 {
-#ifdef CONFIG_HIGHMEM
 	unsigned long vaddr;
-	pgd_t *pgd, *pgd_base;
+	pgd_t *pgd_base;
+#ifdef CONFIG_HIGHMEM
+	pgd_t *pgd;
 	pud_t *pud;
 	pmd_t *pmd;
 	pte_t *pte;
@@ -44,7 +45,6 @@ #endif
 	pgd_init((unsigned long)swapper_pg_dir
 		 + sizeof(pgd_t) * USER_PTRS_PER_PGD);
 
-#ifdef CONFIG_HIGHMEM
 	pgd_base = swapper_pg_dir;
 
 	/*
@@ -53,6 +53,7 @@ #ifdef CONFIG_HIGHMEM
 	vaddr = __fix_to_virt(__end_of_fixed_addresses - 1) & PMD_MASK;
 	fixrange_init(vaddr, 0, pgd_base);
 
+#ifdef CONFIG_HIGHMEM
 	/*
 	 * Permanent kmaps:
 	 */
diff --git a/arch/mips/mm/pgtable-64.c b/arch/mips/mm/pgtable-64.c
index 44b5e97..8d600d3 100644
--- a/arch/mips/mm/pgtable-64.c
+++ b/arch/mips/mm/pgtable-64.c
@@ -8,6 +8,7 @@
  */
 #include <linux/init.h>
 #include <linux/mm.h>
+#include <asm/fixmap.h>
 #include <asm/pgtable.h>
 
 void pgd_init(unsigned long page)
@@ -52,7 +53,17 @@ void pmd_init(unsigned long addr, unsign
 
 void __init pagetable_init(void)
 {
+	unsigned long vaddr;
+	pgd_t *pgd_base;
+
 	/* Initialize the entire pgd.  */
 	pgd_init((unsigned long)swapper_pg_dir);
 	pmd_init((unsigned long)invalid_pmd_table, (unsigned long)invalid_pte_table);
+
+	pgd_base = swapper_pg_dir;
+	/*
+	 * Fixed mappings:
+	 */
+	vaddr = __fix_to_virt(__end_of_fixed_addresses - 1) & PMD_MASK;
+	fixrange_init(vaddr, 0, pgd_base);
 }
diff --git a/include/asm-mips/cacheflush.h b/include/asm-mips/cacheflush.h
index 36416fd..d10517c 100644
--- a/include/asm-mips/cacheflush.h
+++ b/include/asm-mips/cacheflush.h
@@ -57,24 +57,13 @@ extern void (*flush_icache_range)(unsign
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 
-static inline void copy_to_user_page(struct vm_area_struct *vma,
+extern void copy_to_user_page(struct vm_area_struct *vma,
 	struct page *page, unsigned long vaddr, void *dst, const void *src,
-	unsigned long len)
-{
-	if (cpu_has_dc_aliases)
-		flush_cache_page(vma, vaddr, page_to_pfn(page));
-	memcpy(dst, src, len);
-	__flush_icache_page(vma, page);
-}
+	unsigned long len);
 
-static inline void copy_from_user_page(struct vm_area_struct *vma,
+extern void copy_from_user_page(struct vm_area_struct *vma,
 	struct page *page, unsigned long vaddr, void *dst, const void *src,
-	unsigned long len)
-{
-	if (cpu_has_dc_aliases)
-		flush_cache_page(vma, vaddr, page_to_pfn(page));
-	memcpy(dst, src, len);
-}
+	unsigned long len);
 
 extern void (*flush_cache_sigtramp)(unsigned long addr);
 extern void (*flush_icache_all)(void);
diff --git a/include/asm-mips/fixmap.h b/include/asm-mips/fixmap.h
index 6959bdb..02c8a13 100644
--- a/include/asm-mips/fixmap.h
+++ b/include/asm-mips/fixmap.h
@@ -45,8 +45,16 @@ #endif
  * fix-mapped?
  */
 enum fixed_addresses {
+#define FIX_N_COLOURS 8
+	FIX_CMAP_BEGIN,
+#ifdef CONFIG_MIPS_MT_SMTC
+	FIX_CMAP_END = FIX_CMAP_BEGIN + (FIX_N_COLOURS * NR_CPUS),
+#else
+	FIX_CMAP_END = FIX_CMAP_BEGIN + FIX_N_COLOURS,
+#endif
 #ifdef CONFIG_HIGHMEM
-	FIX_KMAP_BEGIN,	/* reserved pte's for temporary kernel mappings */
+	/* reserved pte's for temporary kernel mappings */
+	FIX_KMAP_BEGIN = FIX_CMAP_END + 1,
 	FIX_KMAP_END = FIX_KMAP_BEGIN+(KM_TYPE_NR*NR_CPUS)-1,
 #endif
 	__end_of_fixed_addresses
@@ -70,9 +78,9 @@ #define set_fixmap_nocache(idx, phys) \
  * at the top of mem..
  */
 #if defined(CONFIG_CPU_TX39XX) || defined(CONFIG_CPU_TX49XX)
-#define FIXADDR_TOP	(0xff000000UL - 0x2000)
+#define FIXADDR_TOP	((unsigned long)(long)(int)(0xff000000 - 0x20000))
 #else
-#define FIXADDR_TOP	(0xffffe000UL)
+#define FIXADDR_TOP	((unsigned long)(long)(int)0xfffe0000)
 #endif
 #define FIXADDR_SIZE	(__end_of_fixed_addresses << PAGE_SHIFT)
 #define FIXADDR_START	(FIXADDR_TOP - FIXADDR_SIZE)
diff --git a/include/asm-mips/page.h b/include/asm-mips/page.h
index 6882c80..28172a0 100644
--- a/include/asm-mips/page.h
+++ b/include/asm-mips/page.h
@@ -35,8 +35,6 @@ #define PAGE_MASK       (~((1 << PAGE_SH
 
 #ifndef __ASSEMBLY__
 
-#include <asm/cpu-features.h>
-
 extern void clear_page(void * page);
 extern void copy_page(void * to, void * from);
 
@@ -60,16 +58,13 @@ static inline void clear_user_page(void 
 		flush_data_cache_page((unsigned long)addr);
 }
 
-static inline void copy_user_page(void *vto, void *vfrom, unsigned long vaddr,
-	struct page *to)
-{
-	extern void (*flush_data_cache_page)(unsigned long addr);
+extern void copy_user_page(void *vto, void *vfrom, unsigned long vaddr,
+	struct page *to);
+struct vm_area_struct;
+extern void copy_user_highpage(struct page *to, struct page *from,
+	unsigned long vaddr, struct vm_area_struct *vma);
 
-	copy_page(vto, vfrom);
-	if (!cpu_has_ic_fills_f_dc ||
-	    pages_do_alias((unsigned long)vto, vaddr & PAGE_MASK))
-		flush_data_cache_page((unsigned long)vto);
-}
+#define __HAVE_ARCH_COPY_USER_HIGHPAGE
 
 /*
  * These are used to make use of C type-checking..
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 85ce7ef..aa4f32d 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -89,7 +89,10 @@ static inline void memclear_highpage_flu
 	kunmap_atomic(kaddr, KM_USER0);
 }
 
-static inline void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr)
+#ifndef __HAVE_ARCH_COPY_USER_HIGHPAGE
+
+static inline void copy_user_highpage(struct page *to, struct page *from,
+	unsigned long vaddr, struct vm_area_struct *vma)
 {
 	char *vfrom, *vto;
 
@@ -102,6 +105,8 @@ static inline void copy_user_highpage(st
 	smp_wmb();
 }
 
+#endif
+
 static inline void copy_highpage(struct page *to, struct page *from)
 {
 	char *vfrom, *vto;
diff --git a/mm/memory.c b/mm/memory.c
index 109e986..2e4ee87 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1407,7 +1407,7 @@ static inline pte_t maybe_mkwrite(pte_t 
 	return pte;
 }
 
-static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va)
+static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma)
 {
 	/*
 	 * If the source page was a PFN mapping, we don't have
@@ -1429,9 +1429,9 @@ static inline void cow_user_page(struct 
 			memset(kaddr, 0, PAGE_SIZE);
 		kunmap_atomic(kaddr, KM_USER0);
 		return;
-		
+
 	}
-	copy_user_highpage(dst, src, va);
+	copy_user_highpage(dst, src, va, vma);
 }
 
 /*
@@ -1531,7 +1531,7 @@ gotten:
 		new_page = alloc_page_vma(GFP_HIGHUSER, vma, address);
 		if (!new_page)
 			goto oom;
-		cow_user_page(new_page, old_page, address);
+		cow_user_page(new_page, old_page, address, vma);
 	}
 
 	/*
@@ -2135,7 +2135,7 @@ retry:
 			page = alloc_page_vma(GFP_HIGHUSER, vma, address);
 			if (!page)
 				goto oom;
-			copy_user_highpage(page, new_page, address);
+			copy_user_highpage(page, new_page, address, vma);
 			page_cache_release(new_page);
 			new_page = page;
 			anon = 1;

Re: Cache alias handling in copy_user_page / copy_user_highpage

[David Miller]

From: Ralf Baechle <ralf@linux-mips.org>
Date: Thu, 5 Oct 2006 17:54:29 +0100

> I found it to be much more maintainable to deal with this in C due to a
> large number of processor variations to be supported.

It is a reasonable reason to use code patching :-)

Anyways, I have no problems with your patch.

Re: Cache alias handling in copy_user_page / copy_user_highpage

[Ralf Baechle]

On Thu, Oct 05, 2006 at 01:06:31PM -0700, David Miller wrote:

> > I found it to be much more maintainable to deal with this in C due to a
> > large number of processor variations to be supported.
> 
> It is a reasonable reason to use code patching :-)

In which case I guess you will enjoy arch/mips/mm/tlbex.c which is
custom tayloring TLB exception handler for the particular CPU type, errata
workaround, UP/SMP and alot of other stuff ;)

  Ralf