[PATCH] FCSE

[PATCH] FCSE

[Gilles Chanteperdrix]

Hi,

Here comes the FCSE patch for Linux 2.6.32-rc1. 

To summarize a bit the story of this patch which you will find in
various posts on this list, this patch adds an option to the kernel
configuration which allows using the ARM FCSE in one of two modes:
- the guaranteed mode, where context switches are guaranteed to run
without flushing the caches, but where the number of processes is
limited to 95 with a virtual memory space limited to 32 MB;
- the best-effort mode, where the limitations are removed, but where
the cache is flushed when needed during context switches.

The patch can be considered as relatively stable, it has not changed
much since Linux 2.6.27, and has been validated by running LTP with
the best-effort mode, with no differences in the LTP results compared
to an unpatched kernel. Furthermore, it has been reported to work in
the field.

As for the performance improvement, running the hackbench test with
400 processes (so, well above the 95 processes limit) on an AT91RM9200
showed an improvement in the runing time which can be visualized here:
http://sisyphus.hd.free.fr/~gilles/hackbench.png.

The user-space worst-case scheduling latency is also improved by the
guaranteed mode, but this only interests people running an RT kernel
of any kind.

The patch is relatively simple and only modifies ARM-specific code,
but has been split to ease review, as I have been advised off-list to
do. Some parts of it, however, could be implemented more elegantly
(to say the least), especially patches 8 and 9 which intercept
set_pte_at() to count pages belonging to some types of mappings
whereas counting mappings would be enough but would probably require
to modify generic code.

Thanks in advance for your review.
Regards

-- 
					    Gilles.

[PATCH] FCSE: pid life cycle.

[Gilles Chanteperdrix]

Add a pid member to the mm_context_t structure, allocate this pid in
the init_new_context, free it in the destroy_context, switch it in
switch_mm.

A bitfield is maintained to know what pids are currently in use.

Signed-off-by: Richard Cochran <richard.cochran@omicron.at>
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/fcse.h        |   44 +++++++++++++++++++++++++++
 arch/arm/include/asm/mmu.h         |    3 ++
 arch/arm/include/asm/mmu_context.h |   24 ++++++++++++++-
 arch/arm/kernel/fcse.c             |   57 ++++++++++++++++++++++++++++++++++++
 4 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/include/asm/fcse.h
 create mode 100644 arch/arm/kernel/fcse.c

diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h
new file mode 100644
index 0000000..b44e866
--- /dev/null
+++ b/arch/arm/include/asm/fcse.h
@@ -0,0 +1,44 @@
+/*
+ * Filename:    arch/arm/include/asm/fcse.h
+ * Description: ARM Process ID (PID) includes for Fast Address Space Switching
+ *              (FASS) in ARM Linux.
+ * Created:     14/10/2001
+ * Changes:     19/02/2002 - Macros added.
+ *              03/08/2007 - Adapted to kernel 2.6.21 (ssm)
+ *              Feb 2008   - Simplified a bit (rco)
+ *
+ * Copyright:   (C) 2001, 2002 Adam Wiggins <awiggins@cse.unsw.edu.au>
+ *              (C) 2007 Sebastian Smolorz <ssm@emlix.com>
+ *              (C) 2008 Richard Cochran
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of teh GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#ifndef __ASM_ARM_FCSE_H
+#define __ASM_ARM_FCSE_H
+
+#ifdef CONFIG_ARM_FCSE
+
+#include <linux/mm_types.h>	/* For struct mm_struct */
+
+#define FCSE_PID_SHIFT 25
+
+/* Size of PID relocation area */
+#define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT)
+
+/* Sets the CPU's PID Register */
+static inline void fcse_pid_set(unsigned long pid)
+{
+	__asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0"
+			      : /* */: "r" (pid) : "memory");
+}
+
+int fcse_pid_alloc(void);
+void fcse_pid_free(unsigned pid);
+
+#else /* ! CONFIG_ARM_FCSE */
+#define fcse_pid_set(pid) do { } while (0)
+#endif /* ! CONFIG_ARM_FCSE */
+
+#endif /* __ASM_ARM_FCSE_H */
diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h
index b561584..4a5a8db 100644
--- a/arch/arm/include/asm/mmu.h
+++ b/arch/arm/include/asm/mmu.h
@@ -7,6 +7,9 @@ typedef struct {
 #ifdef CONFIG_CPU_HAS_ASID
 	unsigned int id;
 #endif
+#ifdef CONFIG_ARM_FCSE
+	unsigned long pid;
+#endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
 } mm_context_t;
 
diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index de6cefb..ec7a84d 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -19,6 +19,7 @@
 #include <asm/cachetype.h>
 #include <asm/proc-fns.h>
 #include <asm-generic/mm_hooks.h>
+#include <asm/fcse.h>
 
 void __check_kvm_seq(struct mm_struct *mm);
 
@@ -68,11 +69,29 @@ static inline void check_context(struct mm_struct *mm)
 #endif
 }
 
-#define init_new_context(tsk,mm)	0
+static inline int
+init_new_context(struct task_struct *tsk, struct mm_struct *mm)
+{
+#ifdef CONFIG_ARM_FCSE
+	int pid;
+
+	pid = fcse_pid_alloc();
+	if (pid < 0)
+		return pid;
+	mm->context.pid = pid << FCSE_PID_SHIFT;
+#endif /* CONFIG_ARM_FCSE */
+
+	return 0;
+}
 
 #endif
 
-#define destroy_context(mm)		do { } while(0)
+static inline void destroy_context(struct mm_struct *mm)
+{
+#ifdef CONFIG_ARM_FCSE
+	fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT);
+#endif /* CONFIG_ARM_FCSE */
+}
 
 /*
  * This is called when "tsk" is about to enter lazy TLB mode.
@@ -109,6 +128,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 #endif
 	if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) {
 		check_context(next);
+		fcse_pid_set(next->context.pid);
 		cpu_switch_mm(next->pgd, next);
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, mm_cpumask(prev));
diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c
new file mode 100644
index 0000000..da074b8
--- /dev/null
+++ b/arch/arm/kernel/fcse.c
@@ -0,0 +1,57 @@
+#include <linux/bitops.h>
+#include <linux/memory.h>
+#include <linux/spinlock.h>
+#include <linux/mm.h>
+
+#include <asm/fcse.h>
+#include <asm/cacheflush.h>
+#include <asm/tlbflush.h>
+
+#define NR_PIDS (TASK_SIZE / FCSE_PID_TASK_SIZE)
+#define PIDS_LONGS ((NR_PIDS + BITS_PER_LONG - 1) / BITS_PER_LONG)
+
+static DEFINE_SPINLOCK(fcse_lock);
+static unsigned long fcse_pids_bits[PIDS_LONGS];
+
+static void fcse_pid_reference_inner(unsigned pid)
+{
+	__set_bit(pid, fcse_pids_bits);
+}
+
+static void fcse_pid_dereference(unsigned pid)
+{
+	__clear_bit(pid, fcse_pids_bits);
+}
+
+int fcse_pid_alloc(void)
+{
+	unsigned long flags;
+	unsigned pid;
+
+	spin_lock_irqsave(&fcse_lock, flags);
+	pid = find_next_zero_bit(fcse_pids_bits, NR_PIDS, 1);
+	if (pid == NR_PIDS) {
+		/* Allocate zero pid last, since zero pid is also used by
+		   processes with address space larger than 32MB in
+		   best-effort mode. */
+		if (!test_bit(0, fcse_pids_bits))
+			pid = 0;
+		else {
+			spin_unlock_irqrestore(&fcse_lock, flags);
+			return -EAGAIN;
+		}
+	}
+	fcse_pid_reference_inner(pid);
+	spin_unlock_irqrestore(&fcse_lock, flags);
+
+	return pid;
+}
+
+void fcse_pid_free(unsigned pid)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fcse_lock, flags);
+	fcse_pid_dereference(pid);
+	spin_unlock_irqrestore(&fcse_lock, flags);
+}
-- 
1.5.6.5

[PATCH] FCSE: Conversions between VA and MVA.

[Gilles Chanteperdrix]

Add to asm/fcse.h the functions needed to allow conversion between
virtual addresses (a.k.a VA) and modified virtual addresses (a.k.a
MVA), that is virtual addresses translated by the FCSE pid.

Though the whole kernel handles VA, and do not see MVA, there are a
few exceptions:
- the function handling page table walking must use MVA, this is
solved by getting the pgd_offset function to use fcse_va_to_mva, which
turns out to be sufficient;
- the TLB and cache operations which may be passed user-space
addresses must use MVA, so, they call fcse_va_to_mva too;
- the function do_DataAbort is passed an MVA which has to be converted
into a VA, this is done by calling fcse_mva_to_va.

Signed-off-by: Richard Cochran <richard.cochran@omicron.at>
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/cacheflush.h |   26 ++++++++++++++++++++------
 arch/arm/include/asm/fcse.h       |   29 +++++++++++++++++++++++++++++
 arch/arm/include/asm/pgtable.h    |   10 ++++++++--
 arch/arm/include/asm/tlbflush.h   |   14 ++++++++++++--
 arch/arm/mm/fault.c               |    6 ++++++
 arch/arm/mm/flush.c               |   15 ++++++++++-----
 arch/arm/mm/pgd.c                 |    7 ++++++-
 7 files changed, 91 insertions(+), 16 deletions(-)

diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h
index fd03fb6..db43631 100644
--- a/arch/arm/include/asm/cacheflush.h
+++ b/arch/arm/include/asm/cacheflush.h
@@ -15,6 +15,7 @@
 #include <asm/glue.h>
 #include <asm/shmparam.h>
 #include <asm/cachetype.h>
+#include <asm/fcse.h>
 
 #define CACHE_COLOUR(vaddr)	((vaddr & (SHMLBA - 1)) >> PAGE_SHIFT)
 
@@ -341,16 +342,20 @@ static inline void flush_cache_mm(struct mm_struct *mm)
 static inline void
 flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)
 {
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm)))
+	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
+		start = fcse_va_to_mva(vma->vm_mm, start);
+		end = fcse_va_to_mva(vma->vm_mm, end);
 		__cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end),
 					vma->vm_flags);
+	}
 }
 
 static inline void
 flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn)
 {
 	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
-		unsigned long addr = user_addr & PAGE_MASK;
+		unsigned long addr;
+		addr = fcse_va_to_mva(vma->vm_mm, user_addr) & PAGE_MASK;
 		__cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags);
 	}
 }
@@ -381,14 +386,22 @@ extern void flush_ptrace_access(struct vm_area_struct *vma, struct page *page,
  * Harvard caches are synchronised for the user space address range.
  * This is used for the ARM private sys_cacheflush system call.
  */
-#define flush_cache_user_range(vma,start,end) \
-	__cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end))
+#define flush_cache_user_range(vma, start, end)				\
+	({								\
+		struct mm_struct *_mm = (vma)->vm_mm;			\
+		unsigned long _start, _end;				\
+		_start = fcse_va_to_mva(_mm, start) & PAGE_MASK;	\
+		_end = PAGE_ALIGN(fcse_va_to_mva(_mm, end));		\
+		__cpuc_coherent_user_range(_start, _end);		\
+	})
 
 /*
  * Perform necessary cache operations to ensure that data previously
  * stored within this range of addresses can be executed by the CPU.
  */
-#define flush_icache_range(s,e)		__cpuc_coherent_kern_range(s,e)
+#define flush_icache_range(s,e)						\
+	__cpuc_coherent_kern_range(fcse_va_to_mva(current->mm, (s)),	\
+				   fcse_va_to_mva(current->mm, (e)))
 
 /*
  * Perform necessary cache operations to ensure that the TLB will
@@ -426,7 +439,8 @@ static inline void flush_anon_page(struct vm_area_struct *vma,
 	extern void __flush_anon_page(struct vm_area_struct *vma,
 				struct page *, unsigned long);
 	if (PageAnon(page))
-		__flush_anon_page(vma, page, vmaddr);
+		__flush_anon_page(vma, page,
+				  fcse_va_to_mva(vma->vm_mm, vmaddr));
 }
 
 #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE
diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h
index b44e866..e443278 100644
--- a/arch/arm/include/asm/fcse.h
+++ b/arch/arm/include/asm/fcse.h
@@ -27,6 +27,9 @@
 /* Size of PID relocation area */
 #define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT)
 
+/* Mask to get rid of PID from relocated address */
+#define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1)
+
 /* Sets the CPU's PID Register */
 static inline void fcse_pid_set(unsigned long pid)
 {
@@ -34,11 +37,37 @@ static inline void fcse_pid_set(unsigned long pid)
 			      : /* */: "r" (pid) : "memory");
 }
 
+/* Returns the state of the CPU's PID Register */
+static inline unsigned long fcse_pid_get(void)
+{
+	unsigned long pid;
+	__asm__ __volatile__("mrc p15, 0, %0, c13, c0, 0" : "=&r" (pid));
+	return pid & ~FCSE_PID_MASK;
+}
+
+static inline unsigned long fcse_mva_to_va(unsigned long mva)
+{
+	unsigned long pid = fcse_pid_get();
+	if (pid && (pid == (mva & ~FCSE_PID_MASK)))
+		return mva & FCSE_PID_MASK;
+	return mva;
+}
+
+static inline unsigned long
+fcse_va_to_mva(struct mm_struct *mm, unsigned long va)
+{
+	if (va < FCSE_PID_TASK_SIZE)
+		return mm->context.pid | va;
+	return va;
+}
+
 int fcse_pid_alloc(void);
 void fcse_pid_free(unsigned pid);
 
 #else /* ! CONFIG_ARM_FCSE */
 #define fcse_pid_set(pid) do { } while (0)
+#define fcse_mva_to_va(x) (x)
+#define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); })
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index 201ccaa..197c596 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -112,6 +112,8 @@
 #define LIBRARY_TEXT_START	0x0c000000
 
 #ifndef __ASSEMBLY__
+#include <asm/fcse.h>
+
 extern void __pte_error(const char *file, int line, unsigned long val);
 extern void __pmd_error(const char *file, int line, unsigned long val);
 extern void __pgd_error(const char *file, int line, unsigned long val);
@@ -362,10 +364,14 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd)
 /* to find an entry in a page-table-directory */
 #define pgd_index(addr)		((addr) >> PGDIR_SHIFT)
 
-#define pgd_offset(mm, addr)	((mm)->pgd+pgd_index(addr))
+#define pgd_offset(mm, addr)						\
+	({								\
+		struct mm_struct *_mm = (mm);				\
+		(_mm->pgd + pgd_index(fcse_va_to_mva(_mm, (addr))));	\
+	})
 
 /* to find an entry in a kernel page-table-directory */
-#define pgd_offset_k(addr)	pgd_offset(&init_mm, addr)
+#define pgd_offset_k(addr)	(init_mm.pgd + pgd_index(addr))
 
 /* Find an entry in the second-level page table.. */
 #define pmd_offset(dir, addr)	((pmd_t *)(dir))
diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h
index a45ab5d..7e14a84 100644
--- a/arch/arm/include/asm/tlbflush.h
+++ b/arch/arm/include/asm/tlbflush.h
@@ -210,6 +210,7 @@
 #ifndef __ASSEMBLY__
 
 #include <linux/sched.h>
+#include <asm/fcse.h>
 
 struct cpu_tlb_fns {
 	void (*flush_user_range)(unsigned long, unsigned long, struct vm_area_struct *);
@@ -383,7 +384,8 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr)
 	const int zero = 0;
 	const unsigned int __tlb_flag = __cpu_tlb_flags;
 
-	uaddr = (uaddr & PAGE_MASK) | ASID(vma->vm_mm);
+	uaddr = (fcse_va_to_mva(vma->vm_mm, uaddr) & PAGE_MASK)
+		| ASID(vma->vm_mm);
 
 	if (tlb_flag(TLB_WB))
 		dsb();
@@ -504,7 +506,15 @@ static inline void clean_pmd_entry(pmd_t *pmd)
 /*
  * Convert calls to our calling convention.
  */
-#define local_flush_tlb_range(vma,start,end)	__cpu_flush_user_tlb_range(start,end,vma)
+#define local_flush_tlb_range(vma, start, end)			\
+	({							\
+		struct mm_struct *_mm = (vma)->vm_mm;		\
+		unsigned long _start, _end;			\
+		_start = fcse_va_to_mva(_mm, start);		\
+		_end = fcse_va_to_mva(_mm, end);		\
+		__cpu_flush_user_tlb_range(_start, _end, vma);	\
+	})
+
 #define local_flush_tlb_kernel_range(s,e)	__cpu_flush_kern_tlb_range(s,e)
 
 #ifndef CONFIG_SMP
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index 379f785..493c980 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -73,6 +73,10 @@ void show_pte(struct mm_struct *mm, unsigned long addr)
 	if (!mm)
 		mm = &init_mm;
 
+#ifdef CONFIG_ARM_FCSE
+	printk(KERN_ALERT "fcse pid: %ld, 0x%08lx\n",
+	       mm->context.pid >> FCSE_PID_SHIFT, mm->context.pid);
+#endif /* CONFIG_ARM_FCSE */
 	printk(KERN_ALERT "pgd = %p\n", mm->pgd);
 	pgd = pgd_offset(mm, addr);
 	printk(KERN_ALERT "[%08lx] *pgd=%08lx", addr, pgd_val(*pgd));
@@ -506,6 +510,8 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 	const struct fsr_info *inf = fsr_info + fsr_fs(fsr);
 	struct siginfo info;
 
+	addr = fcse_mva_to_va(addr);
+
 	if (!inf->fn(addr, fsr & ~FSR_LNX_PF, regs))
 		return;
 
diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c
index b279429..ec6fcda 100644
--- a/arch/arm/mm/flush.c
+++ b/arch/arm/mm/flush.c
@@ -73,9 +73,11 @@ void flush_cache_mm(struct mm_struct *mm)
 void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)
 {
 	if (cache_is_vivt()) {
-		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm)))
-			__cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end),
-						vma->vm_flags);
+		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
+			start = fcse_va_to_mva(vma->vm_mm, start) & PAGE_MASK;
+			end = PAGE_ALIGN(fcse_va_to_mva(vma->vm_mm, end));
+			__cpuc_flush_user_range(start, end, vma->vm_flags);
+		}
 		return;
 	}
 
@@ -98,8 +100,11 @@ void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsig
 {
 	if (cache_is_vivt()) {
 		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
-			unsigned long addr = user_addr & PAGE_MASK;
-			__cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags);
+			unsigned long addr;
+			addr = fcse_va_to_mva(vma->vm_mm, user_addr);
+			addr &= PAGE_MASK;
+			__cpuc_flush_user_range(addr, addr + PAGE_SIZE,
+						vma->vm_flags);
 		}
 		return;
 	}
diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c
index 2690146..9796798 100644
--- a/arch/arm/mm/pgd.c
+++ b/arch/arm/mm/pgd.c
@@ -43,6 +43,11 @@ pgd_t *get_pgd_slow(struct mm_struct *mm)
 	clean_dcache_area(new_pgd, PTRS_PER_PGD * sizeof(pgd_t));
 
 	if (!vectors_high()) {
+#ifdef CONFIG_ARM_FCSE
+		/* FCSE does not work without high vectors. */
+		BUG();
+#endif /* CONFIG_ARM_FCSE */
+
 		/*
 		 * On ARM, first page must always be allocated since it
 		 * contains the machine vectors.
@@ -81,7 +86,7 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd)
 		return;
 
 	/* pgd is always present and good */
-	pmd = pmd_off(pgd, 0);
+	pmd = pmd_off(pgd + pgd_index(fcse_va_to_mva(mm, 0)), 0);
 	if (pmd_none(*pmd))
 		goto free;
 	if (pmd_bad(*pmd)) {
-- 
1.5.6.5

[PATCH] FCSE: Address-space limits.

[Gilles Chanteperdrix]

Get the kernel macros to reflect address-space limits imposed by the FCSE.

Signed-off-by: Richard Cochran <richard.cochran@omicron.at>
Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/memory.h    |    5 +++++
 arch/arm/include/asm/processor.h |    5 +++++
 arch/arm/mm/mmap.c               |   22 +++++++++++++++++++---
 3 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
index cefedf0..db9781a 100644
--- a/arch/arm/include/asm/memory.h
+++ b/arch/arm/include/asm/memory.h
@@ -33,7 +33,12 @@
  */
 #define PAGE_OFFSET		UL(CONFIG_PAGE_OFFSET)
 #define TASK_SIZE		(UL(CONFIG_PAGE_OFFSET) - UL(0x01000000))
+#ifndef CONFIG_ARM_FCSE
 #define TASK_UNMAPPED_BASE	(UL(CONFIG_PAGE_OFFSET) / 3)
+#else /* CONFIG_ARM_FCSE */
+#define TASK_UNMAPPED_BASE	UL(0x01000000)
+#define FCSE_TASK_SIZE		UL(0x02000000)
+#endif /* CONFIG_ARM_FCSE */
 
 /*
  * The maximum size of a 26-bit user space task.
diff --git a/arch/arm/include/asm/processor.h b/arch/arm/include/asm/processor.h
index 6a89567..bc33df1 100644
--- a/arch/arm/include/asm/processor.h
+++ b/arch/arm/include/asm/processor.h
@@ -23,9 +23,14 @@
 #include <asm/types.h>
 
 #ifdef __KERNEL__
+#ifndef CONFIG_ARM_FCSE
 #define STACK_TOP	((current->personality & ADDR_LIMIT_32BIT) ? \
 			 TASK_SIZE : TASK_SIZE_26)
 #define STACK_TOP_MAX	TASK_SIZE
+#else /* CONFIG_ARM_FCSE */
+#define STACK_TOP	FCSE_TASK_SIZE
+#define STACK_TOP_MAX	FCSE_TASK_SIZE
+#endif /* CONFIG_ARM_FCSE */
 #endif
 
 union debug_insn {
diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c
index f7457fe..e608bda 100644
--- a/arch/arm/mm/mmap.c
+++ b/arch/arm/mm/mmap.c
@@ -50,13 +50,16 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,
 #define aliasing 0
 #endif
 
+#ifdef CONFIG_ARM_FCSE
+	start_addr = addr;
+#endif /* CONFIG_ARM_FCSE */
 	/*
 	 * We enforce the MAP_FIXED case.
 	 */
 	if (flags & MAP_FIXED) {
 		if (aliasing && flags & MAP_SHARED && addr & (SHMLBA - 1))
 			return -EINVAL;
-		return addr;
+		goto found_addr;
 	}
 
 	if (len > TASK_SIZE)
@@ -71,7 +74,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,
 		vma = find_vma(mm, addr);
 		if (TASK_SIZE - len >= addr &&
 		    (!vma || addr + len <= vma->vm_start))
-			return addr;
+			goto found_addr;
 	}
 	if (len > mm->cached_hole_size) {
 	        start_addr = addr = mm->free_area_cache;
@@ -105,7 +108,7 @@ full_search:
 			 * Remember the place where we stopped the search:
 			 */
 			mm->free_area_cache = addr + len;
-			return addr;
+			goto found_addr;
 		}
 		if (addr + mm->cached_hole_size < vma->vm_start)
 		        mm->cached_hole_size = vma->vm_start - addr;
@@ -113,6 +116,19 @@ full_search:
 		if (do_align)
 			addr = COLOUR_ALIGN(addr, pgoff);
 	}
+
+found_addr:
+#ifdef CONFIG_ARM_FCSE
+	if (addr + len > FCSE_TASK_SIZE) {
+		if (!(flags & MAP_FIXED) && start_addr != TASK_UNMAPPED_BASE) {
+			start_addr = addr = TASK_UNMAPPED_BASE;
+			mm->cached_hole_size = 0;
+			goto full_search;
+		}
+		return -ENOMEM;
+	}
+#endif /* CONFIG_ARM_FCSE */
+	return addr;
 }
 
 
-- 
1.5.6.5

[PATCH] FCSE: differentiate cpu_tlb_mask from cpu_vm_mask.

[Gilles Chanteperdrix]

The ARM Linux kernel uses mm->cpu_vm_mask to avoid unnecessary TLB and
cache flushes when the process about to be flushed has already been
flushed because of a context switch.

When enabling FCSE, a context switch flushes the TLB, but no longer
flushes the cache, so, for correctness, we need to distinguish whether
a process is "present" in the TLB by using a second mask
mm->context.cpu_tlb_mask, or whether it is "present" in the cache by
using mm->cpu_vm_mask.

The current cpu bit is set and cleared from cpu_tlb_mask upon context
switch, but only set in cpu_vm_mask.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/fcse.h        |    5 +++++
 arch/arm/include/asm/mmu.h         |    1 +
 arch/arm/include/asm/mmu_context.h |    7 +++++--
 arch/arm/include/asm/tlbflush.h    |    4 ++--
 arch/arm/kernel/smp.c              |    7 ++++---
 5 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h
index e443278..4e9503d 100644
--- a/arch/arm/include/asm/fcse.h
+++ b/arch/arm/include/asm/fcse.h
@@ -30,6 +30,9 @@
 /* Mask to get rid of PID from relocated address */
 #define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1)
 
+#define fcse_tlb_mask(mm) (&(mm)->context.cpu_tlb_mask)
+#define fcse_cpu_set_vm_mask(cpu, mm) cpumask_set_cpu(cpu, mm_cpumask(mm))
+
 /* Sets the CPU's PID Register */
 static inline void fcse_pid_set(unsigned long pid)
 {
@@ -68,6 +71,8 @@ void fcse_pid_free(unsigned pid);
 #define fcse_pid_set(pid) do { } while (0)
 #define fcse_mva_to_va(x) (x)
 #define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); })
+#define fcse_tlb_mask(mm) mm_cpumask(mm)
+#define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0)
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h
index 4a5a8db..7472598 100644
--- a/arch/arm/include/asm/mmu.h
+++ b/arch/arm/include/asm/mmu.h
@@ -9,6 +9,7 @@ typedef struct {
 #endif
 #ifdef CONFIG_ARM_FCSE
 	unsigned long pid;
+	cpumask_t cpu_tlb_mask;
 #endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
 } mm_context_t;
diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index ec7a84d..042a26d 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -75,6 +75,8 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 #ifdef CONFIG_ARM_FCSE
 	int pid;
 
+	cpus_clear(mm->context.cpu_tlb_mask);
+
 	pid = fcse_pid_alloc();
 	if (pid < 0)
 		return pid;
@@ -126,12 +128,13 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 	    !cpumask_test_cpu(cpu, mm_cpumask(next)))
 		__flush_icache_all();
 #endif
-	if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) {
+	if (!cpumask_test_and_set_cpu(cpu, fcse_tlb_mask(next)) || prev != next) {
+		fcse_cpu_set_vm_mask(cpu, next);
 		check_context(next);
 		fcse_pid_set(next->context.pid);
 		cpu_switch_mm(next->pgd, next);
 		if (cache_is_vivt())
-			cpumask_clear_cpu(cpu, mm_cpumask(prev));
+			cpumask_clear_cpu(cpu, fcse_tlb_mask(prev));
 	}
 #endif
 }
diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h
index 7e14a84..86355b3 100644
--- a/arch/arm/include/asm/tlbflush.h
+++ b/arch/arm/include/asm/tlbflush.h
@@ -351,7 +351,7 @@ static inline void local_flush_tlb_mm(struct mm_struct *mm)
 	if (tlb_flag(TLB_WB))
 		dsb();
 
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) {
+	if (cpumask_test_cpu(smp_processor_id(), fcse_tlb_mask(mm))) {
 		if (tlb_flag(TLB_V3_FULL))
 			asm("mcr p15, 0, %0, c6, c0, 0" : : "r" (zero) : "cc");
 		if (tlb_flag(TLB_V4_U_FULL))
@@ -390,7 +390,7 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr)
 	if (tlb_flag(TLB_WB))
 		dsb();
 
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) {
+	if (cpumask_test_cpu(smp_processor_id(), fcse_tlb_mask(vma->vm_mm))) {
 		if (tlb_flag(TLB_V3_PAGE))
 			asm("mcr p15, 0, %0, c6, c0, 0" : : "r" (uaddr) : "cc");
 		if (tlb_flag(TLB_V4_U_PAGE))
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index e0d3277..a0eb423 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -36,6 +36,7 @@
 #include <asm/tlbflush.h>
 #include <asm/ptrace.h>
 #include <asm/localtimer.h>
+#include <asm/fcse.h>
 
 /*
  * as from 2.5, kernels no longer have an init_tasks structure
@@ -643,7 +644,7 @@ void flush_tlb_all(void)
 void flush_tlb_mm(struct mm_struct *mm)
 {
 	if (tlb_ops_need_broadcast())
-		on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, mm_cpumask(mm));
+		on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, fcse_tlb_mask(mm));
 	else
 		local_flush_tlb_mm(mm);
 }
@@ -654,7 +655,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr)
 		struct tlb_args ta;
 		ta.ta_vma = vma;
 		ta.ta_start = uaddr;
-		on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, mm_cpumask(vma->vm_mm));
+		on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, fcse_tlb_mask(vma->vm_mm));
 	} else
 		local_flush_tlb_page(vma, uaddr);
 }
@@ -677,7 +678,7 @@ void flush_tlb_range(struct vm_area_struct *vma,
 		ta.ta_vma = vma;
 		ta.ta_start = start;
 		ta.ta_end = end;
-		on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, mm_cpumask(vma->vm_mm));
+		on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, fcse_tlb_mask(vma->vm_mm));
 	} else
 		local_flush_tlb_range(vma, start, end);
 }
-- 
1.5.6.5

[PATCH] FCSE: Make the shared mappings uncacheable.

[Gilles Chanteperdrix]

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/mm/fault-armv.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c
index bc0099d..40bccb9 100644
--- a/arch/arm/mm/fault-armv.c
+++ b/arch/arm/mm/fault-armv.c
@@ -94,6 +94,7 @@ no_pmd:
 static void
 make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigned long addr, unsigned long pfn)
 {
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 	struct mm_struct *mm = vma->vm_mm;
 	struct vm_area_struct *mpnt;
 	struct prio_tree_iter iter;
@@ -127,6 +128,10 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
 		adjust_pte(vma, addr);
 	else
 		flush_cache_page(vma, addr, pfn);
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
+	if (vma->vm_flags & VM_MAYSHARE)
+		adjust_pte(vma, addr);
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 }
 
 /*
-- 
1.5.6.5

[PATCH] "Best-effort" FCSE: choose whether to flush cache at run-time.

[Gilles Chanteperdrix]

The implementation of the FCSE "best-effort" mode requires to decide,
when switching contexts, whether to flush the cache. So, we add a
parameter to the mm switching function telling whether we want to
flush the cache, and the switch_mm macro uses the fcse_needs_flush
to get the value of this parameter.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/cpu-multi32.h |    5 +++--
 arch/arm/include/asm/cpu-single.h  |    3 ++-
 arch/arm/include/asm/fcse.h        |    6 ++++++
 arch/arm/include/asm/mmu_context.h |    2 +-
 arch/arm/include/asm/proc-fns.h    |    3 ++-
 arch/arm/kernel/fcse.c             |   11 +++++++++++
 arch/arm/mm/proc-arm920.S          |    9 +++++++++
 arch/arm/mm/proc-arm926.S          |    9 +++++++++
 arch/arm/mm/proc-xscale.S          |    9 +++++++++
 9 files changed, 52 insertions(+), 5 deletions(-)

diff --git a/arch/arm/include/asm/cpu-multi32.h b/arch/arm/include/asm/cpu-multi32.h
index e2b5b0b..18ca8a0 100644
--- a/arch/arm/include/asm/cpu-multi32.h
+++ b/arch/arm/include/asm/cpu-multi32.h
@@ -52,7 +52,8 @@ extern struct processor {
 	/*
 	 * Set the page table
 	 */
-	void (*switch_mm)(unsigned long pgd_phys, struct mm_struct *mm);
+	void (*switch_mm)(unsigned long pgd_phys,
+			  struct mm_struct *mm, unsigned flush);
 	/*
 	 * Set a possibly extended PTE.  Non-extended PTEs should
 	 * ignore 'ext'.
@@ -66,4 +67,4 @@ extern struct processor {
 #define cpu_do_idle()			processor._do_idle()
 #define cpu_dcache_clean_area(addr,sz)	processor.dcache_clean_area(addr,sz)
 #define cpu_set_pte_ext(ptep,pte,ext)	processor.set_pte_ext(ptep,pte,ext)
-#define cpu_do_switch_mm(pgd,mm)	processor.switch_mm(pgd,mm)
+#define cpu_do_switch_mm(pgd,mm,flush)	processor.switch_mm(pgd,mm,flush)
diff --git a/arch/arm/include/asm/cpu-single.h b/arch/arm/include/asm/cpu-single.h
index f073a6d..ec556e3 100644
--- a/arch/arm/include/asm/cpu-single.h
+++ b/arch/arm/include/asm/cpu-single.h
@@ -39,6 +39,7 @@ extern void cpu_proc_init(void);
 extern void cpu_proc_fin(void);
 extern int cpu_do_idle(void);
 extern void cpu_dcache_clean_area(void *, int);
-extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm);
+extern void cpu_do_switch_mm(unsigned long pgd_phys,
+			     struct mm_struct *mm, unsigned flush);
 extern void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext);
 extern void cpu_reset(unsigned long addr) __attribute__((noreturn));
diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h
index 4e9503d..d769a29 100644
--- a/arch/arm/include/asm/fcse.h
+++ b/arch/arm/include/asm/fcse.h
@@ -66,6 +66,11 @@ fcse_va_to_mva(struct mm_struct *mm, unsigned long va)
 
 int fcse_pid_alloc(void);
 void fcse_pid_free(unsigned pid);
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
+#define fcse_needs_flush(prev, next) (0)
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 
 #else /* ! CONFIG_ARM_FCSE */
 #define fcse_pid_set(pid) do { } while (0)
@@ -73,6 +78,7 @@ void fcse_pid_free(unsigned pid);
 #define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); })
 #define fcse_tlb_mask(mm) mm_cpumask(mm)
 #define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0)
+#define fcse_needs_flush(prev, next) (1)
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index 042a26d..ad1bbb4 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -132,7 +132,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 		fcse_cpu_set_vm_mask(cpu, next);
 		check_context(next);
 		fcse_pid_set(next->context.pid);
-		cpu_switch_mm(next->pgd, next);
+		cpu_switch_mm(next->pgd, next, fcse_needs_flush(prev, next));
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, fcse_tlb_mask(prev));
 	}
diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h
index 3976412..86f3131 100644
--- a/arch/arm/include/asm/proc-fns.h
+++ b/arch/arm/include/asm/proc-fns.h
@@ -239,7 +239,8 @@
 
 #ifdef CONFIG_MMU
 
-#define cpu_switch_mm(pgd,mm) cpu_do_switch_mm(virt_to_phys(pgd),mm)
+#define cpu_switch_mm(pgd,mm,flush) \
+	cpu_do_switch_mm(virt_to_phys(pgd), (mm), (flush))
 
 #define cpu_get_pgd()	\
 	({						\
diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c
index da074b8..7918be6 100644
--- a/arch/arm/kernel/fcse.c
+++ b/arch/arm/kernel/fcse.c
@@ -55,3 +55,14 @@ void fcse_pid_free(unsigned pid)
 	fcse_pid_dereference(pid);
 	spin_unlock_irqrestore(&fcse_lock, flags);
 }
+
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next)
+{
+	unsigned res;
+
+	res = 0;
+
+	return res;
+}
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
diff --git a/arch/arm/mm/proc-arm920.S b/arch/arm/mm/proc-arm920.S
index 914d688..3eff19c 100644
--- a/arch/arm/mm/proc-arm920.S
+++ b/arch/arm/mm/proc-arm920.S
@@ -321,6 +321,11 @@ ENTRY(cpu_arm920_dcache_clean_area)
 ENTRY(cpu_arm920_switch_mm)
 #ifdef CONFIG_MMU
 	mov	ip, #0
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	cmp	r2, #0
+	beq	3f
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH
 	mcr	p15, 0, ip, c7, c6, 0		@ invalidate D cache
 #else
@@ -338,6 +343,10 @@ ENTRY(cpu_arm920_switch_mm)
 #endif
 	mcr	p15, 0, ip, c7, c5, 0		@ invalidate I cache
 	mcr	p15, 0, ip, c7, c10, 4		@ drain WB
+#endif /* !CONFIG_ARM_FCSE_GUARANTEED */
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+3:
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	mcr	p15, 0, r0, c2, c0, 0		@ load page table pointer
 	mcr	p15, 0, ip, c8, c7, 0		@ invalidate I & D TLBs
 #endif
diff --git a/arch/arm/mm/proc-arm926.S b/arch/arm/mm/proc-arm926.S
index 5446693..f7a6021 100644
--- a/arch/arm/mm/proc-arm926.S
+++ b/arch/arm/mm/proc-arm926.S
@@ -337,6 +337,11 @@ ENTRY(cpu_arm926_dcache_clean_area)
 ENTRY(cpu_arm926_switch_mm)
 #ifdef CONFIG_MMU
 	mov	ip, #0
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	cmp	r2, #0
+	beq	2f
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH
 	mcr	p15, 0, ip, c7, c6, 0		@ invalidate D cache
 #else
@@ -346,6 +351,10 @@ ENTRY(cpu_arm926_switch_mm)
 #endif
 	mcr	p15, 0, ip, c7, c5, 0		@ invalidate I cache
 	mcr	p15, 0, ip, c7, c10, 4		@ drain WB
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+2:
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	mcr	p15, 0, r0, c2, c0, 0		@ load page table pointer
 	mcr	p15, 0, ip, c8, c7, 0		@ invalidate I & D TLBs
 #endif
diff --git a/arch/arm/mm/proc-xscale.S b/arch/arm/mm/proc-xscale.S
index 4233942..f09d135 100644
--- a/arch/arm/mm/proc-xscale.S
+++ b/arch/arm/mm/proc-xscale.S
@@ -415,9 +415,18 @@ ENTRY(cpu_xscale_dcache_clean_area)
  */
 	.align	5
 ENTRY(cpu_xscale_switch_mm)
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	cmp	r2, #0
+	beq	2f
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+#ifndef CONFIG_ARM_FCSE_GUARANTEED
 	clean_d_cache r1, r2
 	mcr	p15, 0, ip, c7, c5, 0		@ Invalidate I cache & BTB
 	mcr	p15, 0, ip, c7, c10, 4		@ Drain Write (& Fill) Buffer
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+2:
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	mcr	p15, 0, r0, c2, c0, 0		@ load page table pointer
 	mcr	p15, 0, ip, c8, c7, 0		@ invalidate I & D TLBs
 	cpwait_ret lr, ip
-- 
1.5.6.5

[PATCH] "Best-effort" FCSE: Allow PID re-use.

[Gilles Chanteperdrix]

In addition to the bit telling whether or not a pid is allocated,
maintain:
- a count of processes using a pid;
- the current process using a given pid;
- a bitfield telling whether or not the cache may contain entries for a
given pid.

A function fcse_notify_flush_all() is called upon calls to
flush_cache_mm() and clears all bits in this last bitfield, in order
to avoid that fcse_needs_flush() believe that a pid is reused whereas
the cache has been cleaned.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/cacheflush.h |    4 ++-
 arch/arm/include/asm/fcse.h       |    3 ++
 arch/arm/kernel/fcse.c            |   73 +++++++++++++++++++++++++++++++++++--
 arch/arm/mm/flush.c               |    4 ++-
 4 files changed, 79 insertions(+), 5 deletions(-)

diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h
index db43631..7ced6d6 100644
--- a/arch/arm/include/asm/cacheflush.h
+++ b/arch/arm/include/asm/cacheflush.h
@@ -335,8 +335,10 @@ static inline void outer_flush_range(unsigned long start, unsigned long end)
 #ifndef CONFIG_CPU_CACHE_VIPT
 static inline void flush_cache_mm(struct mm_struct *mm)
 {
-	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)))
+	if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) {
+		fcse_notify_flush_all();
 		__cpuc_flush_user_all();
+	}
 }
 
 static inline void
diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h
index d769a29..c8680c6 100644
--- a/arch/arm/include/asm/fcse.h
+++ b/arch/arm/include/asm/fcse.h
@@ -68,8 +68,10 @@ int fcse_pid_alloc(void);
 void fcse_pid_free(unsigned pid);
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
+void fcse_notify_flush_all(void);
 #else /* CONFIG_ARM_FCSE_GUARANTEED */
 #define fcse_needs_flush(prev, next) (0)
+#define fcse_notify_flush_all() do { } while (0)
 #endif /* CONFIG_ARM_FCSE_GUARANTEED */
 
 #else /* ! CONFIG_ARM_FCSE */
@@ -79,6 +81,7 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
 #define fcse_tlb_mask(mm) mm_cpumask(mm)
 #define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0)
 #define fcse_needs_flush(prev, next) (1)
+#define fcse_notify_flush_all() do { } while (0)
 #endif /* ! CONFIG_ARM_FCSE */
 
 #endif /* __ASM_ARM_FCSE_H */
diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c
index 7918be6..774717c 100644
--- a/arch/arm/kernel/fcse.c
+++ b/arch/arm/kernel/fcse.c
@@ -13,14 +13,33 @@
 static DEFINE_SPINLOCK(fcse_lock);
 static unsigned long fcse_pids_bits[PIDS_LONGS];
 
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+static unsigned long fcse_pids_cache_dirty[PIDS_LONGS];
+static unsigned random_pid;
+struct {
+	struct mm_struct *last_mm;
+	unsigned count;
+} per_pid[NR_PIDS];
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+
 static void fcse_pid_reference_inner(unsigned pid)
 {
-	__set_bit(pid, fcse_pids_bits);
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	if (++per_pid[pid].count == 1)
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
+		__set_bit(pid, fcse_pids_bits);
 }
 
 static void fcse_pid_dereference(unsigned pid)
 {
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	if (--per_pid[pid].count == 0) {
+		__clear_bit(pid, fcse_pids_bits);
+		per_pid[pid].last_mm = NULL;
+	}
+#else /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	__clear_bit(pid, fcse_pids_bits);
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 }
 
 int fcse_pid_alloc(void)
@@ -37,8 +56,14 @@ int fcse_pid_alloc(void)
 		if (!test_bit(0, fcse_pids_bits))
 			pid = 0;
 		else {
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+			if (++random_pid == NR_PIDS)
+				random_pid = 0;
+			pid = random_pid;
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
 			spin_unlock_irqrestore(&fcse_lock, flags);
 			return -EAGAIN;
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 		}
 	}
 	fcse_pid_reference_inner(pid);
@@ -57,12 +82,54 @@ void fcse_pid_free(unsigned pid)
 }
 
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+static void fcse_notify_flush_all_inner(struct mm_struct *next)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fcse_lock, flags);
+	switch (ARRAY_SIZE(fcse_pids_cache_dirty)) {
+	case 4:
+		fcse_pids_cache_dirty[3] = 0UL;
+	case 3:
+		fcse_pids_cache_dirty[2] = 0UL;
+	case 2:
+		fcse_pids_cache_dirty[1] = 0UL;
+	case 1:
+		fcse_pids_cache_dirty[0] = 0UL;
+	}
+	if (next != &init_mm && next) {
+		unsigned pid = next->context.pid >> FCSE_PID_SHIFT;
+		__set_bit(pid, fcse_pids_cache_dirty);
+	}
+	spin_unlock_irqrestore(&fcse_lock, flags);
+}
+
 int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next)
 {
-	unsigned res;
+	unsigned res, reused_pid = 0, pid = next->context.pid >> FCSE_PID_SHIFT;
+	unsigned long flags;
+
+	spin_lock_irqsave(&fcse_lock, flags);
+	if (per_pid[pid].last_mm != next) {
+		if (per_pid[pid].last_mm)
+			reused_pid = test_bit(pid, fcse_pids_cache_dirty);
+		per_pid[pid].last_mm = next;
+	}
+	__set_bit(pid, fcse_pids_cache_dirty);
+	spin_unlock_irqrestore(&fcse_lock, flags);
 
-	res = 0;
+	res = reused_pid;
+
+	if (res) {
+		cpu_clear(smp_processor_id(), prev->cpu_vm_mask);
+		fcse_notify_flush_all_inner(next);
+	}
 
 	return res;
 }
+
+void fcse_notify_flush_all(void)
+{
+	fcse_notify_flush_all_inner(current->mm);
+}
 #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c
index ec6fcda..fba8619 100644
--- a/arch/arm/mm/flush.c
+++ b/arch/arm/mm/flush.c
@@ -50,8 +50,10 @@ static void flush_pfn_alias(unsigned long pfn, unsigned long vaddr)
 void flush_cache_mm(struct mm_struct *mm)
 {
 	if (cache_is_vivt()) {
-		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)))
+		if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) {
+			fcse_notify_flush_all();
 			__cpuc_flush_user_all();
+		}
 		return;
 	}
 
-- 
1.5.6.5

[PATCH] "Best-effort" FCSE: Handle shared mappings.

[Gilles Chanteperdrix]

Account for writable shared mappings, so as to be able to flush cache
when switching out a process with such a mapping.

This accounting is done by adding the L_PTE_SHARED flag to page
belonging to such mappings in make_coherent(), then, set_pte_at() is
intercepted to account for both addition and removal of pages with
this flag set.

Note that using the L_PTE_SHARED flag makes the FCSE patch
incompatible with ARMv6+ and Xscale3, but the FCSE feature is
deprecated in ARMv6 anyway.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/mmu.h         |    3 ++
 arch/arm/include/asm/mmu_context.h |    6 ++++
 arch/arm/include/asm/pgtable.h     |   42 ++++++++++++++++++++++++--
 arch/arm/kernel/fcse.c             |    3 +-
 arch/arm/mm/fault-armv.c           |   55 +++++++++++++++++++++++++++++++++++-
 5 files changed, 103 insertions(+), 6 deletions(-)

diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h
index 7472598..9a2fb51 100644
--- a/arch/arm/include/asm/mmu.h
+++ b/arch/arm/include/asm/mmu.h
@@ -10,6 +10,9 @@ typedef struct {
 #ifdef CONFIG_ARM_FCSE
 	unsigned long pid;
 	cpumask_t cpu_tlb_mask;
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	unsigned shared_dirty_pages;
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 #endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
 } mm_context_t;
diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index ad1bbb4..035a9f2 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -76,6 +76,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 	int pid;
 
 	cpus_clear(mm->context.cpu_tlb_mask);
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	mm->context.shared_dirty_pages = 0;
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 
 	pid = fcse_pid_alloc();
 	if (pid < 0)
@@ -91,6 +94,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 static inline void destroy_context(struct mm_struct *mm)
 {
 #ifdef CONFIG_ARM_FCSE
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	BUG_ON(mm->context.shared_dirty_pages);
+#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT);
 #endif /* CONFIG_ARM_FCSE */
 }
diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index 197c596..7775c1b 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -250,6 +250,33 @@ extern pgprot_t		pgprot_kernel;
 #define __S111  __PAGE_SHARED_EXEC
 
 #ifndef __ASSEMBLY__
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+#define fcse_account_page_removal(mm, addr, val) do {		\
+	struct mm_struct *_mm = (mm);				\
+	unsigned long _addr = (addr);				\
+	unsigned long _val = (val);				\
+	if (pte_present(_val) && ((_val) & L_PTE_SHARED))	\
+		--_mm->context.shared_dirty_pages;		\
+} while (0)
+
+#define fcse_account_page_addition(mm, addr, val) ({			\
+	struct mm_struct *_mm = (mm);					\
+	unsigned long _addr = (addr);					\
+	unsigned long _val = (val);					\
+	if (pte_present(_val) && (_val & L_PTE_SHARED)) {		\
+		if ((_val & (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \
+		    != (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY))	\
+			_val &= ~L_PTE_SHARED;				\
+		else							\
+			++_mm->context.shared_dirty_pages;		\
+	}								\
+	_val;								\
+})
+#else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */
+#define fcse_account_page_removal(mm, addr, val) do { } while (0)
+#define fcse_account_page_addition(mm, addr, val) (val)
+#endif /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */
+
 /*
  * ZERO_PAGE is a global shared page that is always zero: used
  * for zero-mapped memory areas etc..
@@ -261,7 +288,10 @@ extern struct page *empty_zero_page;
 #define pfn_pte(pfn,prot)	(__pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot)))
 
 #define pte_none(pte)		(!pte_val(pte))
-#define pte_clear(mm,addr,ptep)	set_pte_ext(ptep, __pte(0), 0)
+#define pte_clear(mm,addr,ptep)	do {				\
+	fcse_account_page_removal(mm, addr, pte_val(*ptep));	\
+	set_pte_ext(ptep, __pte(0), 0);				\
+} while (0)
 #define pte_page(pte)		(pfn_to_page(pte_pfn(pte)))
 #define pte_offset_kernel(dir,addr)	(pmd_page_vaddr(*(dir)) + __pte_index(addr))
 
@@ -280,9 +310,13 @@ extern struct page *empty_zero_page;
 
 #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext)
 
-#define set_pte_at(mm,addr,ptep,pteval) do { \
-	set_pte_ext(ptep, pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \
- } while (0)
+#define set_pte_at(mm,addr,ptep,pteval) do {				\
+	unsigned long _pteval = (pteval);				\
+	fcse_account_page_removal(mm, addr, pte_val(*ptep));		\
+	pte_val(_pteval) =						\
+		fcse_account_page_addition(mm, addr, pte_val(_pteval)); \
+	set_pte_ext(ptep, _pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \
+} while (0)
 
 /*
  * The following only work if pte_present() is true.
diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c
index 774717c..5b257c9 100644
--- a/arch/arm/kernel/fcse.c
+++ b/arch/arm/kernel/fcse.c
@@ -118,7 +118,8 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next)
 	__set_bit(pid, fcse_pids_cache_dirty);
 	spin_unlock_irqrestore(&fcse_lock, flags);
 
-	res = reused_pid;
+	res = reused_pid
+		|| prev->context.shared_dirty_pages;
 
 	if (res) {
 		cpu_clear(smp_processor_id(), prev->cpu_vm_mask);
diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c
index 40bccb9..3135a6b 100644
--- a/arch/arm/mm/fault-armv.c
+++ b/arch/arm/mm/fault-armv.c
@@ -25,6 +25,57 @@
 
 static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE;
 
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+static void fcse_set_pte_shared(struct vm_area_struct *vma,
+				unsigned long address)
+{
+	pgd_t *pgd;
+	pmd_t *pmd;
+	pte_t *pte, entry;
+
+	if (!(vma->vm_flags & VM_MAYSHARE) || address >= TASK_SIZE)
+		return;
+
+	pgd = pgd_offset(vma->vm_mm, address);
+	if (pgd_none(*pgd))
+		goto no_pgd;
+	if (pgd_bad(*pgd))
+		goto bad_pgd;
+
+	pmd = pmd_offset(pgd, address);
+	if (pmd_none(*pmd))
+		goto no_pmd;
+	if (pmd_bad(*pmd))
+		goto bad_pmd;
+
+	pte = pte_offset_map(pmd, address);
+	entry = *pte;
+
+	if ((pte_val(entry)
+	     & (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY))
+	    == (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) {
+		pte_val(entry) |= L_PTE_SHARED;
+		set_pte_at(vma->vm_mm, address, pte, entry);
+	}
+	pte_unmap(pte);
+	return;
+
+bad_pgd:
+	pgd_ERROR(*pgd);
+	pgd_clear(pgd);
+no_pgd:
+	return;
+
+bad_pmd:
+	pmd_ERROR(*pmd);
+	pmd_clear(pmd);
+no_pmd:
+	return;
+}
+#else /* !CONFIG_ARM_FCSE_BEST_EFFORT */
+#define fcse_set_pte_shared(vma, addr) do { } while (0)
+#endif /* !CONFIG_ARM_FCSE_BEST_EFFORT */
+
 /*
  * We take the easy way out of this problem - we make the
  * PTE uncacheable.  However, we leave the write buffer on.
@@ -126,8 +177,10 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
 	flush_dcache_mmap_unlock(mapping);
 	if (aliases)
 		adjust_pte(vma, addr);
-	else
+	else {
+		fcse_set_pte_shared(vma, addr);
 		flush_cache_page(vma, addr, pfn);
+	}
 #else /* CONFIG_ARM_FCSE_GUARANTEED */
 	if (vma->vm_flags & VM_MAYSHARE)
 		adjust_pte(vma, addr);
-- 
1.5.6.5

[PATCH] "Best-effort" FCSE: Handle mappings above 32 MB

[Gilles Chanteperdrix]

In case an address larger than 32 MB is returned by
arch_get_unmapped_area(), relocate the calling process to the null
pid. When such a process forks, also give the child process a null
pid. Account for pages above 32 MB, and when a process with such pages
is switched in or out, flush the cache.

As in handling shared mappings, we use set_pte_at() to do the pages
accounting.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/include/asm/fcse.h        |    2 +
 arch/arm/include/asm/mmu.h         |    2 +
 arch/arm/include/asm/mmu_context.h |   17 +++++++++++++-
 arch/arm/include/asm/pgtable.h     |    7 ++++++
 arch/arm/kernel/fcse.c             |   41 +++++++++++++++++++++++++++++++++++-
 arch/arm/mm/mmap.c                 |    7 ++++++
 6 files changed, 73 insertions(+), 3 deletions(-)

diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h
index c8680c6..66fa9de 100644
--- a/arch/arm/include/asm/fcse.h
+++ b/arch/arm/include/asm/fcse.h
@@ -69,6 +69,8 @@ void fcse_pid_free(unsigned pid);
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next);
 void fcse_notify_flush_all(void);
+void fcse_pid_reference(unsigned pid);
+void fcse_relocate_mm_to_null_pid(struct mm_struct *mm);
 #else /* CONFIG_ARM_FCSE_GUARANTEED */
 #define fcse_needs_flush(prev, next) (0)
 #define fcse_notify_flush_all() do { } while (0)
diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h
index 9a2fb51..13c844f 100644
--- a/arch/arm/include/asm/mmu.h
+++ b/arch/arm/include/asm/mmu.h
@@ -12,6 +12,8 @@ typedef struct {
 	cpumask_t cpu_tlb_mask;
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 	unsigned shared_dirty_pages;
+	unsigned big;
+	unsigned high_pages;
 #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 #endif /* CONFIG_ARM_FCSE */
 	unsigned int kvm_seq;
diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index 035a9f2..85eac0a 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -77,13 +77,25 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 
 	cpus_clear(mm->context.cpu_tlb_mask);
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+	if (!mm->context.big) {
+		pid = fcse_pid_alloc();
+		mm->context.pid = pid << FCSE_PID_SHIFT;
+	} else {
+		/* We are normally forking a process vith a virtual address
+		   space larger than 32 MB, so its pid should be 0. */
+		BUG_ON(mm->context.pid);
+		fcse_pid_reference(0);
+	}
+	/* If we are forking, set_pte_at will restore the correct high pages
+	   count, and shared writable pages are write-protected again. */
 	mm->context.shared_dirty_pages = 0;
-#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
-
+	mm->context.high_pages = 0;
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
 	pid = fcse_pid_alloc();
 	if (pid < 0)
 		return pid;
 	mm->context.pid = pid << FCSE_PID_SHIFT;
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 #endif /* CONFIG_ARM_FCSE */
 
 	return 0;
@@ -96,6 +108,7 @@ static inline void destroy_context(struct mm_struct *mm)
 #ifdef CONFIG_ARM_FCSE
 #ifdef CONFIG_ARM_FCSE_BEST_EFFORT
 	BUG_ON(mm->context.shared_dirty_pages);
+	BUG_ON(mm->context.high_pages);
 #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
 	fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT);
 #endif /* CONFIG_ARM_FCSE */
diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index 7775c1b..6c79d71 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -257,6 +257,10 @@ extern pgprot_t		pgprot_kernel;
 	unsigned long _val = (val);				\
 	if (pte_present(_val) && ((_val) & L_PTE_SHARED))	\
 		--_mm->context.shared_dirty_pages;		\
+	if (pte_present(_val) && _addr < TASK_SIZE) {		\
+		if (_addr >= FCSE_TASK_SIZE)			\
+			--_mm->context.high_pages;		\
+	}							\
 } while (0)
 
 #define fcse_account_page_addition(mm, addr, val) ({			\
@@ -270,6 +274,9 @@ extern pgprot_t		pgprot_kernel;
 		else							\
 			++_mm->context.shared_dirty_pages;		\
 	}								\
+	if (pte_present(_val)						\
+	    && _addr < TASK_SIZE && _addr >= FCSE_TASK_SIZE)		\
+		++_mm->context.high_pages;				\
 	_val;								\
 })
 #else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */
diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c
index 5b257c9..d15ac30 100644
--- a/arch/arm/kernel/fcse.c
+++ b/arch/arm/kernel/fcse.c
@@ -119,7 +119,9 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next)
 	spin_unlock_irqrestore(&fcse_lock, flags);
 
 	res = reused_pid
-		|| prev->context.shared_dirty_pages;
+		|| prev->context.shared_dirty_pages
+		|| prev->context.high_pages
+		|| next->context.high_pages;
 
 	if (res) {
 		cpu_clear(smp_processor_id(), prev->cpu_vm_mask);
@@ -133,4 +135,41 @@ void fcse_notify_flush_all(void)
 {
 	fcse_notify_flush_all_inner(current->mm);
 }
+
+void fcse_pid_reference(unsigned pid)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fcse_lock, flags);
+	fcse_pid_reference_inner(pid);
+	spin_unlock_irqrestore(&fcse_lock, flags);
+}
+
+/* Called with mm->mmap_sem write-locked. */
+void fcse_relocate_mm_to_null_pid(struct mm_struct *mm)
+{
+	pgd_t *to = mm->pgd + pgd_index(0);
+	pgd_t *from = pgd_offset(mm, 0);
+	unsigned len = pgd_index(FCSE_TASK_SIZE) * sizeof(*from);
+	unsigned long flags;
+
+	preempt_disable();
+
+	memcpy(to, from, len);
+	spin_lock_irqsave(&fcse_lock, flags);
+	fcse_pid_dereference(mm->context.pid >> FCSE_PID_SHIFT);
+	fcse_pid_reference_inner(0);
+	per_pid[0].last_mm = mm;
+	spin_unlock_irqrestore(&fcse_lock, flags);
+
+	mm->context.pid = 0;
+	fcse_pid_set(0);
+	memset(from, '\0', len);
+	mb();
+	flush_cache_mm(mm);
+	flush_tlb_mm(mm);
+
+	preempt_enable();
+}
+
 #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */
diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c
index e608bda..ac81369 100644
--- a/arch/arm/mm/mmap.c
+++ b/arch/arm/mm/mmap.c
@@ -125,7 +125,14 @@ found_addr:
 			mm->cached_hole_size = 0;
 			goto full_search;
 		}
+#ifdef CONFIG_ARM_FCSE_BEST_EFFORT
+		if (mm->context.pid) {
+			mm->context.big = 1;
+			fcse_relocate_mm_to_null_pid(mm);
+		}
+#else /* CONFIG_ARM_FCSE_GUARANTEED */
 		return -ENOMEM;
+#endif /* CONFIG_ARM_FCSE_GUARANTEED */
 	}
 #endif /* CONFIG_ARM_FCSE */
 	return addr;
-- 
1.5.6.5

[PATCH] FCSE: add Makefile and compilation option.

[Gilles Chanteperdrix]

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
---
 arch/arm/kernel/Makefile |    1 +
 arch/arm/mm/Kconfig      |   45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 0 deletions(-)

diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile
index 79087dd..574e77e 100644
--- a/arch/arm/kernel/Makefile
+++ b/arch/arm/kernel/Makefile
@@ -52,5 +52,6 @@ endif
 
 head-y			:= head$(MMUEXT).o
 obj-$(CONFIG_DEBUG_LL)	+= debug.o
+obj-$(CONFIG_ARM_FCSE)	+= fcse.o
 
 extra-y := $(head-y) init_task.o vmlinux.lds
diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index 8d43e58..fb9a4a5 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -776,3 +776,48 @@ config ARM_L1_CACHE_SHIFT
 	int
 	default 6 if ARCH_OMAP3
 	default 5
+
+config ARM_FCSE
+	bool "Fast Context Switch Extension (EXPERIMENTAL)"
+	depends on EXPERIMENTAL && !SMP && !CPU_32v6 && !CPU_32v7
+	default n
+	help
+	  The Fast Context Switch Extension (FCSE for short) is an extension of
+	  some ARM processors which allows to switch contexts between processes
+	  without flushing cache and saves a few tens of microseconds in the
+	  worst case.
+
+	  Enabling this option makes linux use the FCSE.
+
+	  We propose two modes:
+	  - the guaranteed mode: we guarantee that there will never be any cache
+	    flush when switching context, but this means that there can not be
+	    more than 95 running processes in the system, each with a virtual
+	    memory space smaller than 32MB, and that the shared memory
+	    mappings do not use cache;
+	  - the best effort mode: we allow some cache flushes to happen from
+	    time to time, but do not limit the number of processes or the
+	    virtual memory space available for each process, and the shared
+	    memory mappings use cache.
+
+if ARM_FCSE
+
+choice
+	prompt "FCSE mode"
+	default ARM_FCSE_BEST_EFFORT
+	help
+	  This option allow setting which FCSE mode will be used.
+
+config ARM_FCSE_GUARANTEED
+	bool "guaranteed"
+	help
+	  Select guaranteed mode.
+
+config ARM_FCSE_BEST_EFFORT
+	bool "best effort"
+	help
+	  Select best-effort mode.
+
+endchoice
+
+endif
-- 
1.5.6.5

[PATCH] FCSE: pid life cycle.

[Pavel Machek]

On Thu 2009-10-01 23:34:11, Gilles Chanteperdrix wrote:
> Add a pid member to the mm_context_t structure, allocate this pid in
> the init_new_context, free it in the destroy_context, switch it in
> switch_mm.
> 
> A bitfield is maintained to know what pids are currently in use.
> 
> Signed-off-by: Richard Cochran <richard.cochran@omicron.at>
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
...
> --- /dev/null
> +++ b/arch/arm/include/asm/fcse.h
> @@ -0,0 +1,44 @@
> +/*
> + * Filename:    arch/arm/include/asm/fcse.h
> + * Description: ARM Process ID (PID) includes for Fast Address Space Switching
> + *              (FASS) in ARM Linux.
> + * Created:     14/10/2001
> + * Changes:     19/02/2002 - Macros added.
> + *              03/08/2007 - Adapted to kernel 2.6.21 (ssm)
> + *              Feb 2008   - Simplified a bit (rco)

We normally avoid changelogs in sources.

> +/* Sets the CPU's PID Register */
> +static inline void fcse_pid_set(unsigned long pid)
> +{
> +	__asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0"
> +			      : /* */: "r" (pid) : "memory");
> +}

Calling this 'pid' is extremely confusing. PID already means process
id...



-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[PATCH] FCSE: pid life cycle.

[Gilles Chanteperdrix]

Pavel Machek wrote:
> On Thu 2009-10-01 23:34:11, Gilles Chanteperdrix wrote:
>> Add a pid member to the mm_context_t structure, allocate this pid in
>> the init_new_context, free it in the destroy_context, switch it in
>> switch_mm.
>>
>> A bitfield is maintained to know what pids are currently in use.
>>
>> Signed-off-by: Richard Cochran <richard.cochran@omicron.at>
>> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> ...
>> --- /dev/null
>> +++ b/arch/arm/include/asm/fcse.h
>> @@ -0,0 +1,44 @@
>> +/*
>> + * Filename:    arch/arm/include/asm/fcse.h
>> + * Description: ARM Process ID (PID) includes for Fast Address Space Switching
>> + *              (FASS) in ARM Linux.
>> + * Created:     14/10/2001
>> + * Changes:     19/02/2002 - Macros added.
>> + *              03/08/2007 - Adapted to kernel 2.6.21 (ssm)
>> + *              Feb 2008   - Simplified a bit (rco)
> 
> We normally avoid changelogs in sources.

Ok. Will remove that, it was inherited.

> 
>> +/* Sets the CPU's PID Register */
>> +static inline void fcse_pid_set(unsigned long pid)
>> +{
>> +	__asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0"
>> +			      : /* */: "r" (pid) : "memory");
>> +}
> 
> Calling this 'pid' is extremely confusing. PID already means process
> id...

Guess what, it also means "process id" in the FCSE case, and it is the
name it has in the documentation, I always find a good idea to name the
hardware registers as per documentation. I guess the original intent was
to put the real process id into the hardware register, but that is not
really possible with Linux. So, I guess we can call it fcse_pid.

In any case, I will keep all modifications for a second post once I had
sufficient review (namely, I am expecting Russell's review, since he is
the one who will accept or reject the patch in the end).

-- 
					    Gilles.