* [PATCH] FCSE @ 2009-10-01 21:34 Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: pid life cycle Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Hi, Here comes the FCSE patch for Linux 2.6.32-rc1. To summarize a bit the story of this patch which you will find in various posts on this list, this patch adds an option to the kernel configuration which allows using the ARM FCSE in one of two modes: - the guaranteed mode, where context switches are guaranteed to run without flushing the caches, but where the number of processes is limited to 95 with a virtual memory space limited to 32 MB; - the best-effort mode, where the limitations are removed, but where the cache is flushed when needed during context switches. The patch can be considered as relatively stable, it has not changed much since Linux 2.6.27, and has been validated by running LTP with the best-effort mode, with no differences in the LTP results compared to an unpatched kernel. Furthermore, it has been reported to work in the field. As for the performance improvement, running the hackbench test with 400 processes (so, well above the 95 processes limit) on an AT91RM9200 showed an improvement in the runing time which can be visualized here: http://sisyphus.hd.free.fr/~gilles/hackbench.png. The user-space worst-case scheduling latency is also improved by the guaranteed mode, but this only interests people running an RT kernel of any kind. The patch is relatively simple and only modifies ARM-specific code, but has been split to ease review, as I have been advised off-list to do. Some parts of it, however, could be implemented more elegantly (to say the least), especially patches 8 and 9 which intercept set_pte_at() to count pages belonging to some types of mappings whereas counting mappings would be enough but would probably require to modify generic code. Thanks in advance for your review. Regards -- Gilles. ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH] FCSE: pid life cycle. 2009-10-01 21:34 [PATCH] FCSE Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Conversions between VA and MVA Gilles Chanteperdrix 2009-10-04 19:02 ` [PATCH] FCSE: pid life cycle Pavel Machek 0 siblings, 2 replies; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Add a pid member to the mm_context_t structure, allocate this pid in the init_new_context, free it in the destroy_context, switch it in switch_mm. A bitfield is maintained to know what pids are currently in use. Signed-off-by: Richard Cochran <richard.cochran@omicron.at> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/fcse.h | 44 +++++++++++++++++++++++++++ arch/arm/include/asm/mmu.h | 3 ++ arch/arm/include/asm/mmu_context.h | 24 ++++++++++++++- arch/arm/kernel/fcse.c | 57 ++++++++++++++++++++++++++++++++++++ 4 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 arch/arm/include/asm/fcse.h create mode 100644 arch/arm/kernel/fcse.c diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h new file mode 100644 index 0000000..b44e866 --- /dev/null +++ b/arch/arm/include/asm/fcse.h @@ -0,0 +1,44 @@ +/* + * Filename: arch/arm/include/asm/fcse.h + * Description: ARM Process ID (PID) includes for Fast Address Space Switching + * (FASS) in ARM Linux. + * Created: 14/10/2001 + * Changes: 19/02/2002 - Macros added. + * 03/08/2007 - Adapted to kernel 2.6.21 (ssm) + * Feb 2008 - Simplified a bit (rco) + * + * Copyright: (C) 2001, 2002 Adam Wiggins <awiggins@cse.unsw.edu.au> + * (C) 2007 Sebastian Smolorz <ssm@emlix.com> + * (C) 2008 Richard Cochran + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of teh GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#ifndef __ASM_ARM_FCSE_H +#define __ASM_ARM_FCSE_H + +#ifdef CONFIG_ARM_FCSE + +#include <linux/mm_types.h> /* For struct mm_struct */ + +#define FCSE_PID_SHIFT 25 + +/* Size of PID relocation area */ +#define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT) + +/* Sets the CPU's PID Register */ +static inline void fcse_pid_set(unsigned long pid) +{ + __asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0" + : /* */: "r" (pid) : "memory"); +} + +int fcse_pid_alloc(void); +void fcse_pid_free(unsigned pid); + +#else /* ! CONFIG_ARM_FCSE */ +#define fcse_pid_set(pid) do { } while (0) +#endif /* ! CONFIG_ARM_FCSE */ + +#endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index b561584..4a5a8db 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -7,6 +7,9 @@ typedef struct { #ifdef CONFIG_CPU_HAS_ASID unsigned int id; #endif +#ifdef CONFIG_ARM_FCSE + unsigned long pid; +#endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; } mm_context_t; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index de6cefb..ec7a84d 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -19,6 +19,7 @@ #include <asm/cachetype.h> #include <asm/proc-fns.h> #include <asm-generic/mm_hooks.h> +#include <asm/fcse.h> void __check_kvm_seq(struct mm_struct *mm); @@ -68,11 +69,29 @@ static inline void check_context(struct mm_struct *mm) #endif } -#define init_new_context(tsk,mm) 0 +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE + int pid; + + pid = fcse_pid_alloc(); + if (pid < 0) + return pid; + mm->context.pid = pid << FCSE_PID_SHIFT; +#endif /* CONFIG_ARM_FCSE */ + + return 0; +} #endif -#define destroy_context(mm) do { } while(0) +static inline void destroy_context(struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE + fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT); +#endif /* CONFIG_ARM_FCSE */ +} /* * This is called when "tsk" is about to enter lazy TLB mode. @@ -109,6 +128,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, #endif if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) { check_context(next); + fcse_pid_set(next->context.pid); cpu_switch_mm(next->pgd, next); if (cache_is_vivt()) cpumask_clear_cpu(cpu, mm_cpumask(prev)); diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c new file mode 100644 index 0000000..da074b8 --- /dev/null +++ b/arch/arm/kernel/fcse.c @@ -0,0 +1,57 @@ +#include <linux/bitops.h> +#include <linux/memory.h> +#include <linux/spinlock.h> +#include <linux/mm.h> + +#include <asm/fcse.h> +#include <asm/cacheflush.h> +#include <asm/tlbflush.h> + +#define NR_PIDS (TASK_SIZE / FCSE_PID_TASK_SIZE) +#define PIDS_LONGS ((NR_PIDS + BITS_PER_LONG - 1) / BITS_PER_LONG) + +static DEFINE_SPINLOCK(fcse_lock); +static unsigned long fcse_pids_bits[PIDS_LONGS]; + +static void fcse_pid_reference_inner(unsigned pid) +{ + __set_bit(pid, fcse_pids_bits); +} + +static void fcse_pid_dereference(unsigned pid) +{ + __clear_bit(pid, fcse_pids_bits); +} + +int fcse_pid_alloc(void) +{ + unsigned long flags; + unsigned pid; + + spin_lock_irqsave(&fcse_lock, flags); + pid = find_next_zero_bit(fcse_pids_bits, NR_PIDS, 1); + if (pid == NR_PIDS) { + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ + if (!test_bit(0, fcse_pids_bits)) + pid = 0; + else { + spin_unlock_irqrestore(&fcse_lock, flags); + return -EAGAIN; + } + } + fcse_pid_reference_inner(pid); + spin_unlock_irqrestore(&fcse_lock, flags); + + return pid; +} + +void fcse_pid_free(unsigned pid) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_dereference(pid); + spin_unlock_irqrestore(&fcse_lock, flags); +} -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] FCSE: Conversions between VA and MVA. 2009-10-01 21:34 ` [PATCH] FCSE: pid life cycle Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Address-space limits Gilles Chanteperdrix 2009-10-04 19:02 ` [PATCH] FCSE: pid life cycle Pavel Machek 1 sibling, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Add to asm/fcse.h the functions needed to allow conversion between virtual addresses (a.k.a VA) and modified virtual addresses (a.k.a MVA), that is virtual addresses translated by the FCSE pid. Though the whole kernel handles VA, and do not see MVA, there are a few exceptions: - the function handling page table walking must use MVA, this is solved by getting the pgd_offset function to use fcse_va_to_mva, which turns out to be sufficient; - the TLB and cache operations which may be passed user-space addresses must use MVA, so, they call fcse_va_to_mva too; - the function do_DataAbort is passed an MVA which has to be converted into a VA, this is done by calling fcse_mva_to_va. Signed-off-by: Richard Cochran <richard.cochran@omicron.at> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/cacheflush.h | 26 ++++++++++++++++++++------ arch/arm/include/asm/fcse.h | 29 +++++++++++++++++++++++++++++ arch/arm/include/asm/pgtable.h | 10 ++++++++-- arch/arm/include/asm/tlbflush.h | 14 ++++++++++++-- arch/arm/mm/fault.c | 6 ++++++ arch/arm/mm/flush.c | 15 ++++++++++----- arch/arm/mm/pgd.c | 7 ++++++- 7 files changed, 91 insertions(+), 16 deletions(-) diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h index fd03fb6..db43631 100644 --- a/arch/arm/include/asm/cacheflush.h +++ b/arch/arm/include/asm/cacheflush.h @@ -15,6 +15,7 @@ #include <asm/glue.h> #include <asm/shmparam.h> #include <asm/cachetype.h> +#include <asm/fcse.h> #define CACHE_COLOUR(vaddr) ((vaddr & (SHMLBA - 1)) >> PAGE_SHIFT) @@ -341,16 +342,20 @@ static inline void flush_cache_mm(struct mm_struct *mm) static inline void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) + if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { + start = fcse_va_to_mva(vma->vm_mm, start); + end = fcse_va_to_mva(vma->vm_mm, end); __cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end), vma->vm_flags); + } } static inline void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn) { if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { - unsigned long addr = user_addr & PAGE_MASK; + unsigned long addr; + addr = fcse_va_to_mva(vma->vm_mm, user_addr) & PAGE_MASK; __cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags); } } @@ -381,14 +386,22 @@ extern void flush_ptrace_access(struct vm_area_struct *vma, struct page *page, * Harvard caches are synchronised for the user space address range. * This is used for the ARM private sys_cacheflush system call. */ -#define flush_cache_user_range(vma,start,end) \ - __cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end)) +#define flush_cache_user_range(vma, start, end) \ + ({ \ + struct mm_struct *_mm = (vma)->vm_mm; \ + unsigned long _start, _end; \ + _start = fcse_va_to_mva(_mm, start) & PAGE_MASK; \ + _end = PAGE_ALIGN(fcse_va_to_mva(_mm, end)); \ + __cpuc_coherent_user_range(_start, _end); \ + }) /* * Perform necessary cache operations to ensure that data previously * stored within this range of addresses can be executed by the CPU. */ -#define flush_icache_range(s,e) __cpuc_coherent_kern_range(s,e) +#define flush_icache_range(s,e) \ + __cpuc_coherent_kern_range(fcse_va_to_mva(current->mm, (s)), \ + fcse_va_to_mva(current->mm, (e))) /* * Perform necessary cache operations to ensure that the TLB will @@ -426,7 +439,8 @@ static inline void flush_anon_page(struct vm_area_struct *vma, extern void __flush_anon_page(struct vm_area_struct *vma, struct page *, unsigned long); if (PageAnon(page)) - __flush_anon_page(vma, page, vmaddr); + __flush_anon_page(vma, page, + fcse_va_to_mva(vma->vm_mm, vmaddr)); } #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h index b44e866..e443278 100644 --- a/arch/arm/include/asm/fcse.h +++ b/arch/arm/include/asm/fcse.h @@ -27,6 +27,9 @@ /* Size of PID relocation area */ #define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT) +/* Mask to get rid of PID from relocated address */ +#define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1) + /* Sets the CPU's PID Register */ static inline void fcse_pid_set(unsigned long pid) { @@ -34,11 +37,37 @@ static inline void fcse_pid_set(unsigned long pid) : /* */: "r" (pid) : "memory"); } +/* Returns the state of the CPU's PID Register */ +static inline unsigned long fcse_pid_get(void) +{ + unsigned long pid; + __asm__ __volatile__("mrc p15, 0, %0, c13, c0, 0" : "=&r" (pid)); + return pid & ~FCSE_PID_MASK; +} + +static inline unsigned long fcse_mva_to_va(unsigned long mva) +{ + unsigned long pid = fcse_pid_get(); + if (pid && (pid == (mva & ~FCSE_PID_MASK))) + return mva & FCSE_PID_MASK; + return mva; +} + +static inline unsigned long +fcse_va_to_mva(struct mm_struct *mm, unsigned long va) +{ + if (va < FCSE_PID_TASK_SIZE) + return mm->context.pid | va; + return va; +} + int fcse_pid_alloc(void); void fcse_pid_free(unsigned pid); #else /* ! CONFIG_ARM_FCSE */ #define fcse_pid_set(pid) do { } while (0) +#define fcse_mva_to_va(x) (x) +#define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); }) #endif /* ! CONFIG_ARM_FCSE */ #endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 201ccaa..197c596 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -112,6 +112,8 @@ #define LIBRARY_TEXT_START 0x0c000000 #ifndef __ASSEMBLY__ +#include <asm/fcse.h> + extern void __pte_error(const char *file, int line, unsigned long val); extern void __pmd_error(const char *file, int line, unsigned long val); extern void __pgd_error(const char *file, int line, unsigned long val); @@ -362,10 +364,14 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd) /* to find an entry in a page-table-directory */ #define pgd_index(addr) ((addr) >> PGDIR_SHIFT) -#define pgd_offset(mm, addr) ((mm)->pgd+pgd_index(addr)) +#define pgd_offset(mm, addr) \ + ({ \ + struct mm_struct *_mm = (mm); \ + (_mm->pgd + pgd_index(fcse_va_to_mva(_mm, (addr)))); \ + }) /* to find an entry in a kernel page-table-directory */ -#define pgd_offset_k(addr) pgd_offset(&init_mm, addr) +#define pgd_offset_k(addr) (init_mm.pgd + pgd_index(addr)) /* Find an entry in the second-level page table.. */ #define pmd_offset(dir, addr) ((pmd_t *)(dir)) diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h index a45ab5d..7e14a84 100644 --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -210,6 +210,7 @@ #ifndef __ASSEMBLY__ #include <linux/sched.h> +#include <asm/fcse.h> struct cpu_tlb_fns { void (*flush_user_range)(unsigned long, unsigned long, struct vm_area_struct *); @@ -383,7 +384,8 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) const int zero = 0; const unsigned int __tlb_flag = __cpu_tlb_flags; - uaddr = (uaddr & PAGE_MASK) | ASID(vma->vm_mm); + uaddr = (fcse_va_to_mva(vma->vm_mm, uaddr) & PAGE_MASK) + | ASID(vma->vm_mm); if (tlb_flag(TLB_WB)) dsb(); @@ -504,7 +506,15 @@ static inline void clean_pmd_entry(pmd_t *pmd) /* * Convert calls to our calling convention. */ -#define local_flush_tlb_range(vma,start,end) __cpu_flush_user_tlb_range(start,end,vma) +#define local_flush_tlb_range(vma, start, end) \ + ({ \ + struct mm_struct *_mm = (vma)->vm_mm; \ + unsigned long _start, _end; \ + _start = fcse_va_to_mva(_mm, start); \ + _end = fcse_va_to_mva(_mm, end); \ + __cpu_flush_user_tlb_range(_start, _end, vma); \ + }) + #define local_flush_tlb_kernel_range(s,e) __cpu_flush_kern_tlb_range(s,e) #ifndef CONFIG_SMP diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index 379f785..493c980 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -73,6 +73,10 @@ void show_pte(struct mm_struct *mm, unsigned long addr) if (!mm) mm = &init_mm; +#ifdef CONFIG_ARM_FCSE + printk(KERN_ALERT "fcse pid: %ld, 0x%08lx\n", + mm->context.pid >> FCSE_PID_SHIFT, mm->context.pid); +#endif /* CONFIG_ARM_FCSE */ printk(KERN_ALERT "pgd = %p\n", mm->pgd); pgd = pgd_offset(mm, addr); printk(KERN_ALERT "[%08lx] *pgd=%08lx", addr, pgd_val(*pgd)); @@ -506,6 +510,8 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) const struct fsr_info *inf = fsr_info + fsr_fs(fsr); struct siginfo info; + addr = fcse_mva_to_va(addr); + if (!inf->fn(addr, fsr & ~FSR_LNX_PF, regs)) return; diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c index b279429..ec6fcda 100644 --- a/arch/arm/mm/flush.c +++ b/arch/arm/mm/flush.c @@ -73,9 +73,11 @@ void flush_cache_mm(struct mm_struct *mm) void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { if (cache_is_vivt()) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) - __cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end), - vma->vm_flags); + if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { + start = fcse_va_to_mva(vma->vm_mm, start) & PAGE_MASK; + end = PAGE_ALIGN(fcse_va_to_mva(vma->vm_mm, end)); + __cpuc_flush_user_range(start, end, vma->vm_flags); + } return; } @@ -98,8 +100,11 @@ void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsig { if (cache_is_vivt()) { if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { - unsigned long addr = user_addr & PAGE_MASK; - __cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags); + unsigned long addr; + addr = fcse_va_to_mva(vma->vm_mm, user_addr); + addr &= PAGE_MASK; + __cpuc_flush_user_range(addr, addr + PAGE_SIZE, + vma->vm_flags); } return; } diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 2690146..9796798 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -43,6 +43,11 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) clean_dcache_area(new_pgd, PTRS_PER_PGD * sizeof(pgd_t)); if (!vectors_high()) { +#ifdef CONFIG_ARM_FCSE + /* FCSE does not work without high vectors. */ + BUG(); +#endif /* CONFIG_ARM_FCSE */ + /* * On ARM, first page must always be allocated since it * contains the machine vectors. @@ -81,7 +86,7 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) return; /* pgd is always present and good */ - pmd = pmd_off(pgd, 0); + pmd = pmd_off(pgd + pgd_index(fcse_va_to_mva(mm, 0)), 0); if (pmd_none(*pmd)) goto free; if (pmd_bad(*pmd)) { -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] FCSE: Address-space limits. 2009-10-01 21:34 ` [PATCH] FCSE: Conversions between VA and MVA Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: differentiate cpu_tlb_mask from cpu_vm_mask Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Get the kernel macros to reflect address-space limits imposed by the FCSE. Signed-off-by: Richard Cochran <richard.cochran@omicron.at> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/memory.h | 5 +++++ arch/arm/include/asm/processor.h | 5 +++++ arch/arm/mm/mmap.c | 22 +++++++++++++++++++--- 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h index cefedf0..db9781a 100644 --- a/arch/arm/include/asm/memory.h +++ b/arch/arm/include/asm/memory.h @@ -33,7 +33,12 @@ */ #define PAGE_OFFSET UL(CONFIG_PAGE_OFFSET) #define TASK_SIZE (UL(CONFIG_PAGE_OFFSET) - UL(0x01000000)) +#ifndef CONFIG_ARM_FCSE #define TASK_UNMAPPED_BASE (UL(CONFIG_PAGE_OFFSET) / 3) +#else /* CONFIG_ARM_FCSE */ +#define TASK_UNMAPPED_BASE UL(0x01000000) +#define FCSE_TASK_SIZE UL(0x02000000) +#endif /* CONFIG_ARM_FCSE */ /* * The maximum size of a 26-bit user space task. diff --git a/arch/arm/include/asm/processor.h b/arch/arm/include/asm/processor.h index 6a89567..bc33df1 100644 --- a/arch/arm/include/asm/processor.h +++ b/arch/arm/include/asm/processor.h @@ -23,9 +23,14 @@ #include <asm/types.h> #ifdef __KERNEL__ +#ifndef CONFIG_ARM_FCSE #define STACK_TOP ((current->personality & ADDR_LIMIT_32BIT) ? \ TASK_SIZE : TASK_SIZE_26) #define STACK_TOP_MAX TASK_SIZE +#else /* CONFIG_ARM_FCSE */ +#define STACK_TOP FCSE_TASK_SIZE +#define STACK_TOP_MAX FCSE_TASK_SIZE +#endif /* CONFIG_ARM_FCSE */ #endif union debug_insn { diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index f7457fe..e608bda 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -50,13 +50,16 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, #define aliasing 0 #endif +#ifdef CONFIG_ARM_FCSE + start_addr = addr; +#endif /* CONFIG_ARM_FCSE */ /* * We enforce the MAP_FIXED case. */ if (flags & MAP_FIXED) { if (aliasing && flags & MAP_SHARED && addr & (SHMLBA - 1)) return -EINVAL; - return addr; + goto found_addr; } if (len > TASK_SIZE) @@ -71,7 +74,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, vma = find_vma(mm, addr); if (TASK_SIZE - len >= addr && (!vma || addr + len <= vma->vm_start)) - return addr; + goto found_addr; } if (len > mm->cached_hole_size) { start_addr = addr = mm->free_area_cache; @@ -105,7 +108,7 @@ full_search: * Remember the place where we stopped the search: */ mm->free_area_cache = addr + len; - return addr; + goto found_addr; } if (addr + mm->cached_hole_size < vma->vm_start) mm->cached_hole_size = vma->vm_start - addr; @@ -113,6 +116,19 @@ full_search: if (do_align) addr = COLOUR_ALIGN(addr, pgoff); } + +found_addr: +#ifdef CONFIG_ARM_FCSE + if (addr + len > FCSE_TASK_SIZE) { + if (!(flags & MAP_FIXED) && start_addr != TASK_UNMAPPED_BASE) { + start_addr = addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; + goto full_search; + } + return -ENOMEM; + } +#endif /* CONFIG_ARM_FCSE */ + return addr; } -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] FCSE: differentiate cpu_tlb_mask from cpu_vm_mask. 2009-10-01 21:34 ` [PATCH] FCSE: Address-space limits Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Make the shared mappings uncacheable Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel The ARM Linux kernel uses mm->cpu_vm_mask to avoid unnecessary TLB and cache flushes when the process about to be flushed has already been flushed because of a context switch. When enabling FCSE, a context switch flushes the TLB, but no longer flushes the cache, so, for correctness, we need to distinguish whether a process is "present" in the TLB by using a second mask mm->context.cpu_tlb_mask, or whether it is "present" in the cache by using mm->cpu_vm_mask. The current cpu bit is set and cleared from cpu_tlb_mask upon context switch, but only set in cpu_vm_mask. Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/fcse.h | 5 +++++ arch/arm/include/asm/mmu.h | 1 + arch/arm/include/asm/mmu_context.h | 7 +++++-- arch/arm/include/asm/tlbflush.h | 4 ++-- arch/arm/kernel/smp.c | 7 ++++--- 5 files changed, 17 insertions(+), 7 deletions(-) diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h index e443278..4e9503d 100644 --- a/arch/arm/include/asm/fcse.h +++ b/arch/arm/include/asm/fcse.h @@ -30,6 +30,9 @@ /* Mask to get rid of PID from relocated address */ #define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1) +#define fcse_tlb_mask(mm) (&(mm)->context.cpu_tlb_mask) +#define fcse_cpu_set_vm_mask(cpu, mm) cpumask_set_cpu(cpu, mm_cpumask(mm)) + /* Sets the CPU's PID Register */ static inline void fcse_pid_set(unsigned long pid) { @@ -68,6 +71,8 @@ void fcse_pid_free(unsigned pid); #define fcse_pid_set(pid) do { } while (0) #define fcse_mva_to_va(x) (x) #define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); }) +#define fcse_tlb_mask(mm) mm_cpumask(mm) +#define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0) #endif /* ! CONFIG_ARM_FCSE */ #endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index 4a5a8db..7472598 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -9,6 +9,7 @@ typedef struct { #endif #ifdef CONFIG_ARM_FCSE unsigned long pid; + cpumask_t cpu_tlb_mask; #endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; } mm_context_t; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index ec7a84d..042a26d 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -75,6 +75,8 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) #ifdef CONFIG_ARM_FCSE int pid; + cpus_clear(mm->context.cpu_tlb_mask); + pid = fcse_pid_alloc(); if (pid < 0) return pid; @@ -126,12 +128,13 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, !cpumask_test_cpu(cpu, mm_cpumask(next))) __flush_icache_all(); #endif - if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) { + if (!cpumask_test_and_set_cpu(cpu, fcse_tlb_mask(next)) || prev != next) { + fcse_cpu_set_vm_mask(cpu, next); check_context(next); fcse_pid_set(next->context.pid); cpu_switch_mm(next->pgd, next); if (cache_is_vivt()) - cpumask_clear_cpu(cpu, mm_cpumask(prev)); + cpumask_clear_cpu(cpu, fcse_tlb_mask(prev)); } #endif } diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h index 7e14a84..86355b3 100644 --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -351,7 +351,7 @@ static inline void local_flush_tlb_mm(struct mm_struct *mm) if (tlb_flag(TLB_WB)) dsb(); - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) { + if (cpumask_test_cpu(smp_processor_id(), fcse_tlb_mask(mm))) { if (tlb_flag(TLB_V3_FULL)) asm("mcr p15, 0, %0, c6, c0, 0" : : "r" (zero) : "cc"); if (tlb_flag(TLB_V4_U_FULL)) @@ -390,7 +390,7 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) if (tlb_flag(TLB_WB)) dsb(); - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { + if (cpumask_test_cpu(smp_processor_id(), fcse_tlb_mask(vma->vm_mm))) { if (tlb_flag(TLB_V3_PAGE)) asm("mcr p15, 0, %0, c6, c0, 0" : : "r" (uaddr) : "cc"); if (tlb_flag(TLB_V4_U_PAGE)) diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index e0d3277..a0eb423 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -36,6 +36,7 @@ #include <asm/tlbflush.h> #include <asm/ptrace.h> #include <asm/localtimer.h> +#include <asm/fcse.h> /* * as from 2.5, kernels no longer have an init_tasks structure @@ -643,7 +644,7 @@ void flush_tlb_all(void) void flush_tlb_mm(struct mm_struct *mm) { if (tlb_ops_need_broadcast()) - on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, mm_cpumask(mm)); + on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, fcse_tlb_mask(mm)); else local_flush_tlb_mm(mm); } @@ -654,7 +655,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) struct tlb_args ta; ta.ta_vma = vma; ta.ta_start = uaddr; - on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_page(vma, uaddr); } @@ -677,7 +678,7 @@ void flush_tlb_range(struct vm_area_struct *vma, ta.ta_vma = vma; ta.ta_start = start; ta.ta_end = end; - on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_range(vma, start, end); } -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] FCSE: Make the shared mappings uncacheable. 2009-10-01 21:34 ` [PATCH] FCSE: differentiate cpu_tlb_mask from cpu_vm_mask Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: choose whether to flush cache at run-time Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/mm/fault-armv.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index bc0099d..40bccb9 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ no_pmd: static void make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigned long addr, unsigned long pfn) { +#ifndef CONFIG_ARM_FCSE_GUARANTEED struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *mpnt; struct prio_tree_iter iter; @@ -127,6 +128,10 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne adjust_pte(vma, addr); else flush_cache_page(vma, addr, pfn); +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + if (vma->vm_flags & VM_MAYSHARE) + adjust_pte(vma, addr); +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ } /* -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] "Best-effort" FCSE: choose whether to flush cache at run-time. 2009-10-01 21:34 ` [PATCH] FCSE: Make the shared mappings uncacheable Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Allow PID re-use Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel The implementation of the FCSE "best-effort" mode requires to decide, when switching contexts, whether to flush the cache. So, we add a parameter to the mm switching function telling whether we want to flush the cache, and the switch_mm macro uses the fcse_needs_flush to get the value of this parameter. Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/cpu-multi32.h | 5 +++-- arch/arm/include/asm/cpu-single.h | 3 ++- arch/arm/include/asm/fcse.h | 6 ++++++ arch/arm/include/asm/mmu_context.h | 2 +- arch/arm/include/asm/proc-fns.h | 3 ++- arch/arm/kernel/fcse.c | 11 +++++++++++ arch/arm/mm/proc-arm920.S | 9 +++++++++ arch/arm/mm/proc-arm926.S | 9 +++++++++ arch/arm/mm/proc-xscale.S | 9 +++++++++ 9 files changed, 52 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/cpu-multi32.h b/arch/arm/include/asm/cpu-multi32.h index e2b5b0b..18ca8a0 100644 --- a/arch/arm/include/asm/cpu-multi32.h +++ b/arch/arm/include/asm/cpu-multi32.h @@ -52,7 +52,8 @@ extern struct processor { /* * Set the page table */ - void (*switch_mm)(unsigned long pgd_phys, struct mm_struct *mm); + void (*switch_mm)(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); /* * Set a possibly extended PTE. Non-extended PTEs should * ignore 'ext'. @@ -66,4 +67,4 @@ extern struct processor { #define cpu_do_idle() processor._do_idle() #define cpu_dcache_clean_area(addr,sz) processor.dcache_clean_area(addr,sz) #define cpu_set_pte_ext(ptep,pte,ext) processor.set_pte_ext(ptep,pte,ext) -#define cpu_do_switch_mm(pgd,mm) processor.switch_mm(pgd,mm) +#define cpu_do_switch_mm(pgd,mm,flush) processor.switch_mm(pgd,mm,flush) diff --git a/arch/arm/include/asm/cpu-single.h b/arch/arm/include/asm/cpu-single.h index f073a6d..ec556e3 100644 --- a/arch/arm/include/asm/cpu-single.h +++ b/arch/arm/include/asm/cpu-single.h @@ -39,6 +39,7 @@ extern void cpu_proc_init(void); extern void cpu_proc_fin(void); extern int cpu_do_idle(void); extern void cpu_dcache_clean_area(void *, int); -extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm); +extern void cpu_do_switch_mm(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); extern void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext); extern void cpu_reset(unsigned long addr) __attribute__((noreturn)); diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h index 4e9503d..d769a29 100644 --- a/arch/arm/include/asm/fcse.h +++ b/arch/arm/include/asm/fcse.h @@ -66,6 +66,11 @@ fcse_va_to_mva(struct mm_struct *mm, unsigned long va) int fcse_pid_alloc(void); void fcse_pid_free(unsigned pid); +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next); +#else /* CONFIG_ARM_FCSE_GUARANTEED */ +#define fcse_needs_flush(prev, next) (0) +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ #else /* ! CONFIG_ARM_FCSE */ #define fcse_pid_set(pid) do { } while (0) @@ -73,6 +78,7 @@ void fcse_pid_free(unsigned pid); #define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); }) #define fcse_tlb_mask(mm) mm_cpumask(mm) #define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0) +#define fcse_needs_flush(prev, next) (1) #endif /* ! CONFIG_ARM_FCSE */ #endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index 042a26d..ad1bbb4 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -132,7 +132,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, fcse_cpu_set_vm_mask(cpu, next); check_context(next); fcse_pid_set(next->context.pid); - cpu_switch_mm(next->pgd, next); + cpu_switch_mm(next->pgd, next, fcse_needs_flush(prev, next)); if (cache_is_vivt()) cpumask_clear_cpu(cpu, fcse_tlb_mask(prev)); } diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h index 3976412..86f3131 100644 --- a/arch/arm/include/asm/proc-fns.h +++ b/arch/arm/include/asm/proc-fns.h @@ -239,7 +239,8 @@ #ifdef CONFIG_MMU -#define cpu_switch_mm(pgd,mm) cpu_do_switch_mm(virt_to_phys(pgd),mm) +#define cpu_switch_mm(pgd,mm,flush) \ + cpu_do_switch_mm(virt_to_phys(pgd), (mm), (flush)) #define cpu_get_pgd() \ ({ \ diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c index da074b8..7918be6 100644 --- a/arch/arm/kernel/fcse.c +++ b/arch/arm/kernel/fcse.c @@ -55,3 +55,14 @@ void fcse_pid_free(unsigned pid) fcse_pid_dereference(pid); spin_unlock_irqrestore(&fcse_lock, flags); } + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned res; + + res = 0; + + return res; +} +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ diff --git a/arch/arm/mm/proc-arm920.S b/arch/arm/mm/proc-arm920.S index 914d688..3eff19c 100644 --- a/arch/arm/mm/proc-arm920.S +++ b/arch/arm/mm/proc-arm920.S @@ -321,6 +321,11 @@ ENTRY(cpu_arm920_dcache_clean_area) ENTRY(cpu_arm920_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 3f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -338,6 +343,10 @@ ENTRY(cpu_arm920_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* !CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +3: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-arm926.S b/arch/arm/mm/proc-arm926.S index 5446693..f7a6021 100644 --- a/arch/arm/mm/proc-arm926.S +++ b/arch/arm/mm/proc-arm926.S @@ -337,6 +337,11 @@ ENTRY(cpu_arm926_dcache_clean_area) ENTRY(cpu_arm926_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -346,6 +351,10 @@ ENTRY(cpu_arm926_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-xscale.S b/arch/arm/mm/proc-xscale.S index 4233942..f09d135 100644 --- a/arch/arm/mm/proc-xscale.S +++ b/arch/arm/mm/proc-xscale.S @@ -415,9 +415,18 @@ ENTRY(cpu_xscale_dcache_clean_area) */ .align 5 ENTRY(cpu_xscale_switch_mm) +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED clean_d_cache r1, r2 mcr p15, 0, ip, c7, c5, 0 @ Invalidate I cache & BTB mcr p15, 0, ip, c7, c10, 4 @ Drain Write (& Fill) Buffer +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs cpwait_ret lr, ip -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] "Best-effort" FCSE: Allow PID re-use. 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: choose whether to flush cache at run-time Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Handle shared mappings Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel In addition to the bit telling whether or not a pid is allocated, maintain: - a count of processes using a pid; - the current process using a given pid; - a bitfield telling whether or not the cache may contain entries for a given pid. A function fcse_notify_flush_all() is called upon calls to flush_cache_mm() and clears all bits in this last bitfield, in order to avoid that fcse_needs_flush() believe that a pid is reused whereas the cache has been cleaned. Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/cacheflush.h | 4 ++- arch/arm/include/asm/fcse.h | 3 ++ arch/arm/kernel/fcse.c | 73 +++++++++++++++++++++++++++++++++++-- arch/arm/mm/flush.c | 4 ++- 4 files changed, 79 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h index db43631..7ced6d6 100644 --- a/arch/arm/include/asm/cacheflush.h +++ b/arch/arm/include/asm/cacheflush.h @@ -335,8 +335,10 @@ static inline void outer_flush_range(unsigned long start, unsigned long end) #ifndef CONFIG_CPU_CACHE_VIPT static inline void flush_cache_mm(struct mm_struct *mm) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) + if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) { + fcse_notify_flush_all(); __cpuc_flush_user_all(); + } } static inline void diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h index d769a29..c8680c6 100644 --- a/arch/arm/include/asm/fcse.h +++ b/arch/arm/include/asm/fcse.h @@ -68,8 +68,10 @@ int fcse_pid_alloc(void); void fcse_pid_free(unsigned pid); #ifdef CONFIG_ARM_FCSE_BEST_EFFORT int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next); +void fcse_notify_flush_all(void); #else /* CONFIG_ARM_FCSE_GUARANTEED */ #define fcse_needs_flush(prev, next) (0) +#define fcse_notify_flush_all() do { } while (0) #endif /* CONFIG_ARM_FCSE_GUARANTEED */ #else /* ! CONFIG_ARM_FCSE */ @@ -79,6 +81,7 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next); #define fcse_tlb_mask(mm) mm_cpumask(mm) #define fcse_cpu_set_vm_mask(cpu, mm) do { } while (0) #define fcse_needs_flush(prev, next) (1) +#define fcse_notify_flush_all() do { } while (0) #endif /* ! CONFIG_ARM_FCSE */ #endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c index 7918be6..774717c 100644 --- a/arch/arm/kernel/fcse.c +++ b/arch/arm/kernel/fcse.c @@ -13,14 +13,33 @@ static DEFINE_SPINLOCK(fcse_lock); static unsigned long fcse_pids_bits[PIDS_LONGS]; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static unsigned long fcse_pids_cache_dirty[PIDS_LONGS]; +static unsigned random_pid; +struct { + struct mm_struct *last_mm; + unsigned count; +} per_pid[NR_PIDS]; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + static void fcse_pid_reference_inner(unsigned pid) { - __set_bit(pid, fcse_pids_bits); +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (++per_pid[pid].count == 1) +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __set_bit(pid, fcse_pids_bits); } static void fcse_pid_dereference(unsigned pid) { +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (--per_pid[pid].count == 0) { + __clear_bit(pid, fcse_pids_bits); + per_pid[pid].last_mm = NULL; + } +#else /* CONFIG_ARM_FCSE_BEST_EFFORT */ __clear_bit(pid, fcse_pids_bits); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ } int fcse_pid_alloc(void) @@ -37,8 +56,14 @@ int fcse_pid_alloc(void) if (!test_bit(0, fcse_pids_bits)) pid = 0; else { +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (++random_pid == NR_PIDS) + random_pid = 0; + pid = random_pid; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ spin_unlock_irqrestore(&fcse_lock, flags); return -EAGAIN; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ } } fcse_pid_reference_inner(pid); @@ -57,12 +82,54 @@ void fcse_pid_free(unsigned pid) } #ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static void fcse_notify_flush_all_inner(struct mm_struct *next) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + switch (ARRAY_SIZE(fcse_pids_cache_dirty)) { + case 4: + fcse_pids_cache_dirty[3] = 0UL; + case 3: + fcse_pids_cache_dirty[2] = 0UL; + case 2: + fcse_pids_cache_dirty[1] = 0UL; + case 1: + fcse_pids_cache_dirty[0] = 0UL; + } + if (next != &init_mm && next) { + unsigned pid = next->context.pid >> FCSE_PID_SHIFT; + __set_bit(pid, fcse_pids_cache_dirty); + } + spin_unlock_irqrestore(&fcse_lock, flags); +} + int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next) { - unsigned res; + unsigned res, reused_pid = 0, pid = next->context.pid >> FCSE_PID_SHIFT; + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + if (per_pid[pid].last_mm != next) { + if (per_pid[pid].last_mm) + reused_pid = test_bit(pid, fcse_pids_cache_dirty); + per_pid[pid].last_mm = next; + } + __set_bit(pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); - res = 0; + res = reused_pid; + + if (res) { + cpu_clear(smp_processor_id(), prev->cpu_vm_mask); + fcse_notify_flush_all_inner(next); + } return res; } + +void fcse_notify_flush_all(void) +{ + fcse_notify_flush_all_inner(current->mm); +} #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c index ec6fcda..fba8619 100644 --- a/arch/arm/mm/flush.c +++ b/arch/arm/mm/flush.c @@ -50,8 +50,10 @@ static void flush_pfn_alias(unsigned long pfn, unsigned long vaddr) void flush_cache_mm(struct mm_struct *mm) { if (cache_is_vivt()) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) + if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) { + fcse_notify_flush_all(); __cpuc_flush_user_all(); + } return; } -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] "Best-effort" FCSE: Handle shared mappings. 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Allow PID re-use Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Handle mappings above 32 MB Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Account for writable shared mappings, so as to be able to flush cache when switching out a process with such a mapping. This accounting is done by adding the L_PTE_SHARED flag to page belonging to such mappings in make_coherent(), then, set_pte_at() is intercepted to account for both addition and removal of pages with this flag set. Note that using the L_PTE_SHARED flag makes the FCSE patch incompatible with ARMv6+ and Xscale3, but the FCSE feature is deprecated in ARMv6 anyway. Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/mmu.h | 3 ++ arch/arm/include/asm/mmu_context.h | 6 ++++ arch/arm/include/asm/pgtable.h | 42 ++++++++++++++++++++++++-- arch/arm/kernel/fcse.c | 3 +- arch/arm/mm/fault-armv.c | 55 +++++++++++++++++++++++++++++++++++- 5 files changed, 103 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index 7472598..9a2fb51 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -10,6 +10,9 @@ typedef struct { #ifdef CONFIG_ARM_FCSE unsigned long pid; cpumask_t cpu_tlb_mask; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned shared_dirty_pages; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ #endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; } mm_context_t; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index ad1bbb4..035a9f2 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -76,6 +76,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) int pid; cpus_clear(mm->context.cpu_tlb_mask); +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + mm->context.shared_dirty_pages = 0; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ pid = fcse_pid_alloc(); if (pid < 0) @@ -91,6 +94,9 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) static inline void destroy_context(struct mm_struct *mm) { #ifdef CONFIG_ARM_FCSE +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + BUG_ON(mm->context.shared_dirty_pages); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT); #endif /* CONFIG_ARM_FCSE */ } diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 197c596..7775c1b 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -250,6 +250,33 @@ extern pgprot_t pgprot_kernel; #define __S111 __PAGE_SHARED_EXEC #ifndef __ASSEMBLY__ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#define fcse_account_page_removal(mm, addr, val) do { \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && ((_val) & L_PTE_SHARED)) \ + --_mm->context.shared_dirty_pages; \ +} while (0) + +#define fcse_account_page_addition(mm, addr, val) ({ \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && (_val & L_PTE_SHARED)) { \ + if ((_val & (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + != (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + _val &= ~L_PTE_SHARED; \ + else \ + ++_mm->context.shared_dirty_pages; \ + } \ + _val; \ +}) +#else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ +#define fcse_account_page_removal(mm, addr, val) do { } while (0) +#define fcse_account_page_addition(mm, addr, val) (val) +#endif /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ + /* * ZERO_PAGE is a global shared page that is always zero: used * for zero-mapped memory areas etc.. @@ -261,7 +288,10 @@ extern struct page *empty_zero_page; #define pfn_pte(pfn,prot) (__pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot))) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte_ext(ptep, __pte(0), 0) +#define pte_clear(mm,addr,ptep) do { \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + set_pte_ext(ptep, __pte(0), 0); \ +} while (0) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) #define pte_offset_kernel(dir,addr) (pmd_page_vaddr(*(dir)) + __pte_index(addr)) @@ -280,9 +310,13 @@ extern struct page *empty_zero_page; #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext) -#define set_pte_at(mm,addr,ptep,pteval) do { \ - set_pte_ext(ptep, pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ - } while (0) +#define set_pte_at(mm,addr,ptep,pteval) do { \ + unsigned long _pteval = (pteval); \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + pte_val(_pteval) = \ + fcse_account_page_addition(mm, addr, pte_val(_pteval)); \ + set_pte_ext(ptep, _pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ +} while (0) /* * The following only work if pte_present() is true. diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c index 774717c..5b257c9 100644 --- a/arch/arm/kernel/fcse.c +++ b/arch/arm/kernel/fcse.c @@ -118,7 +118,8 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next) __set_bit(pid, fcse_pids_cache_dirty); spin_unlock_irqrestore(&fcse_lock, flags); - res = reused_pid; + res = reused_pid + || prev->context.shared_dirty_pages; if (res) { cpu_clear(smp_processor_id(), prev->cpu_vm_mask); diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 40bccb9..3135a6b 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -25,6 +25,57 @@ static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static void fcse_set_pte_shared(struct vm_area_struct *vma, + unsigned long address) +{ + pgd_t *pgd; + pmd_t *pmd; + pte_t *pte, entry; + + if (!(vma->vm_flags & VM_MAYSHARE) || address >= TASK_SIZE) + return; + + pgd = pgd_offset(vma->vm_mm, address); + if (pgd_none(*pgd)) + goto no_pgd; + if (pgd_bad(*pgd)) + goto bad_pgd; + + pmd = pmd_offset(pgd, address); + if (pmd_none(*pmd)) + goto no_pmd; + if (pmd_bad(*pmd)) + goto bad_pmd; + + pte = pte_offset_map(pmd, address); + entry = *pte; + + if ((pte_val(entry) + & (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) + == (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) { + pte_val(entry) |= L_PTE_SHARED; + set_pte_at(vma->vm_mm, address, pte, entry); + } + pte_unmap(pte); + return; + +bad_pgd: + pgd_ERROR(*pgd); + pgd_clear(pgd); +no_pgd: + return; + +bad_pmd: + pmd_ERROR(*pmd); + pmd_clear(pmd); +no_pmd: + return; +} +#else /* !CONFIG_ARM_FCSE_BEST_EFFORT */ +#define fcse_set_pte_shared(vma, addr) do { } while (0) +#endif /* !CONFIG_ARM_FCSE_BEST_EFFORT */ + /* * We take the easy way out of this problem - we make the * PTE uncacheable. However, we leave the write buffer on. @@ -126,8 +177,10 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne flush_dcache_mmap_unlock(mapping); if (aliases) adjust_pte(vma, addr); - else + else { + fcse_set_pte_shared(vma, addr); flush_cache_page(vma, addr, pfn); + } #else /* CONFIG_ARM_FCSE_GUARANTEED */ if (vma->vm_flags & VM_MAYSHARE) adjust_pte(vma, addr); -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] "Best-effort" FCSE: Handle mappings above 32 MB 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Handle shared mappings Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: add Makefile and compilation option Gilles Chanteperdrix 0 siblings, 1 reply; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel In case an address larger than 32 MB is returned by arch_get_unmapped_area(), relocate the calling process to the null pid. When such a process forks, also give the child process a null pid. Account for pages above 32 MB, and when a process with such pages is switched in or out, flush the cache. As in handling shared mappings, we use set_pte_at() to do the pages accounting. Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/include/asm/fcse.h | 2 + arch/arm/include/asm/mmu.h | 2 + arch/arm/include/asm/mmu_context.h | 17 +++++++++++++- arch/arm/include/asm/pgtable.h | 7 ++++++ arch/arm/kernel/fcse.c | 41 +++++++++++++++++++++++++++++++++++- arch/arm/mm/mmap.c | 7 ++++++ 6 files changed, 73 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h index c8680c6..66fa9de 100644 --- a/arch/arm/include/asm/fcse.h +++ b/arch/arm/include/asm/fcse.h @@ -69,6 +69,8 @@ void fcse_pid_free(unsigned pid); #ifdef CONFIG_ARM_FCSE_BEST_EFFORT int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next); void fcse_notify_flush_all(void); +void fcse_pid_reference(unsigned pid); +void fcse_relocate_mm_to_null_pid(struct mm_struct *mm); #else /* CONFIG_ARM_FCSE_GUARANTEED */ #define fcse_needs_flush(prev, next) (0) #define fcse_notify_flush_all() do { } while (0) diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index 9a2fb51..13c844f 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -12,6 +12,8 @@ typedef struct { cpumask_t cpu_tlb_mask; #ifdef CONFIG_ARM_FCSE_BEST_EFFORT unsigned shared_dirty_pages; + unsigned big; + unsigned high_pages; #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ #endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index 035a9f2..85eac0a 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -77,13 +77,25 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) cpus_clear(mm->context.cpu_tlb_mask); #ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (!mm->context.big) { + pid = fcse_pid_alloc(); + mm->context.pid = pid << FCSE_PID_SHIFT; + } else { + /* We are normally forking a process vith a virtual address + space larger than 32 MB, so its pid should be 0. */ + BUG_ON(mm->context.pid); + fcse_pid_reference(0); + } + /* If we are forking, set_pte_at will restore the correct high pages + count, and shared writable pages are write-protected again. */ mm->context.shared_dirty_pages = 0; -#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ - + mm->context.high_pages = 0; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ pid = fcse_pid_alloc(); if (pid < 0) return pid; mm->context.pid = pid << FCSE_PID_SHIFT; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ #endif /* CONFIG_ARM_FCSE */ return 0; @@ -96,6 +108,7 @@ static inline void destroy_context(struct mm_struct *mm) #ifdef CONFIG_ARM_FCSE #ifdef CONFIG_ARM_FCSE_BEST_EFFORT BUG_ON(mm->context.shared_dirty_pages); + BUG_ON(mm->context.high_pages); #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ fcse_pid_free(mm->context.pid >> FCSE_PID_SHIFT); #endif /* CONFIG_ARM_FCSE */ diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 7775c1b..6c79d71 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -257,6 +257,10 @@ extern pgprot_t pgprot_kernel; unsigned long _val = (val); \ if (pte_present(_val) && ((_val) & L_PTE_SHARED)) \ --_mm->context.shared_dirty_pages; \ + if (pte_present(_val) && _addr < TASK_SIZE) { \ + if (_addr >= FCSE_TASK_SIZE) \ + --_mm->context.high_pages; \ + } \ } while (0) #define fcse_account_page_addition(mm, addr, val) ({ \ @@ -270,6 +274,9 @@ extern pgprot_t pgprot_kernel; else \ ++_mm->context.shared_dirty_pages; \ } \ + if (pte_present(_val) \ + && _addr < TASK_SIZE && _addr >= FCSE_TASK_SIZE) \ + ++_mm->context.high_pages; \ _val; \ }) #else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c index 5b257c9..d15ac30 100644 --- a/arch/arm/kernel/fcse.c +++ b/arch/arm/kernel/fcse.c @@ -119,7 +119,9 @@ int fcse_needs_flush(struct mm_struct *prev, struct mm_struct *next) spin_unlock_irqrestore(&fcse_lock, flags); res = reused_pid - || prev->context.shared_dirty_pages; + || prev->context.shared_dirty_pages + || prev->context.high_pages + || next->context.high_pages; if (res) { cpu_clear(smp_processor_id(), prev->cpu_vm_mask); @@ -133,4 +135,41 @@ void fcse_notify_flush_all(void) { fcse_notify_flush_all_inner(current->mm); } + +void fcse_pid_reference(unsigned pid) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_reference_inner(pid); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +/* Called with mm->mmap_sem write-locked. */ +void fcse_relocate_mm_to_null_pid(struct mm_struct *mm) +{ + pgd_t *to = mm->pgd + pgd_index(0); + pgd_t *from = pgd_offset(mm, 0); + unsigned len = pgd_index(FCSE_TASK_SIZE) * sizeof(*from); + unsigned long flags; + + preempt_disable(); + + memcpy(to, from, len); + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_dereference(mm->context.pid >> FCSE_PID_SHIFT); + fcse_pid_reference_inner(0); + per_pid[0].last_mm = mm; + spin_unlock_irqrestore(&fcse_lock, flags); + + mm->context.pid = 0; + fcse_pid_set(0); + memset(from, '\0', len); + mb(); + flush_cache_mm(mm); + flush_tlb_mm(mm); + + preempt_enable(); +} + #endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index e608bda..ac81369 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -125,7 +125,14 @@ found_addr: mm->cached_hole_size = 0; goto full_search; } +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (mm->context.pid) { + mm->context.big = 1; + fcse_relocate_mm_to_null_pid(mm); + } +#else /* CONFIG_ARM_FCSE_GUARANTEED */ return -ENOMEM; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ } #endif /* CONFIG_ARM_FCSE */ return addr; -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] FCSE: add Makefile and compilation option. 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Handle mappings above 32 MB Gilles Chanteperdrix @ 2009-10-01 21:34 ` Gilles Chanteperdrix 0 siblings, 0 replies; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-01 21:34 UTC (permalink / raw) To: linux-arm-kernel Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> --- arch/arm/kernel/Makefile | 1 + arch/arm/mm/Kconfig | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 0 deletions(-) diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile index 79087dd..574e77e 100644 --- a/arch/arm/kernel/Makefile +++ b/arch/arm/kernel/Makefile @@ -52,5 +52,6 @@ endif head-y := head$(MMUEXT).o obj-$(CONFIG_DEBUG_LL) += debug.o +obj-$(CONFIG_ARM_FCSE) += fcse.o extra-y := $(head-y) init_task.o vmlinux.lds diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index 8d43e58..fb9a4a5 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -776,3 +776,48 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 default 5 + +config ARM_FCSE + bool "Fast Context Switch Extension (EXPERIMENTAL)" + depends on EXPERIMENTAL && !SMP && !CPU_32v6 && !CPU_32v7 + default n + help + The Fast Context Switch Extension (FCSE for short) is an extension of + some ARM processors which allows to switch contexts between processes + without flushing cache and saves a few tens of microseconds in the + worst case. + + Enabling this option makes linux use the FCSE. + + We propose two modes: + - the guaranteed mode: we guarantee that there will never be any cache + flush when switching context, but this means that there can not be + more than 95 running processes in the system, each with a virtual + memory space smaller than 32MB, and that the shared memory + mappings do not use cache; + - the best effort mode: we allow some cache flushes to happen from + time to time, but do not limit the number of processes or the + virtual memory space available for each process, and the shared + memory mappings use cache. + +if ARM_FCSE + +choice + prompt "FCSE mode" + default ARM_FCSE_BEST_EFFORT + help + This option allow setting which FCSE mode will be used. + +config ARM_FCSE_GUARANTEED + bool "guaranteed" + help + Select guaranteed mode. + +config ARM_FCSE_BEST_EFFORT + bool "best effort" + help + Select best-effort mode. + +endchoice + +endif -- 1.5.6.5 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH] FCSE: pid life cycle. 2009-10-01 21:34 ` [PATCH] FCSE: pid life cycle Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Conversions between VA and MVA Gilles Chanteperdrix @ 2009-10-04 19:02 ` Pavel Machek 2009-10-04 19:25 ` Gilles Chanteperdrix 1 sibling, 1 reply; 13+ messages in thread From: Pavel Machek @ 2009-10-04 19:02 UTC (permalink / raw) To: linux-arm-kernel On Thu 2009-10-01 23:34:11, Gilles Chanteperdrix wrote: > Add a pid member to the mm_context_t structure, allocate this pid in > the init_new_context, free it in the destroy_context, switch it in > switch_mm. > > A bitfield is maintained to know what pids are currently in use. > > Signed-off-by: Richard Cochran <richard.cochran@omicron.at> > Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> ... > --- /dev/null > +++ b/arch/arm/include/asm/fcse.h > @@ -0,0 +1,44 @@ > +/* > + * Filename: arch/arm/include/asm/fcse.h > + * Description: ARM Process ID (PID) includes for Fast Address Space Switching > + * (FASS) in ARM Linux. > + * Created: 14/10/2001 > + * Changes: 19/02/2002 - Macros added. > + * 03/08/2007 - Adapted to kernel 2.6.21 (ssm) > + * Feb 2008 - Simplified a bit (rco) We normally avoid changelogs in sources. > +/* Sets the CPU's PID Register */ > +static inline void fcse_pid_set(unsigned long pid) > +{ > + __asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0" > + : /* */: "r" (pid) : "memory"); > +} Calling this 'pid' is extremely confusing. PID already means process id... -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH] FCSE: pid life cycle. 2009-10-04 19:02 ` [PATCH] FCSE: pid life cycle Pavel Machek @ 2009-10-04 19:25 ` Gilles Chanteperdrix 0 siblings, 0 replies; 13+ messages in thread From: Gilles Chanteperdrix @ 2009-10-04 19:25 UTC (permalink / raw) To: linux-arm-kernel Pavel Machek wrote: > On Thu 2009-10-01 23:34:11, Gilles Chanteperdrix wrote: >> Add a pid member to the mm_context_t structure, allocate this pid in >> the init_new_context, free it in the destroy_context, switch it in >> switch_mm. >> >> A bitfield is maintained to know what pids are currently in use. >> >> Signed-off-by: Richard Cochran <richard.cochran@omicron.at> >> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> > ... >> --- /dev/null >> +++ b/arch/arm/include/asm/fcse.h >> @@ -0,0 +1,44 @@ >> +/* >> + * Filename: arch/arm/include/asm/fcse.h >> + * Description: ARM Process ID (PID) includes for Fast Address Space Switching >> + * (FASS) in ARM Linux. >> + * Created: 14/10/2001 >> + * Changes: 19/02/2002 - Macros added. >> + * 03/08/2007 - Adapted to kernel 2.6.21 (ssm) >> + * Feb 2008 - Simplified a bit (rco) > > We normally avoid changelogs in sources. Ok. Will remove that, it was inherited. > >> +/* Sets the CPU's PID Register */ >> +static inline void fcse_pid_set(unsigned long pid) >> +{ >> + __asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0" >> + : /* */: "r" (pid) : "memory"); >> +} > > Calling this 'pid' is extremely confusing. PID already means process > id... Guess what, it also means "process id" in the FCSE case, and it is the name it has in the documentation, I always find a good idea to name the hardware registers as per documentation. I guess the original intent was to put the real process id into the hardware register, but that is not really possible with Linux. So, I guess we can call it fcse_pid. In any case, I will keep all modifications for a second post once I had sufficient review (namely, I am expecting Russell's review, since he is the one who will accept or reject the patch in the end). -- Gilles. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2009-10-04 19:25 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-10-01 21:34 [PATCH] FCSE Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: pid life cycle Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Conversions between VA and MVA Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Address-space limits Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: differentiate cpu_tlb_mask from cpu_vm_mask Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: Make the shared mappings uncacheable Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: choose whether to flush cache at run-time Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Allow PID re-use Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Handle shared mappings Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] "Best-effort" FCSE: Handle mappings above 32 MB Gilles Chanteperdrix 2009-10-01 21:34 ` [PATCH] FCSE: add Makefile and compilation option Gilles Chanteperdrix 2009-10-04 19:02 ` [PATCH] FCSE: pid life cycle Pavel Machek 2009-10-04 19:25 ` Gilles Chanteperdrix
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).