* [Xenomai-help] Xenomai and Gumstix Overo @ 2010-06-29 19:40 Fredrik Asplund 2010-06-29 20:03 ` Gilles Chanteperdrix 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-06-29 19:40 UTC (permalink / raw) To: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 520 bytes --] Hello, I am trying to get Xenomai to work with Gumstix Overo. I managed to set up a working build for the User Side, but when I tried to patch the Kernel source folder (which contains linux-omap3-2.6.33-r78 downloaded using "git://www.sakoman.com/git/linux-omap-2.6;branch=omap3-2.6.33") using the prepare-kernel.sh script I ran into problems. I guess the problem is that the patches do not work with that Kernel Version (see the attached log for details)? But how should I proceed to fix this? Sincerely, / Fredrik [-- Attachment #2: logg.rtf --] [-- Type: application/rtf, Size: 12434 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-29 19:40 [Xenomai-help] Xenomai and Gumstix Overo Fredrik Asplund @ 2010-06-29 20:03 ` Gilles Chanteperdrix 2010-06-30 8:17 ` Fredrik Asplund 0 siblings, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-06-29 20:03 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org Fredrik Asplund wrote: > Hello, I am trying to get Xenomai to work with Gumstix Overo. I > managed to set up a working build for the User Side, but when I tried > to patch the Kernel source folder (which contains > linux-omap3-2.6.33-r78 downloaded using > "git://www.sakoman.com/git/linux-omap-2.6;branch=omap3-2.6.33") using > the prepare-kernel.sh script I ran into problems. I guess the problem > is that the patches do not work with that Kernel Version (see the > attached log for details)? But how should I proceed to fix this? > Sincerely, / Fredrik Xenomai 2.5.3 was tested on OMAP3 with the mainline kernel. If you absolutely want to use this vendor branch, you will have to fix the patch rejects by hand. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-29 20:03 ` Gilles Chanteperdrix @ 2010-06-30 8:17 ` Fredrik Asplund 2010-06-30 8:28 ` Gilles Chanteperdrix 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-06-30 8:17 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai@xenomai.org >Xenomai 2.5.3 was tested on OMAP3 with the mainline kernel. If you >absolutely want to use this vendor branch, you will have to fix the >patch rejects by hand. Ok, yes, I thought so. I guess I was hoping someone had done the same thing and had some pointers on what to do in this specific case (i.e. applying Xenomai using Bitbake on a vendor branch). I will try to apply the patch rejects (I want to deviate from the original setup as little as possible, at least to begin with), but when I went through the source folder I couldn´t find any "*.rej" files (even though the log suggests they were created). Do I need to call the prepare-kernel.sh script differently somehow or is the problem in my tool environment? / Fredrik ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-30 8:17 ` Fredrik Asplund @ 2010-06-30 8:28 ` Gilles Chanteperdrix 2010-06-30 21:58 ` Felipe Brandão Cavalcanti 0 siblings, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-06-30 8:28 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org Fredrik Asplund wrote: >> Xenomai 2.5.3 was tested on OMAP3 with the mainline kernel. If you >> absolutely want to use this vendor branch, you will have to fix the >> patch rejects by hand. > > Ok, yes, I thought so. I guess I was hoping someone had done the same > thing and had some pointers on what to do in this specific case (i.e. > applying Xenomai using Bitbake on a vendor branch). > > I will try to apply the patch rejects (I want to deviate from the > original setup as little as possible, at least to begin with) If you do not want to deviate from Xenomai usual setup, use the mainline kernel. If I were you, I would have serious doubts on a vendor kernel which touches generic files such as include/asm/cacheflush.h in order to add some board-specific code. , but > when I went through the source folder I couldn´t find any "*.rej" > files (even though the log suggests they were created). Do I need to > call the prepare-kernel.sh script differently somehow or is the > problem in my tool environment? The log is the result of running patch in "dry-run" mode, so that if it fails, prepare-kernel.sh does not break your sources. Apply the adeos patch by hand with the patch command. Once the sources are fixed, prepare-kernel.sh will be able to detect that your kernel is already patched and will skip that step. > > / Fredrik -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-30 8:28 ` Gilles Chanteperdrix @ 2010-06-30 21:58 ` Felipe Brandão Cavalcanti 2010-07-01 3:01 ` Thomas Lockhart ` (3 more replies) 0 siblings, 4 replies; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-06-30 21:58 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 3381 bytes --] I am actually trying to do the exact same thing. On Wed, Jun 30, 2010 at 5:28 AM, Gilles Chanteperdrix < gilles.chanteperdrix@xenomai.org> wrote: > Fredrik Asplund wrote: > >> Xenomai 2.5.3 was tested on OMAP3 with the mainline kernel. If you > >> absolutely want to use this vendor branch, you will have to fix the > >> patch rejects by hand. > > > > Ok, yes, I thought so. I guess I was hoping someone had done the same > > thing and had some pointers on what to do in this specific case (i.e. > > applying Xenomai using Bitbake on a vendor branch). > > > > I will try to apply the patch rejects (I want to deviate from the > > original setup as little as possible, at least to begin with) > > If you do not want to deviate from Xenomai usual setup, use the mainline > kernel. If I were you, I would have serious doubts on a vendor kernel > which touches generic files such as include/asm/cacheflush.h in order to > add some board-specific code. > > My first choice was to run a vanilla kernel, but no luck - didn't find any suitable config files, or any reports of success. I am on the same system - Gumstix Overo, kernel 2.6.33, Xenomai 2.5.3. Anyway, I've managed to change the files manually - the changes seem very minor (the changes in include/asm/cacheflush.h just move a function elsewhere, no code changes at all). However, I am having problems with the kernel - any user-space Xenomai program gives me the following error: Xenomai or CONFIG_OPT_PERVASIVE disabled. (modprobe xeno_nucleus?) Kernel-space programs give me the following error: open(/proc/xenomai/registry/native/pipes/klat_pipe): No such file or directory modprobe klat_mod or try the -P latency I've tried the obvious solutions (tweaking kernel configs, enabling debug, etc...) but no clues on how to fix the error. All the logs seem normal to me (I-pipe reports normal starup, Xenomai reports successful start). I am starting to think that one of my changes to the kernel might be making the system to fail silently. So, any clues? I can send my changes and logs tomorrow morning, if they are of any help. Another option which I am considering is trying Xenomai with the 2.6.31 kernel, or maybe even 2.6.30 or 2.6.28. Thanks in advance, -Felipe Brandão Cavalcanti , but > > when I went through the source folder I couldn´t find any "*.rej" > > files (even though the log suggests they were created). Do I need to > > call the prepare-kernel.sh script differently somehow or is the > > problem in my tool environment? > > The log is the result of running patch in "dry-run" mode, so that if it > fails, prepare-kernel.sh does not break your sources. Apply the adeos > patch by hand with the patch command. Once the sources are fixed, > prepare-kernel.sh will be able to detect that your kernel is already > patched and will skip that step. > > > > > / Fredrik > > > -- > Gilles. > > > _______________________________________________ > Xenomai-help mailing list > Xenomai-help@domain.hid > https://mail.gna.org/listinfo/xenomai-help > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 4376 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-30 21:58 ` Felipe Brandão Cavalcanti @ 2010-07-01 3:01 ` Thomas Lockhart 2010-07-01 6:49 ` Gilles Chanteperdrix ` (2 subsequent siblings) 3 siblings, 0 replies; 32+ messages in thread From: Thomas Lockhart @ 2010-07-01 3:01 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org > However, I am having problems with the kernel - any user-space Xenomai > program gives me the following error: > Xenomai or CONFIG_OPT_PERVASIVE disabled. > (modprobe xeno_nucleus?) > Kernel-space programs give me the following error: > open(/proc/xenomai/registry/native/pipes/klat_pipe): No such file or > directory > modprobe klat_mod or try the -P latency > > I've tried the obvious solutions (tweaking kernel configs, enabling > debug, etc...) but no clues on how to fix the error. You did not say explicitly: did you do the modprobes as suggested in the error messages above? - Tom ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-30 21:58 ` Felipe Brandão Cavalcanti 2010-07-01 3:01 ` Thomas Lockhart @ 2010-07-01 6:49 ` Gilles Chanteperdrix 2010-07-01 7:17 ` Gilles Chanteperdrix 2010-07-01 8:11 ` Fredrik Asplund 2010-07-01 12:22 ` Fredrik Asplund 3 siblings, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-01 6:49 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org Felipe Brandão Cavalcanti wrote: > I am actually trying to do the exact same thing. > > On Wed, Jun 30, 2010 at 5:28 AM, Gilles Chanteperdrix < > gilles.chanteperdrix@xenomai.org> wrote: > >> Fredrik Asplund wrote: >>>> Xenomai 2.5.3 was tested on OMAP3 with the mainline kernel. If you >>>> absolutely want to use this vendor branch, you will have to fix the >>>> patch rejects by hand. >>> Ok, yes, I thought so. I guess I was hoping someone had done the same >>> thing and had some pointers on what to do in this specific case (i.e. >>> applying Xenomai using Bitbake on a vendor branch). >>> >>> I will try to apply the patch rejects (I want to deviate from the >>> original setup as little as possible, at least to begin with) >> If you do not want to deviate from Xenomai usual setup, use the mainline >> kernel. If I were you, I would have serious doubts on a vendor kernel >> which touches generic files such as include/asm/cacheflush.h in order to >> add some board-specific code. >> >> > My first choice was to run a vanilla kernel, but no luck - didn't find any > suitable config files, or any reports of success. I am on the same system - > Gumstix Overo, kernel 2.6.33, Xenomai 2.5.3. The 2.6.33 kernel has an overo_defconfig. This defconfig sets CONFIG_MACH_OVERO. arch/arm/mach-omap2/Kconfig contains the following lines: config MACH_OVERO bool "Gumstix Overo board" depends on ARCH_OMAP3 && ARCH_OMAP34XX select OMAP_PACKAGE_CBB So, it looks to me like the vanilla kernel supports Gumstix overo. > > Anyway, I've managed to change the files manually - the changes seem very > minor (the changes in include/asm/cacheflush.h just move a function > elsewhere, no code changes at all). That is not the point. The point is that to add support for a board, you should not mess with generic code. So, it is probably safer to get the vanilla support working. The I-pipe patch modifies only SOC-specific files, so, any well written board support which adds board-specific files should merge with the I-pipe patch without any reject. > > However, I am having problems with the kernel - any user-space Xenomai > program gives me the following error: > Xenomai or CONFIG_OPT_PERVASIVE disabled. > (modprobe xeno_nucleus?) is CONFIG_XENO_OPT_PERVASIVE in your configuration? > All the logs seem normal to me (I-pipe reports normal starup, Xenomai > reports successful start). I am starting to think that one of my changes to > the kernel might be making the system to fail silently. I use Xenomai 2.5.3 on an OMAP3 based board without any problems. So, yes, either you are mis-configured (in order to check that, just use Xenomai default configuration), or your kernel is broken. > > So, any clues? I can send my changes and logs tomorrow morning, if they are > of any help. > > Another option which I am considering is trying Xenomai with the 2.6.31 > kernel, or maybe even 2.6.30 or 2.6.28. I am sorry, but I am not willing to debug this, I am afraid you are on your own. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 6:49 ` Gilles Chanteperdrix @ 2010-07-01 7:17 ` Gilles Chanteperdrix 0 siblings, 0 replies; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-01 7:17 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org Gilles Chanteperdrix wrote: >> All the logs seem normal to me (I-pipe reports normal starup, Xenomai >> reports successful start). I am starting to think that one of my changes to >> the kernel might be making the system to fail silently. > > I use Xenomai 2.5.3 on an OMAP3 based board without any problems. So, > yes, either you are mis-configured (in order to check that, just use > Xenomai default configuration), or your kernel is broken. Also, check the EABI settings: if you are running a EABI user-space, you should pass --enable-arm-eabi to Xenomai user-space configure script. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-30 21:58 ` Felipe Brandão Cavalcanti 2010-07-01 3:01 ` Thomas Lockhart 2010-07-01 6:49 ` Gilles Chanteperdrix @ 2010-07-01 8:11 ` Fredrik Asplund 2010-07-01 8:59 ` Fredrik Asplund 2010-07-01 12:22 ` Fredrik Asplund 3 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-01 8:11 UTC (permalink / raw) To: gilles.chanteperdrix@xenomai.org, cavalkaf@domain.hid; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 1134 bytes --] >I am actually trying to do the exact same thing. Looking at forums and email archives it seems a lot of people have been trying to do this, but that no-one has posted their results. :/ >Anyway, I've managed to change the files manually - the changes seem >very minor (the changes in include/asm/cacheflush.h just move a function >elsewhere, no code changes at all). I have attached a patch file showing the changes I have done (and an updated Adeos patch file). This was fairly trivial, but I guess the problem could be if Gumstix have added some changes which will require additional patching for Adeos. Anyway, could you share a patch file of exactly what you have changed? >So, any clues? I can send my changes and logs tomorrow morning, if they are of any help. Sure, please do so. I don´t have much time to put on this, so I have actually only have had to go through the source files since yesterday. Today (or during the weekend if I don´t get the time) I will create a new defconfig file, compile, test and debug. If you have updated BitBake file, Defconfig file, Logs, etc., please share them. / Fredrik [-- Attachment #2: PrepareOveroForXenomai.patch --] [-- Type: application/octet-stream, Size: 12692 bytes --] diff -crB linux-2.6.33.5_patch_original/arch/arm/include/asm/cacheflush.h linux-2.6.33.5_patch_applied/arch/arm/include/asm/cacheflush.h *** linux-2.6.33.5_patch_original/arch/arm/include/asm/cacheflush.h 2010-06-23 16:01:54.000000000 +0200 --- linux-2.6.33.5_patch_applied/arch/arm/include/asm/cacheflush.h 2010-06-30 15:35:36.645975000 +0200 *************** *** 15,20 **** --- 15,21 ---- #include <asm/glue.h> #include <asm/shmparam.h> #include <asm/cachetype.h> + #include <asm/fcse.h> #define CACHE_COLOUR(vaddr) ((vaddr & (SHMLBA - 1)) >> PAGE_SHIFT) *************** *** 312,317 **** --- 313,339 ---- #endif + #ifdef CONFIG_ARM_FCSE + #define FCSE_CACHE_MASK (~(L1_CACHE_BYTES - 1)) + #define FCSE_CACHE_ALIGN(addr) (((addr) + ~FCSE_CACHE_MASK) & FCSE_CACHE_MASK) + + static inline void + fcse_flush_cache_user_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) + { + if (cache_is_vivt() + && fcse_mm_in_cache(vma->vm_mm)) { + start = fcse_va_to_mva(vma->vm_mm, start & FCSE_CACHE_MASK); + end = fcse_va_to_mva(vma->vm_mm, FCSE_CACHE_ALIGN(end)); + __cpuc_flush_user_range(start, end, vma->vm_flags); + } + } + #undef FCSE_CACHE_MASK + #undef FCSE_CACHE_ALIGN + #else /* ! CONFIG_ARM_FCSE */ + #define fcse_flush_cache_user_range(vma, start, end) do { } while (0) + #endif /* ! CONFIG_ARM_FCSE */ + /* * Copy user data from/to a page which is mapped into a different * processes address space. Really, we want to allow our "user *************** *** 320,327 **** extern void copy_to_user_page(struct vm_area_struct *, struct page *, unsigned long, void *, const void *, unsigned long); #define copy_from_user_page(vma, page, vaddr, dst, src, len) \ ! do { \ ! memcpy(dst, src, len); \ } while (0) /* --- 342,350 ---- extern void copy_to_user_page(struct vm_area_struct *, struct page *, unsigned long, void *, const void *, unsigned long); #define copy_from_user_page(vma, page, vaddr, dst, src, len) \ ! do { \ ! fcse_flush_cache_user_range(vma, vaddr, vaddr + len); \ ! memcpy(dst, src, len); \ } while (0) /* *************** *** 331,353 **** static inline void vivt_flush_cache_mm(struct mm_struct *mm) { ! if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) ! __cpuc_flush_user_all(); } static inline void vivt_flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { ! if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) ! __cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end), ! vma->vm_flags); } static inline void vivt_flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn) { ! if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { ! unsigned long addr = user_addr & PAGE_MASK; __cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags); } } --- 354,385 ---- static inline void vivt_flush_cache_mm(struct mm_struct *mm) { ! if (fcse_mm_in_cache(mm)) { ! unsigned seq; ! ! do { ! seq = fcse_flush_all_start(mm); ! __cpuc_flush_user_all(); ! } while (!fcse_flush_all_done(mm, seq, 1)); ! } } static inline void vivt_flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { ! if (fcse_mm_in_cache(vma->vm_mm)) { ! start = fcse_va_to_mva(vma->vm_mm, start & PAGE_MASK); ! end = fcse_va_to_mva(vma->vm_mm, PAGE_ALIGN(end)); ! __cpuc_flush_user_range(start, end, vma->vm_flags); ! } } static inline void vivt_flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn) { ! if (fcse_mm_in_cache(vma->vm_mm)) { ! unsigned long addr; ! addr = fcse_va_to_mva(vma->vm_mm, user_addr) & PAGE_MASK; __cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags); } } *************** *** 372,385 **** * Harvard caches are synchronised for the user space address range. * This is used for the ARM private sys_cacheflush system call. */ ! #define flush_cache_user_range(vma,start,end) \ ! __cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end)) /* * Perform necessary cache operations to ensure that data previously * stored within this range of addresses can be executed by the CPU. */ ! #define flush_icache_range(s,e) __cpuc_coherent_kern_range(s,e) /* * Perform necessary cache operations to ensure that the TLB will --- 404,425 ---- * Harvard caches are synchronised for the user space address range. * This is used for the ARM private sys_cacheflush system call. */ ! #define flush_cache_user_range(vma, start, end) \ ! ({ \ ! struct mm_struct *_mm = (vma)->vm_mm; \ ! unsigned long _start, _end; \ ! _start = fcse_va_to_mva(_mm, start) & PAGE_MASK; \ ! _end = PAGE_ALIGN(fcse_va_to_mva(_mm, end)); \ ! __cpuc_coherent_user_range(_start, _end); \ ! }) /* * Perform necessary cache operations to ensure that data previously * stored within this range of addresses can be executed by the CPU. */ ! #define flush_icache_range(s,e) \ ! __cpuc_coherent_kern_range(fcse_va_to_mva(current->mm, (s)), \ ! fcse_va_to_mva(current->mm, (e))) /* * Perform necessary cache operations to ensure that the TLB will *************** *** 431,437 **** extern void __flush_anon_page(struct vm_area_struct *vma, struct page *, unsigned long); if (PageAnon(page)) ! __flush_anon_page(vma, page, vmaddr); } #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE --- 471,478 ---- extern void __flush_anon_page(struct vm_area_struct *vma, struct page *, unsigned long); if (PageAnon(page)) ! __flush_anon_page(vma, page, ! fcse_va_to_mva(vma->vm_mm, vmaddr)); } #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE diff -crB linux-2.6.33.5_patch_original/arch/arm/mach-omap2/board-igep0020.c linux-2.6.33.5_patch_applied/arch/arm/mach-omap2/board-igep0020.c *** linux-2.6.33.5_patch_original/arch/arm/mach-omap2/board-igep0020.c 2010-06-23 16:01:54.000000000 +0200 --- linux-2.6.33.5_patch_applied/arch/arm/mach-omap2/board-igep0020.c 2010-06-30 18:51:40.413973000 +0200 *************** *** 290,295 **** --- 290,307 ---- .usb_mode = T2_USB_MODE_ULPI, }; + static struct ehci_hcd_omap_platform_data ehci_pdata __initconst = { + + .port_mode[0] = EHCI_HCD_OMAP_MODE_PHY, + .port_mode[1] = EHCI_HCD_OMAP_MODE_UNKNOWN, + .port_mode[2] = EHCI_HCD_OMAP_MODE_UNKNOWN, + + .phy_reset = true, + .reset_gpio_port[0] = 24, + .reset_gpio_port[1] = -EINVAL, + .reset_gpio_port[2] = -EINVAL + }; + static int igep2_enable_dvi(struct omap_dss_device *dssdev) { gpio_direction_output(IGEP2_GPIO_DVI_PUP, 1); *************** *** 442,458 **** .power = 100, }; - static struct ehci_hcd_omap_platform_data ehci_pdata __initconst = { - .port_mode[0] = EHCI_HCD_OMAP_MODE_UNKNOWN, - .port_mode[1] = EHCI_HCD_OMAP_MODE_PHY, - .port_mode[2] = EHCI_HCD_OMAP_MODE_UNKNOWN, - - .phy_reset = true, - .reset_gpio_port[0] = -EINVAL, - .reset_gpio_port[1] = IGEP2_GPIO_USBH_NRESET, - .reset_gpio_port[2] = -EINVAL, - }; - #ifdef CONFIG_OMAP_MUX static struct omap_board_mux board_mux[] __initdata = { { .reg_offset = OMAP_MUX_TERMINATOR }, --- 454,459 ---- diff -crB linux-2.6.33.5_patch_original/arch/arm/mm/context.c linux-2.6.33.5_patch_applied/arch/arm/mm/context.c *** linux-2.6.33.5_patch_original/arch/arm/mm/context.c 2010-06-23 16:01:55.000000000 +0200 --- linux-2.6.33.5_patch_applied/arch/arm/mm/context.c 2010-06-30 19:05:07.137978000 +0200 *************** *** 16,22 **** #include <asm/mmu_context.h> #include <asm/tlbflush.h> ! static DEFINE_SPINLOCK(cpu_asid_lock); unsigned int cpu_last_asid = ASID_FIRST_VERSION; #ifdef CONFIG_SMP DEFINE_PER_CPU(struct mm_struct *, current_mm); --- 16,22 ---- #include <asm/mmu_context.h> #include <asm/tlbflush.h> ! static IPIPE_DEFINE_SPINLOCK(cpu_asid_lock); unsigned int cpu_last_asid = ASID_FIRST_VERSION; #ifdef CONFIG_SMP DEFINE_PER_CPU(struct mm_struct *, current_mm); *************** *** 116,123 **** void __new_context(struct mm_struct *mm) { unsigned int asid; ! spin_lock(&cpu_asid_lock); #ifdef CONFIG_SMP /* * Check the ASID again, in case the change was broadcast from --- 116,124 ---- void __new_context(struct mm_struct *mm) { unsigned int asid; + unsigned long flags; ! spin_lock_irqsave_cond(&cpu_asid_lock, flags); #ifdef CONFIG_SMP /* * Check the ASID again, in case the change was broadcast from *************** *** 153,157 **** } set_mm_context(mm, asid); ! spin_unlock(&cpu_asid_lock); } --- 154,158 ---- } set_mm_context(mm, asid); ! spin_unlock_irqrestore_cond(&cpu_asid_lock, flags); } diff -crB linux-2.6.33.5_patch_original/arch/arm/mm/fault-armv.c linux-2.6.33.5_patch_applied/arch/arm/mm/fault-armv.c *** linux-2.6.33.5_patch_original/arch/arm/mm/fault-armv.c 2010-06-23 16:01:55.000000000 +0200 --- linux-2.6.33.5_patch_applied/arch/arm/mm/fault-armv.c 2010-06-30 19:13:34.057974000 +0200 *************** *** 27,32 **** --- 27,83 ---- static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE; + #ifdef CONFIG_ARM_FCSE_BEST_EFFORT + static void fcse_set_pte_shared(struct vm_area_struct *vma, + unsigned long address) + { + pgd_t *pgd; + pmd_t *pmd; + pte_t *pte, entry; + + if (!(vma->vm_flags & VM_MAYSHARE) || address >= TASK_SIZE) + return; + + pgd = pgd_offset(vma->vm_mm, address); + if (pgd_none(*pgd)) + goto no_pgd; + if (pgd_bad(*pgd)) + goto bad_pgd; + + pmd = pmd_offset(pgd, address); + if (pmd_none(*pmd)) + goto no_pmd; + if (pmd_bad(*pmd)) + goto bad_pmd; + + pte = pte_offset_map(pmd, address); + entry = *pte; + + if ((pte_val(entry) + & (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) + == (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) { + pte_val(entry) |= L_PTE_SHARED; + set_pte_at(vma->vm_mm, address, pte, entry); + } + pte_unmap(pte); + return; + + bad_pgd: + pgd_ERROR(*pgd); + pgd_clear(pgd); + no_pgd: + return; + + bad_pmd: + pmd_ERROR(*pmd); + pmd_clear(pmd); + no_pmd: + return; + } + #else /* !CONFIG_ARM_FCSE_BEST_EFFORT */ + #define fcse_set_pte_shared(vma, addr) do { } while (0) + #endif /* !CONFIG_ARM_FCSE_BEST_EFFORT */ + /* * We take the easy way out of this problem - we make the * PTE uncacheable. However, we leave the write buffer on. *************** *** 102,107 **** --- 153,159 ---- make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned long pfn) { + #ifndef CONFIG_ARM_FCSE_GUARANTEED struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *mpnt; struct prio_tree_iter iter; *************** *** 133,140 **** flush_dcache_mmap_unlock(mapping); if (aliases) do_adjust_pte(vma, addr, pfn, ptep); ! else flush_cache_page(vma, addr, pfn); } /* --- 185,198 ---- flush_dcache_mmap_unlock(mapping); if (aliases) do_adjust_pte(vma, addr, pfn, ptep); ! else { ! fcse_set_pte_shared(vma, addr); flush_cache_page(vma, addr, pfn); + } + #else /* CONFIG_ARM_FCSE_GUARANTEED */ + if (vma->vm_flags & VM_MAYSHARE) + adjust_pte(vma, addr); + #endif /* CONFIG_ARM_FCSE_GUARANTEED */ } /* diff -crB linux-2.6.33.5_patch_original/arch/arm/mm/flush.c linux-2.6.33.5_patch_applied/arch/arm/mm/flush.c *** linux-2.6.33.5_patch_original/arch/arm/mm/flush.c 2010-06-23 16:01:55.000000000 +0200 --- linux-2.6.33.5_patch_applied/arch/arm/mm/flush.c 2010-06-30 19:46:54.753972974 +0200 *************** *** 13,18 **** --- 13,19 ---- #include <asm/cacheflush.h> #include <asm/cachetype.h> + #include <asm/fcse.h> #include <asm/smp_plat.h> #include <asm/system.h> #include <asm/tlbflush.h> *************** *** 104,110 **** unsigned long uaddr, void *kaddr, unsigned long len) { if (cache_is_vivt()) { ! if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { unsigned long addr = (unsigned long)kaddr; __cpuc_coherent_kern_range(addr, addr + len); } --- 105,111 ---- unsigned long uaddr, void *kaddr, unsigned long len) { if (cache_is_vivt()) { ! if (fcse_mm_in_cache(vma->vm_mm)) { unsigned long addr = (unsigned long)kaddr; __cpuc_coherent_kern_range(addr, addr + len); } *************** *** 143,148 **** --- 144,150 ---- #ifdef CONFIG_SMP preempt_disable(); #endif + fcse_flush_cache_user_range(vma, vaddr, vaddr + len); memcpy(dst, src, len); flush_ptrace_access(vma, page, uaddr, dst, len); #ifdef CONFIG_SMP [-- Attachment #3: adeos-ipipe-2.6.33-arm-1.16-01_updated.patch --] [-- Type: application/octet-stream, Size: 438235 bytes --] diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 184a6bd..1a8674b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -254,6 +254,8 @@ config ARCH_VERSATILE config ARCH_AT91 bool "Atmel AT91" select GENERIC_GPIO + select GENERIC_TIME if IPIPE + select GENERIC_CLOCKEVENTS if IPIPE select ARCH_REQUIRE_GPIOLIB select HAVE_CLK help @@ -514,7 +516,7 @@ config ARCH_KS8695 bool "Micrel/Kendin KS8695" select CPU_ARM922T select GENERIC_GPIO - select ARCH_REQUIRE_GPIOLIB + select ARCH_REQUIRE_GPIOLIB help Support for Micrel/Kendin KS8695 "Centaur" (ARM922T) based System-on-Chip devices. @@ -1070,6 +1072,8 @@ config LOCAL_TIMERS accounting to be spread across the timer interval, preventing a "thundering herd" at every timer tick. +source kernel/ipipe/Kconfig + source kernel/Kconfig.preempt config HZ diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S index 4fddc50..cbb2e4c 100644 --- a/arch/arm/boot/compressed/head.S +++ b/arch/arm/boot/compressed/head.S @@ -1074,6 +1074,15 @@ memdump: mov r12, r0 mov pc, r10 #endif +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + .text + .align 0 + .type mcount %function + .global mcount +mcount: + mov pc, lr @ just return +#endif + .ltorg reloc_end: diff --git a/arch/arm/common/it8152.c b/arch/arm/common/it8152.c index 2793447..5ec1999 100644 --- a/arch/arm/common/it8152.c +++ b/arch/arm/common/it8152.c @@ -26,6 +26,7 @@ #include <linux/ioport.h> #include <linux/irq.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <asm/mach/pci.h> #include <asm/hardware/it8152.h> @@ -120,21 +121,21 @@ void it8152_irq_demux(unsigned int irq, struct irq_desc *desc) bits_pd &= ((1 << IT8152_PD_IRQ_COUNT) - 1); while (bits_pd) { i = __ffs(bits_pd); - generic_handle_irq(IT8152_PD_IRQ(i)); + ipipe_handle_irq_cond(IT8152_PD_IRQ(i)); bits_pd &= ~(1 << i); } bits_lp &= ((1 << IT8152_LP_IRQ_COUNT) - 1); while (bits_lp) { i = __ffs(bits_lp); - generic_handle_irq(IT8152_LP_IRQ(i)); + ipipe_handle_irq_cond(IT8152_LP_IRQ(i)); bits_lp &= ~(1 << i); } bits_ld &= ((1 << IT8152_LD_IRQ_COUNT) - 1); while (bits_ld) { i = __ffs(bits_ld); - generic_handle_irq(IT8152_LD_IRQ(i)); + ipipe_handle_irq_cond(IT8152_LD_IRQ(i)); bits_ld &= ~(1 << i); } } diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 00f46d9..84ea0d3 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -81,6 +81,18 @@ .macro enable_irq_notrace cpsie i .endm + + .macro disable_irq_cond +#ifdef CONFIG_IPIPE + cpsid i +#endif /* CONFIG_IPIPE */ + .endm + + .macro enable_irq_cond +#ifdef CONFIG_IPIPE + cpsie i +#endif /* CONFIG_IPIPE */ + .endm #else .macro disable_irq_notrace msr cpsr_c, #PSR_I_BIT | SVC_MODE @@ -89,6 +101,18 @@ .macro enable_irq_notrace msr cpsr_c, #SVC_MODE .endm + + .macro disable_irq_cond +#ifdef CONFIG_IPIPE + msr cpsr_c, #PSR_I_BIT | SVC_MODE +#endif /* CONFIG_IPIPE */ + .endm + + .macro enable_irq_cond +#ifdef CONFIG_IPIPE + msr cpsr_c, #SVC_MODE +#endif /* CONFIG_IPIPE */ + .endm #endif .macro asm_trace_hardirqs_off diff --git a/arch/arm/include/asm/atomic.h b/arch/arm/include/asm/atomic.h index d0daeab..d4f9618 100644 --- a/arch/arm/include/asm/atomic.h +++ b/arch/arm/include/asm/atomic.h @@ -158,10 +158,10 @@ static inline int atomic_add_return(int i, atomic_t *v) unsigned long flags; int val; - raw_local_irq_save(flags); + local_irq_save_hw(flags); val = v->counter; v->counter = val += i; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return val; } @@ -172,10 +172,10 @@ static inline int atomic_sub_return(int i, atomic_t *v) unsigned long flags; int val; - raw_local_irq_save(flags); + local_irq_save_hw(flags); val = v->counter; v->counter = val -= i; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return val; } @@ -186,11 +186,11 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new) int ret; unsigned long flags; - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = v->counter; if (likely(ret == old)) v->counter = new; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return ret; } @@ -199,9 +199,9 @@ static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) { unsigned long flags; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *addr &= ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } #endif /* __LINUX_ARM_ARCH__ */ diff --git a/arch/arm/include/asm/bitops.h b/arch/arm/include/asm/bitops.h index 338ff19..acc35cd 100644 --- a/arch/arm/include/asm/bitops.h +++ b/arch/arm/include/asm/bitops.h @@ -41,9 +41,9 @@ static inline void ____atomic_set_bit(unsigned int bit, volatile unsigned long * p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p |= mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline void ____atomic_clear_bit(unsigned int bit, volatile unsigned long *p) @@ -53,9 +53,9 @@ static inline void ____atomic_clear_bit(unsigned int bit, volatile unsigned long p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p &= ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline void ____atomic_change_bit(unsigned int bit, volatile unsigned long *p) @@ -65,9 +65,9 @@ static inline void ____atomic_change_bit(unsigned int bit, volatile unsigned lon p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p ^= mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline int @@ -79,10 +79,10 @@ ____atomic_test_and_set_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res | mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } @@ -96,10 +96,10 @@ ____atomic_test_and_clear_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res & ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } @@ -113,10 +113,10 @@ ____atomic_test_and_change_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res ^ mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } diff --git a/arch/arm/include/asm/cpu-multi32.h b/arch/arm/include/asm/cpu-multi32.h index e2b5b0b..18ca8a0 100644 --- a/arch/arm/include/asm/cpu-multi32.h +++ b/arch/arm/include/asm/cpu-multi32.h @@ -52,7 +52,8 @@ extern struct processor { /* * Set the page table */ - void (*switch_mm)(unsigned long pgd_phys, struct mm_struct *mm); + void (*switch_mm)(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); /* * Set a possibly extended PTE. Non-extended PTEs should * ignore 'ext'. @@ -66,4 +67,4 @@ extern struct processor { #define cpu_do_idle() processor._do_idle() #define cpu_dcache_clean_area(addr,sz) processor.dcache_clean_area(addr,sz) #define cpu_set_pte_ext(ptep,pte,ext) processor.set_pte_ext(ptep,pte,ext) -#define cpu_do_switch_mm(pgd,mm) processor.switch_mm(pgd,mm) +#define cpu_do_switch_mm(pgd,mm,flush) processor.switch_mm(pgd,mm,flush) diff --git a/arch/arm/include/asm/cpu-single.h b/arch/arm/include/asm/cpu-single.h index f073a6d..ec556e3 100644 --- a/arch/arm/include/asm/cpu-single.h +++ b/arch/arm/include/asm/cpu-single.h @@ -39,6 +39,7 @@ extern void cpu_proc_init(void); extern void cpu_proc_fin(void); extern int cpu_do_idle(void); extern void cpu_dcache_clean_area(void *, int); -extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm); +extern void cpu_do_switch_mm(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); extern void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext); extern void cpu_reset(unsigned long addr) __attribute__((noreturn)); diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h new file mode 100644 index 0000000..cfdf14a --- /dev/null +++ b/arch/arm/include/asm/fcse.h @@ -0,0 +1,178 @@ +/* + * Filename: arch/arm/include/asm/fcse.h + * Description: ARM Process ID (PID) includes for Fast Address Space Switching + * (FASS) in ARM Linux. + * + * Copyright: (C) 2001, 2002 Adam Wiggins <awiggins@cse.unsw.edu.au> + * (C) 2007 Sebastian Smolorz <ssm@emlix.com> + * (C) 2008 Richard Cochran + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of teh GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#ifndef __ASM_ARM_FCSE_H +#define __ASM_ARM_FCSE_H + +#ifdef CONFIG_ARM_FCSE + +#include <linux/mm_types.h> /* For struct mm_struct */ +#include <linux/sched.h> +#include <linux/hardirq.h> + +#include <asm/bitops.h> +#include <asm/cachetype.h> + +#define FCSE_PID_SHIFT 25 + +/* Size of PID relocation area */ +#define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT) + +/* Mask to get rid of PID from relocated address */ +#define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1) + +extern unsigned long fcse_pids_cache_dirty[]; + +#ifdef CONFIG_ARM_FCSE_DEBUG +#define FCSE_BUG_ON(expr) BUG_ON(expr) +#else /* !CONFIG_ARM_FCSE_DEBUG */ +#define FCSE_BUG_ON(expr) do { } while(0) +#endif /* !CONFIG_ARM_FCSE_DEBUG */ + +#if defined(CONFIG_ARM_FCSE_DYNPID) && !defined(CONFIG_PREEMPT_NONE) +#define fcse_check_context(mm) \ + FCSE_BUG_ON(!in_atomic() \ + && (mm)->context.fcse.active \ + && atomic_read(&(mm)->mm_users) \ + && !(mm)->core_state \ + && !rwsem_is_locked(&(mm)->mmap_sem) \ + && !irqs_disabled_hw()); +#else /* !CONFIG_ARM_FCSE_DYNPID */ +#define fcse_check_context(mm) do { (void)(mm); } while(0) +#endif /* !CONFIG_ARM_FCSE_DYNPID */ + +int fcse_pid_alloc(struct mm_struct *mm); +void fcse_pid_free(struct mm_struct *mm); +unsigned fcse_flush_all_start(struct mm_struct *mm); +int fcse_flush_all_done(struct mm_struct *mm, unsigned seq, unsigned dirty); +unsigned long +fcse_check_mmap_inner(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl); + +/* Sets the CPU's PID Register */ +static inline void fcse_pid_set(unsigned long pid) +{ + __asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0" + : /* */: "r" (pid) : "cc", "memory"); +} + +static inline unsigned long +fcse_va_to_mva(struct mm_struct *mm, unsigned long va) +{ + if (cache_is_vivt() && va < FCSE_PID_TASK_SIZE) { + fcse_check_context(mm); + return mm->context.fcse.pid | va; + } + return va; +} + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +struct fcse_user { + struct mm_struct *mm; + unsigned count; +}; +extern struct fcse_user fcse_pids_user[]; +int fcse_switch_mm_inner(struct mm_struct *prev, struct mm_struct *next); +void fcse_pid_reference(unsigned pid); + +static inline int fcse_switch_mm(struct mm_struct *prev, struct mm_struct *next) +{ + if (!cache_is_vivt()) + return 0; + + return fcse_switch_mm_inner(prev, next); +} + +static inline int fcse_mm_in_cache(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + int res; + fcse_check_context(mm); + res = test_bit(fcse_pid, fcse_pids_cache_dirty) + && fcse_pids_user[fcse_pid].mm == mm; + return res; +} + +static inline unsigned long +fcse_check_mmap_addr(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ + const unsigned long stack_base = ALIGN(mm->start_stack, PAGE_SIZE) + - current->signal->rlim[RLIMIT_STACK].rlim_cur; + + if (addr + len <= stack_base) + return addr; + + return fcse_check_mmap_inner(mm, start_addr, addr, len, fl); +} + +#else /* CONFIG_ARM_FCSE_GUARANTEED */ +static inline int +fcse_switch_mm(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned fcse_pid; + + if (!cache_is_vivt()) + return 0; + + fcse_pid = next->context.fcse.pid >> FCSE_PID_SHIFT; + set_bit(fcse_pid, fcse_pids_cache_dirty); + fcse_pid_set(next->context.fcse.pid); + return 0; +} + +static inline int fcse_mm_in_cache(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + return test_bit(fcse_pid, fcse_pids_cache_dirty); +} + +static inline unsigned long +fcse_check_mmap_addr(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ + if (addr + len <= FCSE_TASK_SIZE) + return addr; + + return fcse_check_mmap_inner(mm, start_addr, addr, len, fl); +} +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ + +static inline void fcse_mark_dirty(struct mm_struct *mm) +{ + if (cache_is_vivt()) { + set_bit(mm->context.fcse.pid >> FCSE_PID_SHIFT, + fcse_pids_cache_dirty); + FCSE_BUG_ON(!fcse_mm_in_cache(mm)); + } +} + +#else /* ! CONFIG_ARM_FCSE */ +#define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); }) +#define fcse_mark_dirty(mm) do { (void)(mm); } while(0) +#define fcse_switch_mm(prev, next) (1) +#define fcse_flush_all_start(mm) (0) +#define fcse_flush_all_done(mm, seq, dirty) (1) +#define fcse_notify_flush_all() do { } while (0) +#define fcse_mm_in_cache(mm) \ + (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) +#endif /* ! CONFIG_ARM_FCSE */ + +#ifdef CONFIG_ARM_FCSE_MESSAGES +void fcse_notify_segv(struct mm_struct *mm, + unsigned long addr, struct pt_regs *regs); +#else /* !FCSE_MESSAGES */ +#define fcse_notify_segv(mm, addr, regs) do { } while(0) +#endif /* !FCSE_MESSAGES */ + +#endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/ipipe.h b/arch/arm/include/asm/ipipe.h new file mode 100644 index 0000000..8ae0cb3 --- /dev/null +++ b/arch/arm/include/asm/ipipe.h @@ -0,0 +1,274 @@ +/* -*- linux-c -*- + * arch/arm/include/asm/ipipe.h + * + * Copyright (C) 2002-2005 Philippe Gerum. + * Copyright (C) 2005 Stelian Pop. + * Copyright (C) 2006-2008 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __ARM_IPIPE_H +#define __ARM_IPIPE_H + +#ifdef CONFIG_IPIPE + +#include <linux/ipipe_percpu.h> +#include <mach/irqs.h> /* For __IPIPE_FEATURE_PIC_MUTE */ + +#define IPIPE_ARCH_STRING "1.16-01" +#define IPIPE_MAJOR_NUMBER 1 +#define IPIPE_MINOR_NUMBER 16 +#define IPIPE_PATCH_NUMBER 1 + +#ifdef CONFIG_SMP +#error "I-pipe/arm: SMP not yet implemented" +#define ipipe_processor_id() (current_thread_info()->cpu) +#else /* !CONFIG_SMP */ +#define ipipe_processor_id() 0 +#endif /* CONFIG_SMP */ + +#define smp_processor_id_hw() ipipe_processor_id() + +#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + +#define prepare_arch_switch(next) \ + do { \ + local_irq_enable_hw(); \ + ipipe_schedule_notify(current, next); \ + } while(0) + +#define task_hijacked(p) \ + ({ \ + int x = !ipipe_root_domain_p; \ + clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \ + x; \ + }) + +#define ipipe_mm_switch_protect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#define ipipe_mm_switch_unprotect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + +#define prepare_arch_switch(next) \ + do { \ + ipipe_schedule_notify(current ,next); \ + local_irq_disable_hw(); \ + } while(0) + +#define task_hijacked(p) \ + ({ \ + int x = !ipipe_root_domain_p; \ + __clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \ + if (!x) local_irq_enable_hw(); x; \ + }) + +#define ipipe_mm_switch_protect(flags) \ + local_irq_save_hw_cond(flags) + +#define ipipe_mm_switch_unprotect(flags) \ + local_irq_restore_hw_cond(flags) + +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + +extern unsigned long arm_return_addr(int level); + +#define BROKEN_BUILTIN_RETURN_ADDRESS +#define __BUILTIN_RETURN_ADDRESS0 arm_return_addr(0) +#define __BUILTIN_RETURN_ADDRESS1 arm_return_addr(1) + + +struct ipipe_domain; + +#define IPIPE_TSC_TYPE_NONE 0 +#define IPIPE_TSC_TYPE_FREERUNNING 1 +#define IPIPE_TSC_TYPE_DECREMENTER 2 + +struct __ipipe_tscinfo { + unsigned type; + union { + struct { + unsigned *counter; /* Hw counter physical address */ + unsigned mask; /* Significant bits in the hw counter. */ + unsigned long long *tsc; /* 64 bits tsc value. */ + } fr; + struct { + unsigned *counter; /* Hw counter physical address */ + unsigned mask; /* Significant bits in the hw counter. */ + unsigned *last_cnt; /* Counter value when updating + tsc value. */ + unsigned long long *tsc; /* 64 bits tsc value. */ + } dec; + } u; +}; + +struct ipipe_sysinfo { + + int ncpus; /* Number of CPUs on board */ + u64 cpufreq; /* CPU frequency (in Hz) */ + + /* Arch-dependent block */ + + struct { + unsigned tmirq; /* Timer tick IRQ */ + u64 tmfreq; /* Timer frequency */ + struct __ipipe_tscinfo tsc; /* exported data for u.s. tsc */ + } archdep; +}; + +DECLARE_PER_CPU(struct mm_struct *,ipipe_active_mm); +/* arch specific stuff */ +extern void *__ipipe_tsc_area; +extern int __ipipe_mach_timerint; +extern int __ipipe_mach_timerstolen; +extern unsigned int __ipipe_mach_ticks_per_jiffy; +extern void __ipipe_mach_acktimer(void); +extern unsigned long long __ipipe_mach_get_tsc(void); +extern void __ipipe_mach_set_dec(unsigned long); +extern void __ipipe_mach_release_timer(void); +extern unsigned long __ipipe_mach_get_dec(void); +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info); +int __ipipe_check_tickdev(const char *devname); + +#define ipipe_read_tsc(t) do { t = __ipipe_mach_get_tsc(); } while (0) +#define __ipipe_read_timebase() __ipipe_mach_get_tsc() + +#define ipipe_cpu_freq() (HZ * __ipipe_mach_ticks_per_jiffy) +#define ipipe_tsc2ns(t) \ +({ \ + unsigned long long delta = (t)*1000; \ + do_div(delta, ipipe_cpu_freq() / 1000000 + 1); \ + (unsigned long)delta; \ +}) +#define ipipe_tsc2us(t) \ +({ \ + unsigned long long delta = (t); \ + do_div(delta, ipipe_cpu_freq() / 1000000 + 1); \ + (unsigned long)delta; \ +}) + +#define ipipe_handle_irq_cond(irq) \ + __ipipe_handle_irq(irq, (struct pt_regs *)1) + +/* Private interface -- Internal use only */ + +#define __ipipe_check_platform() do { } while(0) + +#define __ipipe_init_platform() do { } while(0) + +#define __ipipe_enable_irq(irq) irq_desc[irq].chip->enable(irq) + +#define __ipipe_disable_irq(irq) irq_desc[irq].chip->disable(irq) + +#define __ipipe_hook_critical_ipi(ipd) do { } while(0) + +void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq); + +#ifndef __IPIPE_FEATURE_PIC_MUTE +#define __ipipe_disable_irqdesc(ipd, irq) do { } while (0) +#else /* __IPIPE_FEATURE_PIC_MUTE */ + +typedef unsigned long +__ipipe_irqbits_t[(NR_IRQS + BITS_PER_LONG - 1) / BITS_PER_LONG]; +extern __ipipe_irqbits_t __ipipe_irqbits; + +void __ipipe_disable_irqdesc(struct ipipe_domain *ipd, unsigned irq); + +void ipipe_mute_pic(void); + +void ipipe_unmute_pic(void); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +void __ipipe_enable_pipeline(void); + +void __ipipe_do_critical_sync(unsigned irq, + void *cookie); + +DECLARE_PER_CPU(struct pt_regs, __ipipe_tick_regs); + +int __ipipe_handle_irq(int irq, + struct pt_regs *regs); + +#define ipipe_update_tick_evtdev(evtdev) do { } while (0) + +#define __ipipe_tick_irq __ipipe_mach_timerint + +static inline unsigned long __ipipe_ffnz(unsigned long ul) +{ + return ffs(ul) - 1; +} + +/* When running handlers, enable hw interrupts for all domains but the + * one heading the pipeline, so that IRQs can never be significantly + * deferred for the latter. */ +#define __ipipe_run_isr(ipd, irq) \ +do { \ + if (!__ipipe_pipeline_head_p(ipd)) \ + local_irq_enable_hw(); \ + if (ipd == ipipe_root_domain) { \ + if (likely(!ipipe_virtual_irq_p(irq))) \ + ((void (*)(unsigned, struct pt_regs *)) \ + ipd->irqs[irq].handler) (irq, \ + &__raw_get_cpu_var(__ipipe_tick_regs)); \ + else { \ + irq_enter(); \ + ipd->irqs[irq].handler(irq,ipd->irqs[irq].cookie); \ + irq_exit(); \ + } \ + } else { \ + __clear_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \ + ipd->irqs[irq].handler(irq,ipd->irqs[irq].cookie); \ + __set_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \ + } \ + local_irq_disable_hw(); \ +} while(0) + +#define __ipipe_syscall_watched_p(p, sc) \ + (((p)->flags & PF_EVNOTIFY) || (unsigned long)sc >= __ARM_NR_BASE + 64) + +#define __ipipe_root_tick_p(regs) (!raw_irqs_disabled_flags(regs->ARM_cpsr)) + +#else /* !CONFIG_IPIPE */ + +#define task_hijacked(p) 0 + +#define ipipe_update_tick_evtdev(evtdev) do { } while (0) + +#define smp_processor_id_hw() smp_processor_id() + +#define ipipe_handle_irq_cond(irq) \ + generic_handle_irq(irq) + +#define ipipe_mm_switch_protect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#define ipipe_mm_switch_unprotect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__ARM_IPIPE_H */ diff --git a/arch/arm/include/asm/ipipe_base.h b/arch/arm/include/asm/ipipe_base.h new file mode 100644 index 0000000..20e97a4 --- /dev/null +++ b/arch/arm/include/asm/ipipe_base.h @@ -0,0 +1,108 @@ +/* -*- linux-c -*- + * arch/arm/include/asm/ipipe_base.h + * + * Copyright (C) 2007 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __ARM_IPIPE_BASE_H +#define __ARM_IPIPE_BASE_H + +#include <linux/threads.h> +#include <asm/ptrace.h> +#include <asm/irq.h> + +#define IPIPE_NR_XIRQS NR_IRQS +#define IPIPE_IRQ_ISHIFT 5 /* 25 for 32bits arch. */ + +/* ARM traps */ +#define IPIPE_TRAP_ACCESS 0 /* Data or instruction access exception */ +#define IPIPE_TRAP_SECTION 1 /* Section fault */ +#define IPIPE_TRAP_DABT 2 /* Generic data abort */ +#define IPIPE_TRAP_UNKNOWN 3 /* Unknown exception */ +#define IPIPE_TRAP_BREAK 4 /* Instruction breakpoint */ +#define IPIPE_TRAP_FPU 5 /* Floating point exception */ +#define IPIPE_TRAP_VFP 6 /* VFP floating point exception */ +#define IPIPE_TRAP_UNDEFINSTR 7 /* Undefined instruction */ +#define IPIPE_TRAP_ALIGNMENT 8 /* Unaligned access exception */ +#define IPIPE_NR_FAULTS 9 + +/* Pseudo-vectors used for kernel events */ +#define IPIPE_FIRST_EVENT IPIPE_NR_FAULTS +#define IPIPE_EVENT_SYSCALL (IPIPE_FIRST_EVENT) +#define IPIPE_EVENT_SCHEDULE (IPIPE_FIRST_EVENT + 1) +#define IPIPE_EVENT_SIGWAKE (IPIPE_FIRST_EVENT + 2) +#define IPIPE_EVENT_SETSCHED (IPIPE_FIRST_EVENT + 3) +#define IPIPE_EVENT_INIT (IPIPE_FIRST_EVENT + 4) +#define IPIPE_EVENT_EXIT (IPIPE_FIRST_EVENT + 5) +#define IPIPE_EVENT_CLEANUP (IPIPE_FIRST_EVENT + 6) +#define IPIPE_LAST_EVENT IPIPE_EVENT_CLEANUP +#define IPIPE_NR_EVENTS (IPIPE_LAST_EVENT + 1) + +#ifndef __ASSEMBLY__ + +#include <asm/irqflags.h> + +#ifdef CONFIG_SMP +#error "SMP not implemented." +#define __ipipe_root_status ipipe_root_cpudom_var(status) + +#else /* !CONFIG_SMP */ + +#ifdef CONFIG_VFP +#define __IPIPE_FEATURE_VFP_SAFE 1 +#endif + +#if __GNUC__ >= 4 +/* Alias to ipipe_root_cpudom_var(status) */ +extern unsigned long __ipipe_root_status; +#else +extern unsigned long *const __ipipe_root_status_addr; +#define __ipipe_root_status (*__ipipe_root_status_addr) +#endif + +static inline void __ipipe_stall_root(void) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_root_status |= 1; + local_irq_restore_hw(flags); +} + +static inline unsigned __ipipe_test_root(void) +{ + return __ipipe_root_status & 1; +} + +static inline unsigned __ipipe_test_and_stall_root(void) +{ + unsigned long flags, res; + + local_irq_save_hw(flags); + res = __ipipe_root_status; + __ipipe_root_status = res | 1; + local_irq_restore_hw(flags); + + return res & 1; +} + +#endif /* CONFIG_SMP */ + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ARM_IPIPE_BASE_H */ diff --git a/arch/arm/include/asm/irqflags.h b/arch/arm/include/asm/irqflags.h index 6d09974..0aa4056 100644 --- a/arch/arm/include/asm/irqflags.h +++ b/arch/arm/include/asm/irqflags.h @@ -10,30 +10,30 @@ */ #if __LINUX_ARM_ARCH__ >= 6 -#define raw_local_irq_save(x) \ +#define local_irq_save_hw_notrace(x) \ ({ \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_save\n" \ + "mrs %0, cpsr @ local_irq_save_hw\n" \ "cpsid i" \ : "=r" (x) : : "memory", "cc"); \ }) -#define raw_local_irq_enable() __asm__("cpsie i @ __sti" : : : "memory", "cc") -#define raw_local_irq_disable() __asm__("cpsid i @ __cli" : : : "memory", "cc") -#define local_fiq_enable() __asm__("cpsie f @ __stf" : : : "memory", "cc") -#define local_fiq_disable() __asm__("cpsid f @ __clf" : : : "memory", "cc") +#define local_irq_enable_hw_notrace() __asm__("cpsie i @ __sti" : : : "memory", "cc") +#define local_irq_disable_hw_notrace() __asm__("cpsid i @ __cli" : : : "memory", "cc") +#define local_fiq_enable_hw_notrace() __asm__("cpsie f @ __stf" : : : "memory", "cc") +#define local_fiq_disable_hw_notrace() __asm__("cpsid f @ __clf" : : : "memory", "cc") #else /* * Save the current interrupt enable state & disable IRQs */ -#define raw_local_irq_save(x) \ +#define local_irq_save_hw_notrace(x) \ ({ \ unsigned long temp; \ (void) (&temp == &x); \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_save\n" \ + "mrs %0, cpsr @ local_irq_save_hw\n" \ " orr %1, %0, #128\n" \ " msr cpsr_c, %1" \ : "=r" (x), "=r" (temp) \ @@ -44,11 +44,11 @@ /* * Enable IRQs */ -#define raw_local_irq_enable() \ +#define local_irq_enable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_enable\n" \ + "mrs %0, cpsr @ local_irq_enable_hw\n" \ " bic %0, %0, #128\n" \ " msr cpsr_c, %0" \ : "=r" (temp) \ @@ -59,11 +59,11 @@ /* * Disable IRQs */ -#define raw_local_irq_disable() \ +#define local_irq_disable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_disable\n" \ + "mrs %0, cpsr @ local_irq_disable_hw\n" \ " orr %0, %0, #128\n" \ " msr cpsr_c, %0" \ : "=r" (temp) \ @@ -74,7 +74,7 @@ /* * Enable FIQs */ -#define local_fiq_enable() \ +#define local_fiq_enable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ @@ -89,7 +89,7 @@ /* * Disable FIQs */ -#define local_fiq_disable() \ +#define local_fiq_disable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ @@ -106,19 +106,19 @@ /* * Save the current interrupt enable state. */ -#define raw_local_save_flags(x) \ +#define local_save_flags_hw(x) \ ({ \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_save_flags" \ + "mrs %0, cpsr @ local_save_flags_hw" \ : "=r" (x) : : "memory", "cc"); \ }) /* * restore saved IRQ & FIQ state */ -#define raw_local_irq_restore(x) \ +#define local_irq_restore_hw_notrace(x) \ __asm__ __volatile__( \ - "msr cpsr_c, %0 @ local_irq_restore\n" \ + "msr cpsr_c, %0 @ local_irq_restore_hw\n" \ : \ : "r" (x) \ : "memory", "cc") @@ -128,5 +128,99 @@ (int)((flags) & PSR_I_BIT); \ }) +#define irqs_disabled_hw() \ +({ \ + unsigned long flags; \ + local_save_flags_hw(flags); \ + raw_irqs_disabled_flags(flags); \ +}) + +static inline unsigned long raw_mangle_irq_bits(int virt, unsigned long real) +{ + /* Merge virtual and real interrupt mask bits into a single + 32bit word. */ + return (real & ~(1L << 8)) | ((virt != 0) << 8); +} + +static inline int raw_demangle_irq_bits(unsigned long *x) +{ + int virt = (*x & (1 << 8)) != 0; + *x &= ~(1L << 8); + return virt; +} + +#ifdef CONFIG_IPIPE + +void __ipipe_unstall_root(void); +void __ipipe_restore_root(unsigned long flags); + +/* PSR_I_BIT is bit no. 7 and is set if interrupts are _disabled_ */ +#define raw_local_irq_save(flags) ((flags) = __ipipe_test_and_stall_root() << 7) +#define raw_local_irq_enable() __ipipe_unstall_root() +#define raw_local_irq_disable() __ipipe_stall_root() +#define local_fiq_enable() __ipipe_unstall_root() +#define local_fiq_disable() __ipipe_stall_root() +#define raw_local_save_flags(flags) ((flags) = __ipipe_test_root() << 7) +#define raw_local_irq_restore(flags) __ipipe_restore_root(flags & (1 << 7)) + +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF + +#include <linux/ipipe_trace.h> + +#define local_irq_disable_hw() do { \ + if (!irqs_disabled_hw()) { \ + local_irq_disable_hw_notrace(); \ + ipipe_trace_begin(0x80000000); \ + } \ +} while (0) +#define local_irq_enable_hw() do { \ + if (irqs_disabled_hw()) { \ + ipipe_trace_end(0x80000000); \ + local_irq_enable_hw_notrace(); \ + } \ +} while (0) +#define local_irq_save_hw(x) do { \ + local_save_flags_hw(x); \ + if (!raw_irqs_disabled_flags(x)) { \ + local_irq_disable_hw_notrace(); \ + ipipe_trace_begin(0x80000001); \ + } \ +} while (0) +#define local_irq_restore_hw(x) do { \ + if (!raw_irqs_disabled_flags(x)) \ + ipipe_trace_end(0x80000001); \ + local_irq_restore_hw_notrace(x); \ +} while (0) + +#else /* !CONFIG_IPIPE_TRACE_IRQSOFF */ + +#define local_irq_save_hw(flags) local_irq_save_hw_notrace(flags) +#define local_irq_enable_hw() local_irq_enable_hw_notrace() +#define local_irq_disable_hw() local_irq_disable_hw_notrace() +#define local_fiq_enable_hw() local_fiq_enable_hw_notrace() +#define local_fiq_disable_hw() local_fiq_disable_hw_notrace() +#define local_irq_restore_hw(flags) local_irq_restore_hw_notrace(flags) + +#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */ + +#else /* !CONFIG_IPIPE */ + +#define raw_local_irq_save(flags) local_irq_save_hw_notrace(flags) +#define raw_local_irq_enable() local_irq_enable_hw_notrace() +#define raw_local_irq_disable() local_irq_disable_hw_notrace() +#define local_fiq_enable() local_fiq_enable_hw_notrace() +#define local_fiq_disable() local_fiq_disable_hw_notrace() +#define raw_local_save_flags(flags) local_save_flags_hw(flags) +#define raw_local_irq_restore(flags) local_irq_restore_hw_notrace(flags) + +#define local_irq_save_hw(flags) local_irq_save_hw_notrace(flags) +#define local_irq_enable_hw() local_irq_enable_hw_notrace() +#define local_irq_disable_hw() local_irq_disable_hw_notrace() +#define local_fiq_enable_hw() local_fiq_enable_hw_notrace() +#define local_fiq_disable_hw() local_fiq_disable_hw_notrace() +#define local_irq_restore_hw(flags) local_irq_restore_hw_notrace(flags) + +#endif /* CONFIG_IPIPE */ + #endif #endif diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h index 5421d82..578d189 100644 --- a/arch/arm/include/asm/memory.h +++ b/arch/arm/include/asm/memory.h @@ -33,7 +33,12 @@ */ #define PAGE_OFFSET UL(CONFIG_PAGE_OFFSET) #define TASK_SIZE (UL(CONFIG_PAGE_OFFSET) - UL(0x01000000)) +#ifndef CONFIG_ARM_FCSE #define TASK_UNMAPPED_BASE (UL(CONFIG_PAGE_OFFSET) / 3) +#else /* CONFIG_ARM_FCSE */ +#define TASK_UNMAPPED_BASE UL(0x00800000) +#define FCSE_TASK_SIZE UL(0x02000000) +#endif /* CONFIG_ARM_FCSE */ /* * The maximum size of a 26-bit user space task. diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index b561584..4087d0b 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -7,6 +7,17 @@ typedef struct { #ifdef CONFIG_CPU_HAS_ASID unsigned int id; #endif +#ifdef CONFIG_ARM_FCSE + struct { + unsigned long pid; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned shared_dirty_pages; + unsigned active : 1; + unsigned large : 1; + unsigned high_pages; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + } fcse; +#endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; } mm_context_t; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index de6cefb..edb8922 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -19,6 +19,7 @@ #include <asm/cachetype.h> #include <asm/proc-fns.h> #include <asm-generic/mm_hooks.h> +#include <asm/fcse.h> void __check_kvm_seq(struct mm_struct *mm); @@ -68,11 +69,50 @@ static inline void check_context(struct mm_struct *mm) #endif } -#define init_new_context(tsk,mm) 0 +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE + int fcse_pid; + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (!mm->context.fcse.large) { + fcse_pid = fcse_pid_alloc(mm); + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; + } else { + /* We are normally forking a process vith a virtual address + space larger than 32 MB, so its pid should be 0. */ + FCSE_BUG_ON(mm->context.fcse.pid); + fcse_pid_reference(0); + } + /* If we are forking, set_pte_at will restore the correct high pages + count, and shared writable pages are write-protected again. */ + mm->context.fcse.shared_dirty_pages = 0; + mm->context.fcse.high_pages = 0; + mm->context.fcse.active = 0; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + fcse_pid = fcse_pid_alloc(mm); + if (fcse_pid < 0) + return fcse_pid; + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#endif /* CONFIG_ARM_FCSE */ + + return 0; +} #endif -#define destroy_context(mm) do { } while(0) +static inline void destroy_context(struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + FCSE_BUG_ON(mm->context.fcse.shared_dirty_pages); + FCSE_BUG_ON(mm->context.fcse.high_pages); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + fcse_pid_free(mm); +#endif /* CONFIG_ARM_FCSE */ +} /* * This is called when "tsk" is about to enter lazy TLB mode. @@ -95,11 +135,11 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) * actually changed. */ static inline void -switch_mm(struct mm_struct *prev, struct mm_struct *next, - struct task_struct *tsk) +__switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *tsk) { #ifdef CONFIG_MMU - unsigned int cpu = smp_processor_id(); + unsigned int cpu = smp_processor_id_hw(); #ifdef CONFIG_SMP /* check for possible thread migration */ @@ -109,14 +149,64 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, #endif if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) { check_context(next); - cpu_switch_mm(next->pgd, next); - if (cache_is_vivt()) +#if defined(CONFIG_IPIPE) + if (ipipe_root_domain_p) { + /* mark mm state as undefined. */ + per_cpu(ipipe_active_mm, cpu) = NULL; + barrier(); + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(prev, next)); + barrier(); + per_cpu(ipipe_active_mm, cpu) = next; + while (test_and_clear_thread_flag(TIF_MMSWITCH_INT)) { + /* mark mm state as undefined. */ + per_cpu(ipipe_active_mm, cpu) = NULL; + barrier(); + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(NULL, next)); + barrier(); + per_cpu(ipipe_active_mm, cpu) = next; + } + } else +#endif /* CONFIG_IPIPE */ + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(prev, next)); +#if defined(CONFIG_IPIPE) && defined(CONFIG_ARM_FCSE) + if (tsk) + set_tsk_thread_flag(tsk, TIF_SWITCHED); +#endif /* CONFIG_IPIPE && CONFIG_ARM_FCSE */ + if (cache_is_vivt() && prev) cpumask_clear_cpu(cpu, mm_cpumask(prev)); - } + } else + fcse_mark_dirty(next); #endif } +static inline void +switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *tsk) +{ +#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + unsigned long flags; + local_irq_save_hw(flags); +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + __switch_mm(prev, next, tsk); +#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + local_irq_restore_hw(flags); +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ +} + #define deactivate_mm(tsk,mm) do { } while (0) + +#ifndef CONFIG_ARM_FCSE_BEST_EFFORT #define activate_mm(prev,next) switch_mm(prev, next, NULL) +#else +#define activate_mm(prev,next) \ + ({ \ + switch_mm(prev, next, NULL); \ + next->context.fcse.active = 1; \ + FCSE_BUG_ON(current->mm == next && !fcse_mm_in_cache(next)); \ + }) +#endif #endif diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 1139768..3d71ffb 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -112,6 +112,8 @@ #define LIBRARY_TEXT_START 0x0c000000 #ifndef __ASSEMBLY__ +#include <asm/fcse.h> + extern void __pte_error(const char *file, int line, unsigned long val); extern void __pmd_error(const char *file, int line, unsigned long val); extern void __pgd_error(const char *file, int line, unsigned long val); @@ -248,6 +250,40 @@ extern pgprot_t pgprot_kernel; #define __S111 __PAGE_SHARED_EXEC #ifndef __ASSEMBLY__ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#define fcse_account_page_removal(mm, addr, val) do { \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && ((_val) & L_PTE_SHARED)) \ + --_mm->context.fcse.shared_dirty_pages; \ + if (pte_present(_val) && _addr < TASK_SIZE) { \ + if (_addr >= FCSE_TASK_SIZE) \ + --_mm->context.fcse.high_pages; \ + } \ +} while (0) + +#define fcse_account_page_addition(mm, addr, val) ({ \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && (_val & L_PTE_SHARED)) { \ + if ((_val & (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + != (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + _val &= ~L_PTE_SHARED; \ + else \ + ++_mm->context.fcse.shared_dirty_pages; \ + } \ + if (pte_present(_val) \ + && _addr < TASK_SIZE && _addr >= FCSE_TASK_SIZE) \ + ++_mm->context.fcse.high_pages; \ + _val; \ +}) +#else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ +#define fcse_account_page_removal(mm, addr, val) do { } while (0) +#define fcse_account_page_addition(mm, addr, val) (val) +#endif /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ + /* * ZERO_PAGE is a global shared page that is always zero: used * for zero-mapped memory areas etc.. @@ -259,7 +295,10 @@ extern struct page *empty_zero_page; #define pfn_pte(pfn,prot) (__pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot))) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte_ext(ptep, __pte(0), 0) +#define pte_clear(mm,addr,ptep) do { \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + set_pte_ext(ptep, __pte(0), 0); \ +} while (0) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) #define pte_offset_kernel(dir,addr) (pmd_page_vaddr(*(dir)) + __pte_index(addr)) @@ -278,9 +317,13 @@ extern struct page *empty_zero_page; #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext) -#define set_pte_at(mm,addr,ptep,pteval) do { \ - set_pte_ext(ptep, pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ - } while (0) +#define set_pte_at(mm,addr,ptep,pteval) do { \ + unsigned long _pteval = (pteval); \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + pte_val(_pteval) = \ + fcse_account_page_addition(mm, addr, pte_val(_pteval)); \ + set_pte_ext(ptep, _pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ +} while (0) /* * The following only work if pte_present() is true. @@ -372,10 +415,14 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd) /* to find an entry in a page-table-directory */ #define pgd_index(addr) ((addr) >> PGDIR_SHIFT) -#define pgd_offset(mm, addr) ((mm)->pgd+pgd_index(addr)) +#define pgd_offset(mm, addr) \ + ({ \ + struct mm_struct *_mm = (mm); \ + (_mm->pgd + pgd_index(fcse_va_to_mva(_mm, (addr)))); \ + }) /* to find an entry in a kernel page-table-directory */ -#define pgd_offset_k(addr) pgd_offset(&init_mm, addr) +#define pgd_offset_k(addr) (init_mm.pgd + pgd_index(addr)) /* Find an entry in the second-level page table.. */ #define pmd_offset(dir, addr) ((pmd_t *)(dir)) diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h index 8fdae9b..0c280ca 100644 --- a/arch/arm/include/asm/proc-fns.h +++ b/arch/arm/include/asm/proc-fns.h @@ -261,12 +261,13 @@ #ifdef CONFIG_MMU -#define cpu_switch_mm(pgd,mm) cpu_do_switch_mm(virt_to_phys(pgd),mm) +#define cpu_switch_mm(pgd,mm,flush) \ + cpu_do_switch_mm(virt_to_phys(pgd), (mm), (flush)) #define cpu_get_pgd() \ ({ \ unsigned long pg; \ - __asm__("mrc p15, 0, %0, c2, c0, 0" \ + __asm__ __volatile__ ("mrc p15, 0, %0, c2, c0, 0" \ : "=r" (pg) : : "cc"); \ pg &= ~0x3fff; \ (pgd_t *)phys_to_virt(pg); \ diff --git a/arch/arm/include/asm/processor.h b/arch/arm/include/asm/processor.h index 6a89567..bc33df1 100644 --- a/arch/arm/include/asm/processor.h +++ b/arch/arm/include/asm/processor.h @@ -23,9 +23,14 @@ #include <asm/types.h> #ifdef __KERNEL__ +#ifndef CONFIG_ARM_FCSE #define STACK_TOP ((current->personality & ADDR_LIMIT_32BIT) ? \ TASK_SIZE : TASK_SIZE_26) #define STACK_TOP_MAX TASK_SIZE +#else /* CONFIG_ARM_FCSE */ +#define STACK_TOP FCSE_TASK_SIZE +#define STACK_TOP_MAX FCSE_TASK_SIZE +#endif /* CONFIG_ARM_FCSE */ #endif union debug_insn { diff --git a/arch/arm/include/asm/resource.h b/arch/arm/include/asm/resource.h index 734b581..5ba5042 100644 --- a/arch/arm/include/asm/resource.h +++ b/arch/arm/include/asm/resource.h @@ -1,6 +1,16 @@ #ifndef _ARM_RESOURCE_H #define _ARM_RESOURCE_H +/* + * When FCSE is enabled, reduce the default stack size to 1MB, and maximum + * to 16MB, the address space is only 32MB. + */ +#ifdef CONFIG_ARM_FCSE +#define _STK_LIM (1024*1024) + +#define _STK_LIM_MAX (16*1024*1024) +#endif /* CONFIG_ARM_FCSE */ + #include <asm-generic/resource.h> #endif diff --git a/arch/arm/include/asm/system.h b/arch/arm/include/asm/system.h index 058e7e9..6ba023d 100644 --- a/arch/arm/include/asm/system.h +++ b/arch/arm/include/asm/system.h @@ -218,10 +218,19 @@ static inline void set_copro_access(unsigned int val) */ extern struct task_struct *__switch_to(struct task_struct *, struct thread_info *, struct thread_info *); +#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH #define switch_to(prev,next,last) \ do { \ - last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ + local_irq_disable_hw_cond(); \ + last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ + local_irq_enable_hw_cond(); \ } while (0) +#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ +#define switch_to(prev,next,last) \ +do { \ + last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ +} while (0) +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ #if defined(CONFIG_CPU_SA1100) || defined(CONFIG_CPU_SA110) /* @@ -282,17 +291,17 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int size #error SMP is not supported on this platform #endif case 1: - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = *(volatile unsigned char *)ptr; *(volatile unsigned char *)ptr = x; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); break; case 4: - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = *(volatile unsigned long *)ptr; *(volatile unsigned long *)ptr = x; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); break; #else case 1: diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 2dfb7d7..e20256d 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -143,6 +143,12 @@ extern void vfp_sync_state(struct thread_info *thread); #define TIF_MEMDIE 18 #define TIF_FREEZE 19 #define TIF_RESTORE_SIGMASK 20 +#ifdef CONFIG_IPIPE +#define TIF_MMSWITCH_INT 21 +#ifdef CONFIG_ARM_FCSE +#define TIF_SWITCHED 22 +#endif /* CONFIG_ARM_FCSE */ +#endif /* CONFIG_IPIPE */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -152,6 +158,12 @@ extern void vfp_sync_state(struct thread_info *thread); #define _TIF_USING_IWMMXT (1 << TIF_USING_IWMMXT) #define _TIF_FREEZE (1 << TIF_FREEZE) #define _TIF_RESTORE_SIGMASK (1 << TIF_RESTORE_SIGMASK) +#ifdef CONFIG_IPIPE +#define _TIF_MMSWITCH_INT (1 << TIF_MMSWITCH_INT) +#ifdef CONFIG_ARM_FCSE +#define _TIF_SWITCHED (1 << TIF_SWITCHED) +#endif /* CONFIG_ARM_FCSE */ +#endif /* CONFIG_IPIPE */ /* * Change these and you break ASM code in entry-common.S diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h index c2f1605..70a81b1 100644 --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -210,6 +210,7 @@ #ifndef __ASSEMBLY__ #include <linux/sched.h> +#include <asm/fcse.h> struct cpu_tlb_fns { void (*flush_user_range)(unsigned long, unsigned long, struct vm_area_struct *); @@ -384,7 +385,8 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) const int zero = 0; const unsigned int __tlb_flag = __cpu_tlb_flags; - uaddr = (uaddr & PAGE_MASK) | ASID(vma->vm_mm); + uaddr = (fcse_va_to_mva(vma->vm_mm, uaddr) & PAGE_MASK) + | ASID(vma->vm_mm); if (tlb_flag(TLB_WB)) dsb(); @@ -505,7 +507,15 @@ static inline void clean_pmd_entry(pmd_t *pmd) /* * Convert calls to our calling convention. */ -#define local_flush_tlb_range(vma,start,end) __cpu_flush_user_tlb_range(start,end,vma) +#define local_flush_tlb_range(vma, start, end) \ + ({ \ + struct mm_struct *_mm = (vma)->vm_mm; \ + unsigned long _start, _end; \ + _start = fcse_va_to_mva(_mm, start); \ + _end = fcse_va_to_mva(_mm, end); \ + __cpu_flush_user_tlb_range(_start, _end, vma); \ + }) + #define local_flush_tlb_kernel_range(s,e) __cpu_flush_kern_tlb_range(s,e) #ifndef CONFIG_SMP diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile index dd00f74..ca557ee 100644 --- a/arch/arm/kernel/Makefile +++ b/arch/arm/kernel/Makefile @@ -20,7 +20,7 @@ obj-y := compat.o elf.o entry-armv.o entry-common.o irq.o \ obj-$(CONFIG_OC_ETM) += etm.o obj-$(CONFIG_ISA_DMA_API) += dma.o -obj-$(CONFIG_ARCH_ACORN) += ecard.o +obj-$(CONFIG_ARCH_ACORN) += ecard.o obj-$(CONFIG_FIQ) += fiq.o obj-$(CONFIG_MODULES) += armksyms.o module.o obj-$(CONFIG_ARTHUR) += arthur.o @@ -55,5 +55,7 @@ endif head-y := head$(MMUEXT).o obj-$(CONFIG_DEBUG_LL) += debug.o obj-$(CONFIG_EARLY_PRINTK) += early_printk.o +obj-$(CONFIG_ARM_FCSE) += fcse.o +obj-$(CONFIG_IPIPE) += ipipe.o extra-y := $(head-y) init_task.o vmlinux.lds diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 6c5cf36..f485f8c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -4,6 +4,7 @@ * Copyright (C) 1996,1997,1998 Russell King. * ARM700 fix by Matthew Godbolt (linux-user@willothewisp.demon.co.uk) * nommu support by Hyok S. Choi (hyok.choi@samsung.com) + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -29,14 +30,22 @@ * Interrupt handling. Preserves r7, r8, r9 */ .macro irq_handler +#ifdef CONFIG_IPIPE + mov r0, #2 +#endif get_irqnr_preamble r5, lr -1: get_irqnr_and_base r0, r6, r5, lr +1: get_irqnr_and_base r1, r6, r5, lr + movne r0, r1 movne r1, sp @ @ routine called with r0 = irq number, r1 = struct pt_regs * @ adrne lr, BSYM(1b) +#ifdef CONFIG_IPIPE + bne __ipipe_grab_irq +#else bne asm_do_IRQ +#endif #ifdef CONFIG_SMP /* @@ -45,18 +54,22 @@ * this macro assumes that irqstat (r6) and base (r5) are * preserved from get_irqnr_and_base above */ - test_for_ipi r0, r6, r5, lr + test_for_ipi r1, r6, r5, lr movne r0, sp adrne lr, BSYM(1b) bne do_IPI #ifdef CONFIG_LOCAL_TIMERS - test_for_ltirq r0, r6, r5, lr + test_for_ltirq r1, r6, r5, lr movne r0, sp adrne lr, BSYM(1b) bne do_local_timer #endif #endif +#ifdef CONFIG_IPIPE + cmp r0, #2 + bleq __ipipe_check_root_interruptible +#endif .endm @@ -78,6 +91,21 @@ mov r1, #\reason .endm + .macro fcse_dabt_mva_to_va +#ifdef CONFIG_ARM_FCSE + @ + @ If FCSE is enabled the data abort fault address must be + @ converted from MVA to VA. This must happen before the irqs are + @ enabled if PREEMPT is enabled, as context switches may + @ change the FCSE pid. + @ + mrc p15, 0, r2, c13, c0, 0 + eor r2, r2, r0 + tst r2, #0xfe000000 + moveq r0, r2 +#endif + .endm + __pabt_invalid: inv_entry BAD_PREFETCH b common_invalid @@ -193,6 +221,7 @@ __dabt_svc: #else bl CPU_DABORT_HANDLER #endif + fcse_dabt_mva_to_va @ @ set desired IRQ state, then call main handler @@ -220,14 +249,26 @@ __irq_svc: #ifdef CONFIG_PREEMPT get_thread_info tsk +#ifndef CONFIG_IPIPE ldr r8, [tsk, #TI_PREEMPT] @ get preempt count add r7, r8, #1 @ increment it str r7, [tsk, #TI_PREEMPT] #endif +#endif irq_handler +#ifdef CONFIG_IPIPE + cmp r0, #0 + ldreq r4, [sp, #S_PSR] @ irqs are already disabled + beq __ipipe_fast_svc_irq_exit +#endif + #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE str r8, [tsk, #TI_PREEMPT] @ restore preempt count +#else + ldr r8, [tsk, #TI_PREEMPT] +#endif ldr r0, [tsk, #TI_FLAGS] @ get flags teq r8, #0 @ if preempt count != 0 movne r0, #0 @ force flags to 0 @@ -239,6 +280,9 @@ __irq_svc: tst r4, #PSR_I_BIT bleq trace_hardirqs_on #endif +#ifdef CONFIG_IPIPE +__ipipe_fast_svc_irq_exit: +#endif svc_exit r4 @ return from exception UNWIND(.fnend ) ENDPROC(__irq_svc) @@ -248,12 +292,16 @@ ENDPROC(__irq_svc) #ifdef CONFIG_PREEMPT svc_preempt: mov r8, lr +#ifndef CONFIG_IPIPE 1: bl preempt_schedule_irq @ irq en/disable is done inside +#else /* CONFIG_IPIPE */ +1: bl __ipipe_preempt_schedule_irq @ irq en/disable is done inside +#endif /* CONFIG_IPIPE */ ldr r0, [tsk, #TI_FLAGS] @ get new tasks TI_FLAGS tst r0, #_TIF_NEED_RESCHED moveq pc, r8 @ go again b 1b -#endif +#endif /* CONFIG_PREEMPT */ .align 5 __und_svc: @@ -266,6 +314,17 @@ __und_svc: svc_entry #endif +#ifdef CONFIG_IPIPE + /* Beware, do_undefinstr in arch/arm/traps.c will not signal the trap + when in svc mode because of this hunk. */ + mov r4, r2 + mov r0, #7 @ r0 = IPIPE_TRAP_UNDEFINSTR + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + bne 1f + mov r2, r4 +#endif /* CONFIG_IPIPE */ @ @ call emulation code, which returns using r9 if it has emulated @ the instruction, or the more conventional lr if we are to treat @@ -433,6 +492,7 @@ __dabt_usr: #else bl CPU_DABORT_HANDLER #endif + fcse_dabt_mva_to_va @ @ IRQs on, then call the main handler @@ -451,13 +511,22 @@ __irq_usr: get_thread_info tsk #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE ldr r8, [tsk, #TI_PREEMPT] @ get preempt count add r7, r8, #1 @ increment it str r7, [tsk, #TI_PREEMPT] #endif +#endif irq_handler +#ifdef CONFIG_IPIPE + cmp r0, #0 + bne __ipipe_usr_irq_continue + slow_restore_user_regs @ Fast exit path over non-root domains +__ipipe_usr_irq_continue: +#endif #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE ldr r0, [tsk, #TI_PREEMPT] str r8, [tsk, #TI_PREEMPT] teq r0, r7 @@ -465,6 +534,7 @@ __irq_usr: THUMB( movne r0, #0 ) THUMB( strne r0, [r0] ) #endif +#endif #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_on #endif @@ -580,7 +650,7 @@ call_fpe: mov r7, #1 strb r7, [r10, #TI_USED_CP + 10] @ mark CP#10 as used strb r7, [r10, #TI_USED_CP + 11] @ mark CP#11 as used - b do_vfp @ let VFP handler handle this + b _do_vfp @ let VFP handler handle this 1: #endif tst r0, #0x08000000 @ only CDP/CPRT/LDC/STC have bit 27 @@ -626,8 +696,8 @@ call_fpe: movw_pc lr @ CP#8 movw_pc lr @ CP#9 #ifdef CONFIG_VFP - W(b) do_vfp @ CP#10 (VFP) - W(b) do_vfp @ CP#11 (VFP) + W(b) _do_vfp @ CP#10 (VFP) + W(b) _do_vfp @ CP#11 (VFP) #else movw_pc lr @ CP#10 (VFP) movw_pc lr @ CP#11 (VFP) @@ -662,11 +732,43 @@ call_fpe: #endif do_fpe: +#ifdef CONFIG_IPIPE + mov r4, r0 + mov r5, r2 + mov r6, lr + mov r0, #5 @ r0 = IPIPE_TRAP_FPU + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + movne pc, r9 + mov lr, r6 + mov r2, r5 + mov r0, r4 +#endif enable_irq ldr r4, .LCfp add r10, r10, #TI_FPSTATE @ r10 = workspace ldr pc, [r4] @ Call FP module USR entry point +#ifdef CONFIG_VFP +_do_vfp: +#ifdef CONFIG_IPIPE + mov r4, r0 + mov r5, r2 + mov r6, lr + mov r0, #6 @ r0 = IPIPE_TRAP_VFP + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + movne pc, r9 + mov lr, r6 + mov r2, r5 + mov r0, r4 +#endif + b do_vfp +#endif + + /* * The FP module is called with these registers set: * r0 = instruction @@ -715,6 +817,13 @@ __pabt_usr: ENTRY(ret_from_exception) UNWIND(.fnstart ) UNWIND(.cantunwind ) +#ifdef CONFIG_IPIPE + bl __ipipe_check_root + cmp r0, #0 + bne 1f + slow_restore_user_regs @ Fast exit path over non-root domains +1: +#endif /* CONFIG_IPIPE */ get_thread_info tsk mov why, #0 b ret_to_user @@ -752,7 +861,11 @@ ENTRY(__switch_to) add r4, r2, #TI_CPU_SAVE ldr r0, =thread_notify_head mov r1, #THREAD_NOTIFY_SWITCH +#ifndef CONFIG_IPIPE bl atomic_notifier_call_chain +#else /* !CONFIG_IPIPE */ + bl __ipipe_switch_to_notifier_call_chain +#endif /* !CONFIG_IPIPE */ THUMB( mov ip, r4 ) mov r0, r5 ARM( ldmia r4, {r4 - sl, fp, sp, pc} ) @ Load all regs saved previously @@ -795,6 +908,18 @@ ENDPROC(__switch_to) */ THUMB( .arm ) +#ifdef CONFIG_IPIPE +/* We use the same mechanism as Linux user helpers to store variables related to + TSC emulation, so that they can be used in user-space. */ + .align 5 + .globl __ipipe_tsc_area_start + +__ipipe_tsc_area_start: + .rep 12 + .word 0 + .endr +#endif /* CONFIG_IPIPE */ + .macro usr_ret, reg #ifdef CONFIG_ARM_THUMB bx \reg @@ -836,7 +961,7 @@ __kuser_helper_start: * * #define __kernel_dmb() \ * asm volatile ( "mov r0, #0xffff0fff; mov lr, pc; sub pc, r0, #95" \ - * : : : "r0", "lr","cc" ) + * : : : "r0", "lr","cc" ) */ __kuser_memory_barrier: @ 0xffff0fa0 @@ -1004,7 +1129,7 @@ kuser_cmpxchg_fixup: * #define __kernel_get_tls() \ * ({ register unsigned int __val asm("r0"); \ * asm( "mov r0, #0xffff0fff; mov lr, pc; sub pc, r0, #31" \ - * : "=r" (__val) : : "lr","cc" ); \ + * : "=r" (__val) : : "lr","cc" ); \ * __val; }) */ diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S index 2c1db77..57e3bf1 100644 --- a/arch/arm/kernel/entry-common.S +++ b/arch/arm/kernel/entry-common.S @@ -2,6 +2,7 @@ * linux/arch/arm/kernel/entry-common.S * * Copyright (C) 2000 Russell King + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -22,6 +23,11 @@ * possible here, and this includes saving r0 back into the SVC * stack. */ +#ifdef CONFIG_IPIPE +__ipipe_ret_fast_syscall: + ldr r0, [sp, #S_R0+S_OFF] @ returned r0 + /* fall through */ +#endif ret_fast_syscall: UNWIND(.fnstart ) UNWIND(.cantunwind ) @@ -30,10 +36,19 @@ ret_fast_syscall: tst r1, #_TIF_WORK_MASK bne fast_work_pending +fast_restore_user_regs: /* perform architecture specific actions before user return */ arch_ret_to_user r1, lr restore_user_regs fast = 1, offset = S_OFF + +#ifdef CONFIG_IPIPE +__ipipe_fast_exit_syscall: + ldr r0, [sp, #S_R0+S_OFF] @ returned r0 + disable_irq @ disable interrupts + b fast_restore_user_regs +#endif /* CONFIG_IPIPE */ + UNWIND(.fnend ) /* @@ -46,13 +61,16 @@ work_pending: bne work_resched tst r1, #_TIF_SIGPENDING|_TIF_NOTIFY_RESUME beq no_work_pending + enable_irq_cond mov r0, sp @ 'regs' mov r2, why @ 'syscall' bl do_notify_resume b ret_slow_syscall @ Check work again work_resched: + enable_irq_cond bl schedule + /* * "slow" syscall return path. "why" tells us if this was a real syscall. */ @@ -63,16 +81,14 @@ ret_slow_syscall: tst r1, #_TIF_WORK_MASK bne work_pending no_work_pending: - /* perform architecture specific actions before user return */ - arch_ret_to_user r1, lr - - restore_user_regs fast = 0, offset = 0 + slow_restore_user_regs ENDPROC(ret_to_user) /* * This is how we return from a fork. */ ENTRY(ret_from_fork) + enable_irq_cond bl schedule_tail get_thread_info tsk ldr r1, [tsk, #TI_FLAGS] @ check for syscall tracing @@ -169,8 +185,8 @@ ftrace_stub: *----------------------------------------------------------------------------- */ - /* If we're optimising for StrongARM the resulting code won't - run on an ARM7 and we can save a couple of instructions. + /* If we're optimising for StrongARM the resulting code won't + run on an ARM7 and we can save a couple of instructions. --pb */ #ifdef CONFIG_CPU_ARM710 #define A710(code...) code @@ -275,6 +291,18 @@ ENTRY(vector_swi) #endif stmdb sp!, {r4, r5} @ push fifth and sixth args +#ifdef CONFIG_IPIPE + stmfd sp!, {r0-r3, ip} + sub sp, sp, #4 + add r1, sp, #S_OFF+24 + mov r0, scno + bl __ipipe_syscall_root + cmp r0, #0 + add sp, sp, #4 + ldmfd sp!, {r0-r3, ip} + blt __ipipe_ret_fast_syscall + bgt __ipipe_fast_exit_syscall +#endif /* CONFIG_IPIPE */ tst ip, #_TIF_SYSCALL_TRACE @ are we tracing syscalls? bne __sys_trace @@ -286,7 +314,7 @@ ENTRY(vector_swi) 2: mov why, #0 @ no longer a real syscall cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE) eor r0, scno, #__NR_SYSCALL_BASE @ put OS number back - bcs arm_syscall + bcs arm_syscall b sys_ni_syscall @ not private func ENDPROC(vector_swi) @@ -322,6 +350,9 @@ __sys_trace_return: __cr_alignment: .word cr_alignment #endif +#ifdef CONFIG_IPIPE + .word __ipipe_syscall_root +#endif .ltorg /* @@ -475,3 +506,28 @@ ENTRY(sys_oabi_call_table) #endif + +#ifdef CONFIG_FRAME_POINTER + + .text + .align 0 + .type arm_return_addr %function + .global arm_return_addr + +arm_return_addr: + mov ip, r0 + mov r0, fp +3: + cmp r0, #0 + beq 1f @ frame list hit end, bail + cmp ip, #0 + beq 2f @ reached desired frame + ldr r0, [r0, #-12] @ else continue, get next fp + sub ip, ip, #1 + b 3b +2: + ldr r0, [r0, #-4] @ get target return address +1: + mov pc, lr + +#endif diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 7e9ed1e..f465901 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -22,7 +22,7 @@ @ #define S_OFF 8 -/* +/* * The SWI code relies on the fact that R0 is at the bottom of the stack * (due to slow/fast restore user regs). */ @@ -163,6 +163,12 @@ .endm #endif /* !CONFIG_THUMB2_KERNEL */ + .macro slow_restore_user_regs + /* perform architecture specific actions before user return */ + arch_ret_to_user r1, lr + restore_user_regs fast = 0, offset = 0 + .endm + /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6. diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c new file mode 100644 index 0000000..1b9550c --- /dev/null +++ b/arch/arm/kernel/fcse.c @@ -0,0 +1,465 @@ +#include <linux/bitops.h> +#include <linux/memory.h> +#include <linux/spinlock.h> +#include <linux/mm.h> +#include <linux/kernel_stat.h> +#include <linux/mman.h> +#include <linux/dcache.h> +#include <linux/fs.h> + +#include <asm/fcse.h> +#include <asm/cacheflush.h> +#include <asm/tlbflush.h> + +#define NR_PIDS (TASK_SIZE / FCSE_PID_TASK_SIZE) +#define PIDS_LONGS ((NR_PIDS + BITS_PER_LONG - 1) / BITS_PER_LONG) + +static IPIPE_DEFINE_SPINLOCK(fcse_lock); +static unsigned long fcse_pids_bits[PIDS_LONGS]; +unsigned long fcse_pids_cache_dirty[PIDS_LONGS]; +EXPORT_SYMBOL(fcse_pids_cache_dirty); + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static unsigned random_pid; +struct fcse_user fcse_pids_user[NR_PIDS]; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + +static inline void fcse_pid_reference_inner(unsigned pid) +{ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (++fcse_pids_user[pid].count == 1) +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __set_bit(pid, fcse_pids_bits); +} + +static inline void fcse_pid_dereference(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#ifndef CONFIG_IPIPE + /* FIXME: Why ? */ + FCSE_BUG_ON(fcse_mm_in_cache(mm)); +#endif /* CONFIG_IPIPE */ + if (--fcse_pids_user[fcse_pid].count == 0) + __clear_bit(fcse_pid, fcse_pids_bits); + + /* + * The following means we suppose that by the time this + * function is called, this mm is out of cache: + * - when the caller is destroy_context, exit_mmap is called + * by mmput before, which flushes the cache; + * - when the caller is fcse_relocate_mm_to_pid from + * fcse_switch_mm_inner, we only relocate when the mm is out + * of cache; + * - when the caller is fcse_relocate_mm_to_pid from + * fcse_relocate_mm_to_null_pid, we flush the cache in this + * function. + */ + if (fcse_pids_user[fcse_pid].mm == mm) { + fcse_pids_user[fcse_pid].mm = NULL; + __clear_bit(fcse_pid, fcse_pids_cache_dirty); + } +#else /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __clear_bit(fcse_pid, fcse_pids_bits); + __clear_bit(fcse_pid, fcse_pids_cache_dirty); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +} + +static inline unsigned find_free_pid(unsigned long bits[]) +{ + unsigned fcse_pid; + + fcse_pid = find_next_zero_bit(bits, NR_PIDS, 1); + if (fcse_pid == NR_PIDS) + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ + if (!test_bit(0, bits)) + fcse_pid = 0; + + return fcse_pid; +} + +void fcse_pid_free(struct mm_struct *mm) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_dereference(mm); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +int fcse_pid_alloc(struct mm_struct *mm) +{ + unsigned long flags; + unsigned fcse_pid; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid = find_free_pid(fcse_pids_bits); + if (fcse_pid == NR_PIDS) { + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if(++random_pid == NR_PIDS) + random_pid = 0; + fcse_pid = random_pid; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + spin_unlock_irqrestore(&fcse_lock, flags); +#ifdef CONFIG_ARM_FCSE_MESSAGES + printk(KERN_WARNING "FCSE: system would exceed the %lu processes limit.\n", + NR_PIDS); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + return -EAGAIN; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ + } + fcse_pid_reference_inner(fcse_pid); +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + FCSE_BUG_ON(fcse_mm_in_cache(mm)); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + spin_unlock_irqrestore(&fcse_lock, flags); + + return fcse_pid; +} + +static inline void fcse_clear_dirty_all(void) +{ + switch(ARRAY_SIZE(fcse_pids_cache_dirty)) { + case 4: + fcse_pids_cache_dirty[3] = 0UL; + case 3: + fcse_pids_cache_dirty[2] = 0UL; + case 2: + fcse_pids_cache_dirty[1] = 0UL; + case 1: + fcse_pids_cache_dirty[0] = 0UL; + } +} + +unsigned fcse_flush_all_start(struct mm_struct *mm) +{ + if (!cache_is_vivt()) + return 0; + +#ifndef CONFIG_ARM_FCSE_PREEMPT_FLUSH + preempt_disable(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + +#if defined(CONFIG_IPIPE) + clear_ti_thread_flag(current_thread_info(), TIF_SWITCHED); +#elif defined(CONFIG_ARM_FCSE_PREEMPT_FLUSH) + return nr_context_switches(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + + return 0; +} + +noinline int +fcse_flush_all_done(struct mm_struct *mm, unsigned seq, unsigned dirty) +{ + unsigned long flags; + int done = 1; + + if (!cache_is_vivt()) + return 1; + + spin_lock_irqsave(&fcse_lock, flags); +#if defined(CONFIG_IPIPE) + if (test_ti_thread_flag(current_thread_info(), TIF_SWITCHED)) { + if (fcse_mm_in_cache(mm)) { + done = 0; + goto ret; + } + } else +#elif defined(CONFIG_ARM_FCSE_PREEMPT_FLUSH) + if (seq != nr_context_switches()) { + if (fcse_mm_in_cache(mm)) { + done = 0; + goto ret; + } + } else +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + fcse_clear_dirty_all(); + + if (dirty && current->mm != &init_mm && current->mm) { + unsigned fcse_pid = + current->mm->context.fcse.pid >> FCSE_PID_SHIFT; + __set_bit(fcse_pid, fcse_pids_cache_dirty); + } + + ret: + spin_unlock_irqrestore(&fcse_lock, flags); +#ifndef CONFIG_ARM_FCSE_PREEMPT_FLUSH + preempt_enable(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + return done; +} + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +/* Called with preemption disabled, mm->mmap_sem being held for writing. */ +static noinline int fcse_relocate_mm_to_pid(struct mm_struct *mm, int fcse_pid) +{ + const unsigned len = pgd_index(FCSE_TASK_SIZE) * sizeof(pgd_t); + unsigned long flags; + pgd_t *from, *to; + + spin_lock_irqsave(&fcse_lock, flags); +#if defined(CONFIG_ARM_FCSE_DYNPID) + /* pid == -1 means find a free pid. */ + if (fcse_pid == -1) { + fcse_pid = find_free_pid(fcse_pids_bits); + if (fcse_pid == NR_PIDS) { + fcse_pid = find_free_pid(fcse_pids_cache_dirty); + if (unlikely(fcse_pid == NR_PIDS)) { + spin_unlock_irqrestore(&fcse_lock, flags); + return -ENOENT; + } + } + } +#endif /* CONFIG_ARM_FCSE_DYNPID */ + fcse_pid_dereference(mm); + fcse_pid_reference_inner(fcse_pid); + fcse_pids_user[fcse_pid].mm = mm; + __set_bit(fcse_pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); + + from = pgd_offset(mm, 0); + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; + to = pgd_offset(mm, 0); + + memcpy(to, from, len); + memset(from, '\0', len); + barrier(); + clean_dcache_area(from, len); + clean_dcache_area(to, len); + + return fcse_pid; +} + +int fcse_switch_mm_inner(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned fcse_pid = next->context.fcse.pid >> FCSE_PID_SHIFT; + unsigned flush_needed, reused_pid = 0; + unsigned long flags; + + if (unlikely(next == &init_mm)) { + spin_lock_irqsave(&fcse_lock, flags); + goto is_flush_needed; + } + +#ifdef CONFIG_ARM_FCSE_DYNPID + /* + * If the next mm's pid is currently in use, and not by that + * mm, try and find a new, free, pid. + */ + if (unlikely(fcse_pids_user[fcse_pid].mm != next) + && test_bit(fcse_pid, fcse_pids_cache_dirty) + && !rwsem_is_locked(&next->mmap_sem) + && fcse_pids_user[fcse_pid].mm + && !next->context.fcse.large + && !next->core_state) { + int new_fcse_pid = fcse_relocate_mm_to_pid(next, -1); + if (new_fcse_pid >= 0) + fcse_pid = new_fcse_pid; + } +#endif /* CONFIG_ARM_FCSE_DYNPID */ + + spin_lock_irqsave(&fcse_lock, flags); + if (fcse_pids_user[fcse_pid].mm != next) { + if (fcse_pids_user[fcse_pid].mm) + reused_pid = test_bit(fcse_pid, fcse_pids_cache_dirty); + fcse_pids_user[fcse_pid].mm = next; + } + + is_flush_needed: + flush_needed = reused_pid + || next->context.fcse.high_pages + || !prev + || prev->context.fcse.shared_dirty_pages + || prev->context.fcse.high_pages; + + fcse_pid_set(fcse_pid << FCSE_PID_SHIFT); + if (flush_needed) + fcse_clear_dirty_all(); + if (next != &init_mm) + __set_bit(fcse_pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); + + return flush_needed; +} +EXPORT_SYMBOL_GPL(fcse_switch_mm_inner); + +void fcse_pid_reference(unsigned fcse_pid) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_reference_inner(fcse_pid); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +/* Called with mm->mmap_sem write-locked. */ +static noinline void fcse_relocate_mm_to_null_pid(struct mm_struct *mm) +{ + unsigned long seq; + + if (!cache_is_vivt()) + return; + + preempt_disable(); + if (fcse_mm_in_cache(mm)) + do { + preempt_enable(); + + seq = fcse_flush_all_start(mm); + flush_cache_all(); + + preempt_disable(); + } while (!fcse_flush_all_done(mm, seq, 0)); + + fcse_relocate_mm_to_pid(mm, 0); + barrier(); + flush_tlb_mm(mm); + fcse_pid_set(0); + + preempt_enable(); +} +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + +unsigned long +fcse_check_mmap_inner(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned long stack_reserved = + current->signal->rlim[RLIMIT_STACK].rlim_cur; + unsigned long stack_base = PAGE_ALIGN(mm->start_stack) - stack_reserved; + + /* We enfore the RLIMIT_STACK stack size, and here, the return + address would fall in that reserved stack area */ + if ((unsigned long)(addr + len - stack_base) < stack_reserved) { + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED) + && !mm->context.fcse.large) + return TASK_UNMAPPED_BASE; + + /* Forcibly restart from above the stack */ + if (!(fl & MAP_FIXED)) + return PAGE_ALIGN(mm->start_stack); + + /* If MAP_FIXED is set, we encroach upon the reserved + stack area. No choice. */ + } + + /* Address above 32MB */ + if (addr + len > FCSE_TASK_SIZE && !mm->context.fcse.high_pages) { + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED)) + return TASK_UNMAPPED_BASE; + + if (!mm->context.fcse.large) { + /* Ok, the process is going to be larger than 32MB */ +#ifdef CONFIG_ARM_FCSE_MSSAGES + printk(KERN_INFO "FCSE: process %u(%s) VM exceeds 32MB.\n", + current->pid, current->comm); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + mm->context.fcse.large = 1; + } + if (mm->context.fcse.pid) + fcse_relocate_mm_to_null_pid(mm); + } + + return addr; + +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + /* Address above 32MB */ + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED)) + return TASK_UNMAPPED_BASE; + + /* Fail, no 32MB processes in guaranteed mode. */ +#ifdef CONFIG_ARM_FCSE_MESSAGES + printk(KERN_WARNING "FCSE: process %u(%s) VM would exceed the 32MB limit.\n", + current->pid, current->comm); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + return -ENOMEM; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +} + +#ifdef CONFIG_ARM_FCSE_MESSAGES +#define addr_in_vma(vma, addr) \ + ({ \ + struct vm_area_struct *_vma = (vma); \ + ((unsigned long)((addr) - _vma->vm_start) \ + < (unsigned long)((_vma->vm_end - _vma->vm_start))); \ + }) + +#ifdef CONFIG_DEBUG_USER +static noinline void +dump_vmas(struct mm_struct *mm, unsigned long addr, struct pt_regs *regs) +{ + struct vm_area_struct *vma; + char path[128]; + + printk("mappings:\n"); + down_read(&mm->mmap_sem); + for(vma = mm->mmap; vma; vma = vma->vm_next) { + struct file *file = vma->vm_file; + int flags = vma->vm_flags; + const char *name; + + printk("0x%08lx-0x%08lx %c%c%c%c 0x%08llx ", + vma->vm_start, + vma->vm_end, + flags & VM_READ ? 'r' : '-', + flags & VM_WRITE ? 'w' : '-', + flags & VM_EXEC ? 'x' : '-', + flags & VM_MAYSHARE ? 's' : 'p', + ((loff_t)vma->vm_pgoff) << PAGE_SHIFT); + + if (file) + name = d_path(&file->f_path, path, sizeof(path)); + else if ((name = arch_vma_name(vma))) + ; + else if (!vma->vm_mm) + name = "[vdso]"; + else if (vma->vm_start <= mm->start_brk + && vma->vm_end >= mm->brk) + name = "[heap]"; + else if (vma->vm_start <= mm->start_stack && + vma->vm_end >= mm->start_stack) + name = "[stack]"; + else + name = ""; + printk("%s", name); + if (addr_in_vma(vma, regs->ARM_pc)) + printk(" <- PC"); + if (addr_in_vma(vma, regs->ARM_sp)) + printk(" <- SP"); + if (addr_in_vma(vma, addr)) + printk("%s fault", + (addr_in_vma(vma, regs->ARM_pc) + || addr_in_vma(vma, regs->ARM_sp) + ? "," : " <-")); + printk("\n"); + } + up_read(&mm->mmap_sem); +} +#endif /* CONFIG_DEBUG_USER */ + +void fcse_notify_segv(struct mm_struct *mm, + unsigned long addr, struct pt_regs *regs) +{ +#if defined(CONFIG_DEBUG_USER) + if (user_debug & UDBG_SEGV) + dump_vmas(mm, addr, regs); +#endif /* CONFIG_DEBUG_USER */ + + down_read(&mm->mmap_sem); + if (find_vma(mm, addr) == find_vma(mm, regs->ARM_sp)) + printk(KERN_INFO "FCSE: process %u(%s) probably overflowed stack at 0x%08lx.\n", + current->pid, current->comm, regs->ARM_pc); + up_read(&mm->mmap_sem); +} +#endif /* CONFIG_ARM_FCSE_MESSAGES */ diff --git a/arch/arm/kernel/ipipe.c b/arch/arm/kernel/ipipe.c new file mode 100644 index 0000000..41b1013 --- /dev/null +++ b/arch/arm/kernel/ipipe.c @@ -0,0 +1,549 @@ +/* -*- linux-c -*- + * linux/arch/arm/kernel/ipipe.c + * + * Copyright (C) 2002-2005 Philippe Gerum. + * Copyright (C) 2004 Wolfgang Grandegger (Adeos/arm port over 2.4). + * Copyright (C) 2005 Heikki Lindholm (PowerPC 970 fixes). + * Copyright (C) 2005 Stelian Pop. + * Copyright (C) 2006-2008 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Architecture-dependent I-PIPE support for ARM. + */ + +#include <linux/kernel.h> +#include <linux/smp.h> +#include <linux/sched.h> +#include <linux/slab.h> +#include <linux/bitops.h> +#include <linux/interrupt.h> +#include <linux/module.h> +#include <linux/errno.h> +#include <linux/kallsyms.h> +#include <linux/kprobes.h> +#include <linux/ipipe_trace.h> +#include <asm/system.h> +#include <asm/atomic.h> +#include <asm/hardirq.h> +#include <asm/io.h> +#include <asm/irq.h> +#include <asm/unistd.h> +#include <asm/mach/irq.h> +#include <asm/mmu_context.h> + +/* Next tick date (timebase value). */ +DEFINE_PER_CPU(struct pt_regs, __ipipe_tick_regs); +DEFINE_PER_CPU(struct mm_struct *,ipipe_active_mm); +EXPORT_PER_CPU_SYMBOL(ipipe_active_mm); +#ifdef __IPIPE_FEATURE_PIC_MUTE +__ipipe_irqbits_t __ipipe_irqbits; +IPIPE_DEFINE_SPINLOCK(__ipipe_irqbits_lock); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +extern struct irq_desc irq_desc[]; +asmlinkage void asm_do_IRQ(unsigned int irq, struct pt_regs *regs); + +#ifdef CONFIG_SMP + +static cpumask_t __ipipe_cpu_sync_map; + +static cpumask_t __ipipe_cpu_lock_map; + +static IPIPE_DEFINE_SPINLOCK(__ipipe_cpu_barrier); + +static atomic_t __ipipe_critical_count = ATOMIC_INIT(0); + +static void (*__ipipe_cpu_sync) (void); + +/* Always called with hw interrupts off. */ + + +void __ipipe_do_critical_sync(unsigned irq) +{ + int cpu = ipipe_processor_id(); + + cpu_set(cpu, __ipipe_cpu_sync_map); + + /* + * Now we are in sync with the lock requestor running on another + * CPU. Enter a spinning wait until he releases the global + * lock. + */ + spin_lock(&__ipipe_cpu_barrier); + + /* Got it. Now get out. */ + + if (__ipipe_cpu_sync) + /* Call the sync routine if any. */ + __ipipe_cpu_sync(); + + spin_unlock(&__ipipe_cpu_barrier); + + cpu_clear(cpu, __ipipe_cpu_sync_map); +} + +#endif /* CONFIG_SMP */ + +/* + * ipipe_trigger_irq() -- Push the interrupt at front of the pipeline + * just like if it has been actually received from a hw source. Also + * works for virtual interrupts. + */ +int ipipe_trigger_irq(unsigned irq) +{ + unsigned long flags; + +/* FIXME */ +#ifdef CONFIG_IPIPE_DEBUG + if (irq >= IPIPE_NR_IRQS || + (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))) + return -EINVAL; +#else + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + if (ipipe_virtual_irq_p(irq)) { + if (!test_bit(irq - IPIPE_VIRQ_BASE, + &__ipipe_virtual_irq_map)) + return -EINVAL; + } else if (irq_to_desc(irq) == NULL) + return -EINVAL; +#endif + + local_irq_save_hw(flags); + __ipipe_handle_irq(irq, NULL); + local_irq_restore_hw(flags); + + return 1; +} + +int ipipe_get_sysinfo(struct ipipe_sysinfo *info) +{ + info->ncpus = num_online_cpus(); + info->cpufreq = ipipe_cpu_freq(); + info->archdep.tmirq = __ipipe_mach_timerint; + info->archdep.tmfreq = info->cpufreq; + __ipipe_mach_get_tscinfo(&info->archdep.tsc); + + return 0; +} + +static void __ipipe_ack_irq(unsigned irq, struct irq_desc *desc) +{ + desc->ipipe_ack(irq, desc); +} + +static void __ipipe_ack_timerirq(unsigned irq, struct irq_desc *desc) +{ + desc->ipipe_ack(irq, desc); + __ipipe_mach_acktimer(); + desc->ipipe_end(irq, desc); +} + +void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq) +{ + unsigned long flags; + irq_desc[irq].status &= ~IRQ_DISABLED; +#ifdef __IPIPE_FEATURE_PIC_MUTE + if (ipd == &ipipe_root) + return; + + spin_lock_irqsave(&__ipipe_irqbits_lock, flags); + __ipipe_irqbits[irq / BITS_PER_LONG] &= ~(1 << (irq % BITS_PER_LONG)); + spin_unlock_irqrestore(&__ipipe_irqbits_lock, flags); +#else + (void) flags; +#endif /* __IPIPE_FEATURE_PIC_MUTE */ +} +EXPORT_SYMBOL(__ipipe_enable_irqdesc); + +#ifdef __IPIPE_FEATURE_PIC_MUTE +void __ipipe_disable_irqdesc(struct ipipe_domain *ipd, unsigned irq) +{ + unsigned long flags; + + if (ipd == &ipipe_root) + return; + + spin_lock_irqsave(&__ipipe_irqbits_lock, flags); + __ipipe_irqbits[irq / BITS_PER_LONG] |= 1 << (irq % BITS_PER_LONG); + spin_unlock_irqrestore(&__ipipe_irqbits_lock, flags); +} +EXPORT_SYMBOL(__ipipe_disable_irqdesc); +EXPORT_SYMBOL(ipipe_mute_pic); +EXPORT_SYMBOL(ipipe_unmute_pic); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +/* + * __ipipe_enable_pipeline() -- We are running on the boot CPU, hw + * interrupts are off, and secondary CPUs are still lost in space. + */ +void __ipipe_enable_pipeline(void) +{ + unsigned long flags; + unsigned irq; + +/* We do not want "wfi" to be called in arm926ejs based processor, as + this causes the I-cache to be disabled when idle. */ +#ifdef CONFIG_CPU_ARM926T + disable_hlt(); +#endif + + flags = ipipe_critical_enter(NULL); + + /* First, virtualize all interrupts from the root domain. */ + + for (irq = 0; irq < NR_IRQS; irq++) + ipipe_virtualize_irq(ipipe_root_domain, + irq, + (ipipe_irq_handler_t)&asm_do_IRQ, NULL, + ((irq == __ipipe_mach_timerint) + ? &__ipipe_ack_timerirq + : &__ipipe_ack_irq), + IPIPE_HANDLE_MASK | IPIPE_PASS_MASK); + + ipipe_critical_exit(flags); +} + +/* + * ipipe_critical_enter() -- Grab the superlock excluding all CPUs + * but the current one from a critical section. This lock is used when + * we must enforce a global critical section for a single CPU in a + * possibly SMP system whichever context the CPUs are running. + */ +unsigned long ipipe_critical_enter(void (*syncfn) (void)) +{ + unsigned long flags; + + local_irq_save_hw(flags); + +#ifdef CONFIG_SMP + if (likely(num_online_cpus() > 1)) { /* We might be running a SMP-kernel on a UP box... */ + int cpu = ipipe_processor_id(); + cpumask_t lock_map; + + if (!cpu_test_and_set(cpu, __ipipe_cpu_lock_map)) { + while (cpu_test_and_set(BITS_PER_LONG - 1, + __ipipe_cpu_lock_map)) { + int n = 0; + do { + cpu_relax(); + } while (++n < cpu); + } + + spin_lock(&__ipipe_cpu_barrier); + + __ipipe_cpu_sync = syncfn; + + /* Send the sync IPI to all processors but the current one. */ + send_IPI_allbutself(IPIPE_CRITICAL_VECTOR); + + cpus_andnot(lock_map, cpu_online_map, + __ipipe_cpu_lock_map); + + while (!cpus_equal(__ipipe_cpu_sync_map, lock_map)) + cpu_relax(); + } + + atomic_inc(&__ipipe_critical_count); + } +#endif /* CONFIG_SMP */ + + return flags; +} + +/* ipipe_critical_exit() -- Release the superlock. */ + +void ipipe_critical_exit(unsigned long flags) +{ +#ifdef CONFIG_SMP + if (likely(num_online_cpus() > 1)) { /* We might be running a SMP-kernel on a UP box... */ + if (atomic_dec_and_test(&__ipipe_critical_count)) { + spin_unlock(&__ipipe_cpu_barrier); + + while (!cpus_empty(__ipipe_cpu_sync_map)) + cpu_relax(); + + cpu_clear(ipipe_processor_id(), __ipipe_cpu_lock_map); + cpu_clear(BITS_PER_LONG - 1, __ipipe_cpu_lock_map); + } + } +#endif /* CONFIG_SMP */ + + local_irq_restore_hw(flags); +} + +#ifdef CONFIG_PREEMPT + +asmlinkage void __sched __ipipe_preempt_schedule_irq(void) +{ + extern asmlinkage void __sched preempt_schedule_irq(void); + struct ipipe_percpu_domain_data *p; + unsigned long flags; + /* + * preempt_schedule_irq expects the root stage to be stalled. + */ +#ifdef CONFIG_IPIPE_DEBUG + BUG_ON(!irqs_disabled_hw()); +#endif + local_irq_save(flags); + local_irq_enable_hw(); + preempt_schedule_irq(); /* Ok, may reschedule now. */ + local_irq_disable_hw(); + + /* + * Flush any pending interrupt that may have been logged after + * preempt_schedule_irq() stalled the root stage before + * returning to us, and now. + */ + p = ipipe_root_cpudom_ptr(); + if (unlikely(__ipipe_ipending_p(p))) { + add_preempt_count(PREEMPT_ACTIVE); + clear_bit(IPIPE_STALL_FLAG, &p->status); + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + sub_preempt_count(PREEMPT_ACTIVE); + } + + __local_irq_restore_nosync(flags); +} + +#endif /* CONFIG_PREEMPT */ + +asmlinkage int __ipipe_check_root(void) +{ + return ipipe_root_domain_p; +} + +asmlinkage int __ipipe_check_root_interruptible(void) +{ + return ipipe_root_domain_p && !__ipipe_test_root(); +} + +__kprobes int +__ipipe_switch_to_notifier_call_chain(struct atomic_notifier_head *nh, + unsigned long val, void *v) +{ + unsigned long flags; + int rc; + + local_irq_save(flags); + rc = atomic_notifier_call_chain(nh, val, v); + __local_irq_restore_nosync(flags); + + return rc; +} + +asmlinkage int __ipipe_syscall_root(unsigned long scno, struct pt_regs *regs) +{ + struct ipipe_percpu_domain_data *p; + unsigned long flags, origr7; + + /* We use r7 to pass the syscall number to the other domains */ + origr7 = regs->ARM_r7; + regs->ARM_r7 = __NR_SYSCALL_BASE + scno; + /* + * This routine either returns: + * 0 -- if the syscall is to be passed to Linux; + * >0 -- if the syscall should not be passed to Linux, and no + * tail work should be performed; + * <0 -- if the syscall should not be passed to Linux but the + * tail work has to be performed (for handling signals etc). + */ + + WARN_ON_ONCE(irqs_disabled_hw()); + + if (!__ipipe_syscall_watched_p(current, regs->ARM_r7) || + !__ipipe_event_monitored_p(IPIPE_EVENT_SYSCALL)) + goto done; + + if (__ipipe_dispatch_event(IPIPE_EVENT_SYSCALL,regs) > 0){ + if (ipipe_root_domain_p && !in_atomic()) { + /* + * Sync pending VIRQs before _TIF_NEED_RESCHED + * is tested. + */ + local_irq_save_hw(flags); + p = ipipe_root_cpudom_ptr(); + if (__ipipe_ipending_p(p)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOVIRT); + local_irq_restore_hw(flags); + regs->ARM_r7 = origr7; + return -1; + } + regs->ARM_r7 = origr7; + return 1; + } +done: + regs->ARM_r7 = origr7; + return 0; +} + +/* + * __ipipe_handle_irq() -- IPIPE's generic IRQ handler. An optimistic + * interrupt protection log is maintained here for each domain. Hw + * interrupts are off on entry. + */ +int __ipipe_handle_irq(int irq, struct pt_regs *regs) +{ + struct ipipe_domain *this_domain, *next_domain; + struct list_head *head, *pos; + int m_ack; + + m_ack = (regs == NULL); + +#ifdef CONFIG_IPIPE_DEBUG + if (irq >= IPIPE_NR_IRQS) { + printk(KERN_ERR "I-pipe: spurious interrupt %d\n", irq); + goto finalize_nosync; + } +#endif + + this_domain = ipipe_current_domain; + + if (test_bit(IPIPE_STICKY_FLAG, &this_domain->irqs[irq].control)) + head = &this_domain->p_link; + else { + head = __ipipe_pipeline.next; + next_domain = list_entry(head, struct ipipe_domain, p_link); + if (likely(test_bit(IPIPE_WIRED_FLAG, &next_domain->irqs[irq].control))) { + if (!m_ack && next_domain->irqs[irq].acknowledge != NULL) + next_domain->irqs[irq].acknowledge(irq, irq_desc + irq); + __ipipe_dispatch_wired(next_domain, irq); + goto finalize_nosync; + } + } + + /* Ack the interrupt. */ + + pos = head; + + while (pos != &__ipipe_pipeline) { + next_domain = list_entry(pos, struct ipipe_domain, p_link); + prefetch(next_domain); + + /* + * For each domain handling the incoming IRQ, mark it + * as pending in its log. + */ + if (test_bit(IPIPE_HANDLE_FLAG, &next_domain->irqs[irq].control)) { + /* + * Domains that handle this IRQ are polled for + * acknowledging it by decreasing priority + * order. The interrupt must be made pending + * _first_ in the domain's status flags before + * the PIC is unlocked. + */ + __ipipe_set_irq_pending(next_domain, irq); + + if (!m_ack && next_domain->irqs[irq].acknowledge) { + next_domain->irqs[irq].acknowledge(irq, irq_desc + irq); + m_ack = 1; + } + } + + /* + * If the domain does not want the IRQ to be passed + * down the interrupt pipe, exit the loop now. + */ + + if (!test_bit(IPIPE_PASS_FLAG, &next_domain->irqs[irq].control)) + break; + + pos = next_domain->p_link.next; + } + + /* + * If the interrupt preempted the head domain, then do not + * even try to walk the pipeline, unless an interrupt is + * pending for it. + */ + if (test_bit(IPIPE_AHEAD_FLAG, &this_domain->flags) && + ipipe_head_cpudom_var(irqpend_himap) == 0) + goto finalize_nosync; + + /* + * Now walk the pipeline, yielding control to the highest + * priority domain that has pending interrupt(s) or + * immediately to the current domain if the interrupt has been + * marked as 'sticky'. This search does not go beyond the + * current domain in the pipeline. + */ + + __ipipe_walk_pipeline(head); + +finalize_nosync: + if (!ipipe_root_domain_p || __ipipe_test_root()) + return 0; + +#ifdef CONFIG_SMP + /* + * Prevent a spurious rescheduling from being triggered on + * preemptible kernels along the way out through + * ret_from_intr. + */ + if (!regs) + __set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); +#endif /* CONFIG_SMP */ + + return 1; +} + +asmlinkage int __ipipe_grab_irq(int irq, struct pt_regs *regs) +{ + int status; + +#ifdef irq_finish + /* AT91 specific workaround */ + irq_finish(irq); +#endif /* irq_finish */ + + if (irq == __ipipe_mach_timerint) { + /* + * Given our deferred dispatching model for regular IRQs, we + * only record CPU regs for the last timer interrupt, so that + * the timer handler charges CPU times properly. It is assumed + * that other interrupt handlers don't actually care for such + * information. + */ + __raw_get_cpu_var(__ipipe_tick_regs).ARM_cpsr = + (ipipe_root_domain_p + ? regs->ARM_cpsr + : regs->ARM_cpsr | PSR_I_BIT); + __raw_get_cpu_var(__ipipe_tick_regs).ARM_pc = regs->ARM_pc; + } + + ipipe_trace_irq_entry(irq); + status = __ipipe_handle_irq(irq, regs); + ipipe_trace_irq_exit(irq); + + return status; +} + +EXPORT_SYMBOL_GPL(show_stack); +EXPORT_SYMBOL_GPL(init_mm); +#ifndef MULTI_CPU +EXPORT_SYMBOL_GPL(cpu_do_switch_mm); +#endif +EXPORT_SYMBOL_GPL(__check_kvm_seq); +#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) +EXPORT_SYMBOL_GPL(tasklist_lock); +#endif /* CONFIG_SMP || CONFIG_DEBUG_SPINLOCK */ + +#ifdef CONFIG_CPU_HAS_ASID +EXPORT_SYMBOL_GPL(__new_context); +EXPORT_SYMBOL_GPL(cpu_last_asid); +#endif /* CONFIG_CPU_HAS_ASID */ diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index b7cb45b..b759267 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -121,8 +121,10 @@ asmlinkage void __exception asm_do_IRQ(unsigned int irq, struct pt_regs *regs) generic_handle_irq(irq); } +#ifndef CONFIG_IPIPE /* AT91 specific workaround */ irq_finish(irq); +#endif /* !CONFIG_IPIPE */ irq_exit(); set_irq_regs(old_regs); diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index ba2adef..f612624 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -127,8 +127,15 @@ EXPORT_SYMBOL_GPL(arm_pm_restart); */ static void default_idle(void) { - if (!need_resched()) + if (!need_resched()) { +#ifdef CONFIG_IPIPE + __ipipe_unstall_root(); +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF + ipipe_trace_end(0x8000000E); +#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */ +#endif /* CONFIG_IPIPE */ arch_idle(); + } local_irq_enable(); } @@ -148,6 +155,7 @@ void cpu_idle(void) /* endless idle loop with no priority at all */ while (1) { tick_nohz_stop_sched_tick(1); + ipipe_suspend_domain(); leds_event(led_idle_start); while (!need_resched()) { #ifdef CONFIG_HOTPLUG_CPU diff --git a/arch/arm/kernel/ptrace.c b/arch/arm/kernel/ptrace.c index a2ea385..487ab4c 100644 --- a/arch/arm/kernel/ptrace.c +++ b/arch/arm/kernel/ptrace.c @@ -479,6 +479,10 @@ void ptrace_break(struct task_struct *tsk, struct pt_regs *regs) static int break_trap(struct pt_regs *regs, unsigned int instr) { + + if (ipipe_trap_notify(IPIPE_TRAP_BREAK,regs)) + return 0; + ptrace_break(current, regs); return 0; } diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 57162af..b891257 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -37,6 +37,7 @@ #include <asm/ptrace.h> #include <asm/localtimer.h> #include <asm/smp_plat.h> +#include <asm/fcse.h> /* * as from 2.5, kernels no longer have an init_tasks structure @@ -638,7 +639,7 @@ void flush_tlb_all(void) void flush_tlb_mm(struct mm_struct *mm) { if (tlb_ops_need_broadcast()) - on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, mm_cpumask(mm)); + on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, fcse_tlb_mask(mm)); else local_flush_tlb_mm(mm); } @@ -649,7 +650,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) struct tlb_args ta; ta.ta_vma = vma; ta.ta_start = uaddr; - on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_page(vma, uaddr); } @@ -672,7 +673,7 @@ void flush_tlb_range(struct vm_area_struct *vma, ta.ta_vma = vma; ta.ta_start = start; ta.ta_end = end; - on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_range(vma, start, end); } diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 3f361a7..efe92a5 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -331,6 +331,11 @@ asmlinkage void __exception do_undefinstr(struct pt_regs *regs) siginfo_t info; void __user *pc; + /* In kernel mode, the trap was already signaled at the beginning of + * __und_svc in arch/arm/kernel/entry-armv.S */ + if (user_mode(regs) && ipipe_trap_notify(IPIPE_TRAP_UNDEFINSTR, regs)) + return; + /* * According to the ARM ARM, PC is 2 or 4 bytes ahead, * depending whether we're in Thumb mode or not. @@ -728,13 +733,22 @@ void __init trap_init(void) return; } +#ifdef CONFIG_IPIPE +void *__ipipe_tsc_area; +#endif /* CONFIG_IPIPE */ + void __init early_trap_init(void) { unsigned long vectors = CONFIG_VECTORS_BASE; extern char __stubs_start[], __stubs_end[]; extern char __vectors_start[], __vectors_end[]; +#ifndef CONFIG_IPIPE extern char __kuser_helper_start[], __kuser_helper_end[]; int kuser_sz = __kuser_helper_end - __kuser_helper_start; +#else /* !CONFIG_IPIPE */ + extern char __ipipe_tsc_area_start[], __kuser_helper_end[]; + int kuser_sz = __kuser_helper_end - __ipipe_tsc_area_start; +#endif /* !CONFIG_IPIPE */ /* * Copy the vectors, stubs and kuser helpers (in entry-armv.S) @@ -743,7 +757,13 @@ void __init early_trap_init(void) */ memcpy((void *)vectors, __vectors_start, __vectors_end - __vectors_start); memcpy((void *)vectors + 0x200, __stubs_start, __stubs_end - __stubs_start); +#ifndef CONFIG_IPIPE memcpy((void *)vectors + 0x1000 - kuser_sz, __kuser_helper_start, kuser_sz); +#else /* !CONFIG_IPIPE */ + BUG_ON(0x1000 - kuser_sz < 0x200 + __stubs_end - __stubs_start); + memcpy((void *)vectors + 0x1000 - kuser_sz, __ipipe_tsc_area_start, kuser_sz); + __ipipe_tsc_area = (void *)vectors + 0x1000 - kuser_sz; +#endif /* !CONFIG_IPIPE */ /* * Copy signal return handlers into the vector page, and diff --git a/arch/arm/mach-at91/Kconfig b/arch/arm/mach-at91/Kconfig index 0b2ee95..2308048 100644 --- a/arch/arm/mach-at91/Kconfig +++ b/arch/arm/mach-at91/Kconfig @@ -406,6 +406,19 @@ endif # ---------------------------------------------------------- +comment "Adeos I-pipe Options" + +config IPIPE_AT91_TC + depends on IPIPE + int "AT91 TC used as time base by Adeos I-pipe" + default 0 + help + When Adeos interrupt pipeline is enabled, TC0 is used by default + as time base, but you can use TC1 or TC2 by setting this variable to 1 + or 2. This should only be needed to avoid conflicts with other drivers. + +# ---------------------------------------------------------- + comment "AT91 Board Options" config MTD_AT91_DATAFLASH_CARD diff --git a/arch/arm/mach-at91/Makefile b/arch/arm/mach-at91/Makefile index 709fbad..09cfdd0 100644 --- a/arch/arm/mach-at91/Makefile +++ b/arch/arm/mach-at91/Makefile @@ -83,3 +83,14 @@ obj-$(CONFIG_CPU_IDLE) += cpuidle.o ifeq ($(CONFIG_PM_DEBUG),y) CFLAGS_pm.o += -DDEBUG endif + +ifeq ($(CONFIG_IPIPE),y) +obj-y := $(filter-out at91rm9200_time.o at91sam926x_time.o at91x40_time.o, $(obj-y)) +obj-$(CONFIG_ARCH_AT91RM9200) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9260) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9261) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9263) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9RL) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9G20) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91X40) += at91_ipipe_time.o +endif diff --git a/arch/arm/mach-at91/at91_ipipe_time.c b/arch/arm/mach-at91/at91_ipipe_time.c new file mode 100644 index 0000000..a31515c --- /dev/null +++ b/arch/arm/mach-at91/at91_ipipe_time.c @@ -0,0 +1,419 @@ +/* + * linux/arch/arm/mach-at91/at91_ipipe_time.c + * + * Copyright (C) 2007 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> + * + * Adaptation to AT91SAM926x: + * Copyright (C) 2007 Gregory CLEMENT, Adeneo + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include <linux/interrupt.h> +#include <linux/irq.h> +#include <linux/kernel.h> +#include <linux/clockchips.h> +#include <linux/clk.h> +#include <linux/stringify.h> +#include <linux/err.h> +#include <linux/console.h> +#include <linux/module.h> +#include <linux/ipipe.h> + +#include <asm/io.h> +#include <asm/mach/time.h> + +#include <mach/hardware.h> +#include <mach/at91_st.h> +#include <mach/at91_tc.h> +#include <mach/at91_pit.h> +#include "clock.h" + +#if defined(CONFIG_ARCH_AT91RM9200) +#define AT91_ID_TC0 AT91RM9200_ID_TC0 +#define AT91_ID_TC1 AT91RM9200_ID_TC1 +#define AT91_ID_TC2 AT91RM9200_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9G20) +#define AT91_ID_TC0 AT91SAM9260_ID_TC0 +#define AT91_ID_TC1 AT91SAM9260_ID_TC1 +#define AT91_ID_TC2 AT91SAM9260_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9261) +#define AT91_ID_TC0 AT91SAM9261_ID_TC0 +#define AT91_ID_TC1 AT91SAM9261_ID_TC1 +#define AT91_ID_TC2 AT91SAM9261_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9263) +#define AT91_ID_TC0 AT91SAM9263_ID_TCB +#define AT91_ID_TC1 AT91SAM9263_ID_TCB +#define AT91_ID_TC2 AT91SAM9263_ID_TCB +#elif defined(CONFIG_ARCH_AT91SAM9RL) +#define AT91_ID_TC0 AT91SAM9RL_ID_TC0 +#define AT91_ID_TC1 AT91SAM9RL_ID_TC1 +#define AT91_ID_TC2 AT91SAM9RL_ID_TC2 +#elif defined(CONFIG_ARCH_AT91X40) +#define AT91_ID_TC0 AT91X40_ID_TC0 +#define AT91_ID_TC1 AT91X40_ID_TC1 +#define AT91_ID_TC2 AT91X40_ID_TC2 +#else +#error "AT91 processor unsupported by Adeos" +#endif + +#if (CONFIG_IPIPE_AT91_TC==0) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC0 +#elif (CONFIG_IPIPE_AT91_TC==1) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC1 +#elif (CONFIG_IPIPE_AT91_TC==2) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC2 +#else +#error IPIPE_AT91_TC must be 0, 1 or 2. +#endif + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ + +#define TCNXCNS(timer,v) ((v) << ((timer)<<1)) +#define AT91_TC_REG_MASK (0xffff) + +static unsigned long next_match; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + union { + struct { + unsigned short mid; + unsigned short low; + }; + unsigned long mid_low; + }; + }; +#else /* __LITTLE_ENDIAN */ + struct { + union { + struct { + unsigned short low; + unsigned short mid; + }; + unsigned long mid_low; + }; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; +static unsigned max_delta_ticks, min_delta_ticks; +static struct clock_event_device clkevt; +static int tc_timer_clock; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = + (unsigned *) + (AT91_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC + AT91_TC_CV); + info->u.fr.mask = AT91_TC_REG_MASK; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static inline unsigned int at91_tc_read(unsigned int reg_offset) +{ + unsigned long addr = + (AT91_VA_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC); + + return readl((void __iomem *)(addr + reg_offset)); +} + +static inline void at91_tc_write(unsigned int reg_offset, unsigned long value) +{ + unsigned long addr = + (AT91_VA_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC); + + writel(value, (void __iomem *)(addr + reg_offset)); +} + +#define read_CV() at91_tc_read(AT91_TC_CV) +#define read_RC() at91_tc_read(AT91_TC_RC) +#define write_RC(value) at91_tc_write(AT91_TC_RC, value) + +int __ipipe_mach_timerint = KERNEL_TIMER_IRQ_NUM; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static void ipipe_mach_update_tsc(unsigned short stamp); + +static int at91_timer_initialized; + +/* + * IRQ handler for the timer. + */ +static irqreturn_t at91_timer_interrupt(int irq, void *dev_id) +{ + clkevt.event_handler(&clkevt); + return IRQ_HANDLED; +} + +static struct irqaction at91_timer_irq = { + .name = "at91_tick", + .flags = IRQF_DISABLED | IRQF_TIMER, + .handler = &at91_timer_interrupt +}; + +static void ipipe_mach_update_tsc(unsigned short stamp) +{ + union tsc_reg *local_tsc; + + local_tsc = &tsc[ipipe_processor_id()]; + if (unlikely(stamp < local_tsc->low)) + local_tsc->full += AT91_TC_REG_MASK + 1; + local_tsc->low = stamp; +} + +void __ipipe_mach_acktimer(void) +{ + at91_tc_read(AT91_TC_SR); + + if (unlikely(!__ipipe_mach_timerstolen)) { + ipipe_mach_update_tsc(read_CV()); + next_match = (next_match + __ipipe_mach_ticks_per_jiffy) + & AT91_TC_REG_MASK; + write_RC(next_match); + } +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(at91_timer_initialized)) { + union tsc_reg *local_tsc, before, after; + unsigned short stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(after.full): "r"(local_tsc), "m"(*local_tsc)); + do { + before = after; + stamp = read_CV(); + barrier(); + __asm__ ("ldmia %1, %M0\n": + "=r"(after.full): + "r"(local_tsc), "m"(*local_tsc)); + } while (after.mid_low != before.mid_low); + if (unlikely(stamp < before.low)) + before.full += AT91_TC_REG_MASK + 1; + before.low = stamp; + return before.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +cycle_t at91_ipipe_get_clocksource(struct clocksource *cs) +{ + return (cycle_t)__ipipe_mach_get_tsc(); +} + +static struct clocksource clksrc = { + .name = "at91_tc" __stringify(CONFIG_IPIPE_AT91_TC), + .rating = 250, + .read = at91_ipipe_get_clocksource, + .mask = CLOCKSOURCE_MASK(64), + .shift = 20, + .flags = CLOCK_SOURCE_IS_CONTINUOUS, +}; + +static void +at91_tc_set_mode(enum clock_event_mode mode, struct clock_event_device *dev) +{ + /* Disable the channel */ + at91_tc_write(AT91_TC_CCR, AT91_TC_CLKDIS); + + /* Disable all interrupts. */ + at91_tc_write(AT91_TC_IDR, ~0ul); + + if (mode == CLOCK_EVT_MODE_PERIODIC) { + unsigned long v; + +#ifndef CONFIG_ARCH_AT91SAM9263 + clk_enable(clk_get(NULL, "tc" + __stringify(CONFIG_IPIPE_AT91_TC) "_clk")); +#else /* AT91SAM9263 */ + clk_enable(clk_get(NULL, "tcb_clk")); +#endif /* AT91SAM9263 */ + + /* No Sync. */ + at91_tc_write(AT91_TC_BCR, 0); + + /* program NO signal on XCN */ + v = readl((void __iomem *) (AT91_VA_BASE_TCB0 + AT91_TC_BMR)); + v &= ~TCNXCNS(CONFIG_IPIPE_AT91_TC, 3); + v |= TCNXCNS(CONFIG_IPIPE_AT91_TC, 1); /* AT91_TC_TCNXCNS_NONE */ + writel(v, (void __iomem *) (AT91_VA_BASE_TCB0 + AT91_TC_BMR)); + + /* Use the clock selected by at91_timer_init as input clock. */ + at91_tc_write(AT91_TC_CMR, tc_timer_clock); + + /* Load the TC register C. */ + next_match = __ipipe_mach_ticks_per_jiffy; + write_RC(next_match); + + /* Enable CPCS interrupt. */ + at91_tc_write(AT91_TC_IER, AT91_TC_CPCS); + + /* Enable the channel. */ + at91_tc_write(AT91_TC_CCR, AT91_TC_CLKEN | AT91_TC_SWTRG); + + at91_timer_initialized = 1; + } +} + +/* + * Reprogram the timer + */ +void __ipipe_mach_set_dec(unsigned long delay) +{ + unsigned short stamp; + unsigned long flags; + + if (delay > max_delta_ticks) + delay = max_delta_ticks; + + if (likely(delay > min_delta_ticks)) { + local_irq_save_hw(flags); + stamp = read_CV(); + ipipe_mach_update_tsc(stamp); + write_RC((stamp + delay) & AT91_TC_REG_MASK); + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(KERNEL_TIMER_IRQ_NUM); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clkevt.name); +} + +static struct clock_event_device clkevt = { + .name = "at91_tc" __stringify(CONFIG_IPIPE_AT91_TC), + .features = CLOCK_EVT_FEAT_PERIODIC, + .shift = 32, + .rating = 250, + .set_mode = at91_tc_set_mode, +}; + +void __ipipe_mach_release_timer(void) +{ + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return (read_RC() - read_CV()) & AT91_TC_REG_MASK; +} + +void __init at91_timer_init(void) +{ + unsigned char tc_divisors[] = { 2, 8, 32, 128, 0, }; + unsigned master_freq, divisor = 0, divided_freq = 0; + unsigned long long wrap_ns; + + /* Disable (boot loader) timer interrupts. */ +#if defined(CONFIG_ARCH_AT91RM9200) + at91_sys_write(AT91_ST_IDR, AT91_ST_PITS | AT91_ST_WDOVF | AT91_ST_RTTINC | AT91_ST_ALMS); + (void) at91_sys_read(AT91_ST_SR); /* Clear any pending interrupts */ +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9261) \ + || defined(CONFIG_ARCH_AT91SAM9263) || defined(CONFIG_ARCH_AT91SAM9RL) \ + || defined(CONFIG_ARCH_AT91SAM9G20) + at91_sys_write(AT91_PIT_MR, 0); + + /* Clear any pending interrupts */ + (void) at91_sys_read(AT91_PIT_PIVR); +#endif /* CONFIG_ARCH_AT91SAM926x */ + + master_freq = clk_get_rate(clk_get(NULL, "mck")); + /* Find the first frequency above 1 MHz */ + for (tc_timer_clock = ARRAY_SIZE(tc_divisors) - 1; + tc_timer_clock >= 0; tc_timer_clock--) { + divisor = tc_divisors[tc_timer_clock]; + divided_freq = (divisor + ? master_freq / divisor : AT91_SLOW_CLOCK); + if (divided_freq > 1000000) + break; + } + + wrap_ns = (unsigned long long) (AT91_TC_REG_MASK + 1) * NSEC_PER_SEC; + do_div(wrap_ns, divided_freq); + + if (divided_freq < 1000000) + printk(KERN_INFO "AT91 I-pipe warning: could not find a" + " frequency greater than 1MHz\n"); + + printk(KERN_INFO "AT91 I-pipe timer: div: %u, freq: %u.%06u MHz, wrap: " + "%u.%06u ms\n", divisor, + divided_freq / 1000000, divided_freq % 1000000, + (unsigned) wrap_ns / 1000000, (unsigned) wrap_ns % 1000000); + + /* Add a 1ms margin. It means that when an interrupt occurs, update_tsc + must be called within 1ms. update_tsc is called by acktimer when no + higher domain handles the timer, and called through set_dec when a + higher domain handles the timer. */ + wrap_ns -= 1000000; + /* Set up the interrupt. */ + setup_irq(KERNEL_TIMER_IRQ_NUM, &at91_timer_irq); + +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; +#endif /* CONFIG_SMP */ + + clkevt.mult = div_sc(divided_freq, NSEC_PER_SEC, clkevt.shift); + clkevt.max_delta_ns = wrap_ns; + clkevt.min_delta_ns = 2000; + clkevt.cpumask = cpumask_of(0); + clockevents_register_device(&clkevt); + + clksrc.mult = clocksource_hz2mult(divided_freq, clksrc.shift); + clocksource_register(&clksrc); + + __ipipe_mach_ticks_per_jiffy = (divided_freq + HZ/2) / HZ; + max_delta_ticks = (wrap_ns * clkevt.mult) >> clkevt.shift; + min_delta_ticks = ((unsigned long long) clkevt.min_delta_ns + * clkevt.mult) >> clkevt.shift; +} + +#ifdef CONFIG_ARCH_AT91RM9200 +struct sys_timer at91rm9200_timer = { +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9261) \ + || defined(CONFIG_ARCH_AT91SAM9263) || defined(CONFIG_ARCH_AT91SAM9RL) \ + || defined(CONFIG_ARCH_AT91SAM9G20) +struct sys_timer at91sam926x_timer = { +#elif defined(CONFIG_ARCH_AT91X40) +struct sys_timer at91x40_timer = { +#else +#error "Unknown machine" +#endif + .init = at91_timer_init, + .suspend = NULL, + .resume = NULL, +}; diff --git a/arch/arm/mach-at91/at91rm9200.c b/arch/arm/mach-at91/at91rm9200.c index 2e9ecad..2dea6d7 100644 --- a/arch/arm/mach-at91/at91rm9200.c +++ b/arch/arm/mach-at91/at91rm9200.c @@ -33,7 +33,14 @@ static struct map_desc at91rm9200_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91RM9200_BASE_EMAC), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ + }, { .virtual = AT91_IO_VIRT_BASE - AT91RM9200_SRAM_SIZE, .pfn = __phys_to_pfn(AT91RM9200_SRAM_BASE), .length = AT91RM9200_SRAM_SIZE, @@ -300,6 +307,7 @@ void __init at91rm9200_initialize(unsigned long main_clock, unsigned short banks * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91rm9200_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller (FIQ) */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -332,6 +340,42 @@ static unsigned int at91rm9200_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller (IRQ4) */ 0, /* Advanced Interrupt Controller (IRQ5) */ 0 /* Advanced Interrupt Controller (IRQ6) */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 6, /* System Peripheral */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, /* Parallel IO Controller D */ + 5, /* USART 0 */ + 5, /* USART 1 */ + 5, /* USART 2 */ + 5, /* USART 3 */ + 0, /* Multimedia Card Interface */ + 3, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 5, /* Serial Peripheral Interface */ + 4, /* Serial Synchronous Controller */ + 4, /* Serial Synchronous Controller */ + 4, /* Serial Synchronous Controller */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 0, /* Timer Counter 3 */ + 0, /* Timer Counter 4 */ + 0, /* Timer Counter 5 */ + 2, /* USB Host port */ + 2, /* Ethernet MAC */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0 /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91rm9200_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9260.c b/arch/arm/mach-at91/at91sam9260.c index 0894f10..63f05ad 100644 --- a/arch/arm/mach-at91/at91sam9260.c +++ b/arch/arm/mach-at91/at91sam9260.c @@ -31,6 +31,13 @@ static struct map_desc at91sam9260_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ } }; @@ -350,6 +357,7 @@ void __init at91sam9260_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9260_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -382,6 +390,42 @@ static unsigned int at91sam9260_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 7, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, /* Analog-to-Digital Converter */ + 6, /* USART 0 */ + 6, /* USART 1 */ + 6, /* USART 2 */ + 0, /* Multimedia Card Interface */ + 4, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 6, /* Serial Peripheral Interface 0 */ + 6, /* Serial Peripheral Interface 1 */ + 5, /* Serial Synchronous Controller */ + 0, + 0, + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 3, /* USB Host port */ + 3, /* Ethernet */ + 0, /* Image Sensor Interface */ + 6, /* USART 3 */ + 6, /* USART 4 */ + 6, /* USART 5 */ + 7, /* Timer Counter 3 */ + 7, /* Timer Counter 4 */ + 7, /* Timer Counter 5 */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9260_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9261.c b/arch/arm/mach-at91/at91sam9261.c index 4ecf379..8a13d5f 100644 --- a/arch/arm/mach-at91/at91sam9261.c +++ b/arch/arm/mach-at91/at91sam9261.c @@ -31,6 +31,13 @@ static struct map_desc at91sam9261_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, }; @@ -306,6 +313,7 @@ void __init at91sam9261_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9261_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -338,6 +346,42 @@ static unsigned int at91sam9261_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 7, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, + 6, /* USART 0 */ + 6, /* USART 1 */ + 6, /* USART 2 */ + 0, /* Multimedia Card Interface */ + 4, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 6, /* Serial Peripheral Interface 0 */ + 6, /* Serial Peripheral Interface 1 */ + 5, /* Serial Synchronous Controller 0 */ + 5, /* Serial Synchronous Controller 1 */ + 5, /* Serial Synchronous Controller 2 */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 3, /* USB Host port */ + 3, /* LCD Controller */ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9261_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9263.c b/arch/arm/mach-at91/at91sam9263.c index 942792d..c043f3a 100644 --- a/arch/arm/mach-at91/at91sam9263.c +++ b/arch/arm/mach-at91/at91sam9263.c @@ -30,6 +30,13 @@ static struct map_desc at91sam9263_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, { .virtual = AT91_IO_VIRT_BASE - AT91SAM9263_SRAM0_SIZE, .pfn = __phys_to_pfn(AT91SAM9263_SRAM0_BASE), @@ -311,6 +318,7 @@ void __init at91sam9263_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9263_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller (FIQ) */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -343,6 +351,42 @@ static unsigned int at91sam9263_default_irq_priority[NR_AIC_IRQS] __initdata = { 2, /* USB Host port */ 0, /* Advanced Interrupt Controller (IRQ0) */ 0, /* Advanced Interrupt Controller (IRQ1) */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller (FIQ) */ + 6, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C, D and E */ + 0, + 0, + 5, /* USART 0 */ + 5, /* USART 1 */ + 5, /* USART 2 */ + 0, /* Multimedia Card Interface 0 */ + 0, /* Multimedia Card Interface 1 */ + 3, /* CAN */ + 0, /* Two-Wire Interface */ + 5, /* Serial Peripheral Interface 0 */ + 5, /* Serial Peripheral Interface 1 */ + 4, /* Serial Synchronous Controller 0 */ + 4, /* Serial Synchronous Controller 1 */ + 5, /* AC97 Controller */ + 7, /* Timer Counter 0, 1 and 2 */ + 0, /* Pulse Width Modulation Controller */ + 2, /* Ethernet */ + 0, + 0, /* 2D Graphic Engine */ + 2, /* USB Device Port */ + 0, /* Image Sensor Interface */ + 2, /* LDC Controller */ + 0, /* DMA Controller */ + 0, + 2, /* USB Host port */ + 0, /* Advanced Interrupt Controller (IRQ0) */ + 0, /* Advanced Interrupt Controller (IRQ1) */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9263_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9rl.c b/arch/arm/mach-at91/at91sam9rl.c index 211c5c1..9cdad31 100644 --- a/arch/arm/mach-at91/at91sam9rl.c +++ b/arch/arm/mach-at91/at91sam9rl.c @@ -30,6 +30,13 @@ static struct map_desc at91sam9rl_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, }; @@ -303,6 +310,7 @@ void __init at91sam9rl_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9rl_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -335,6 +343,42 @@ static unsigned int at91sam9rl_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, 0, 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 6, /* System Peripherals */ + 1, /* Parallel IO Controller A */ + 1, /* Parallel IO Controller B */ + 1, /* Parallel IO Controller C */ + 1, /* Parallel IO Controller D */ + 4, /* USART 0 */ + 4, /* USART 1 */ + 4, /* USART 2 */ + 4, /* USART 3 */ + 0, /* Multimedia Card Interface */ + 5, /* Two-Wire Interface 0 */ + 5, /* Two-Wire Interface 1 */ + 4, /* Serial Peripheral Interface */ + 3, /* Serial Synchronous Controller 0 */ + 3, /* Serial Synchronous Controller 1 */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 0, + 0, /* Touch Screen Controller */ + 0, /* DMA Controller */ + 2, /* USB Device High speed port */ + 2, /* LCD Controller */ + 5, /* AC97 Controller */ + 0, + 0, + 0, + 0, + 0, + 0, + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9rl_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/gpio.c b/arch/arm/mach-at91/gpio.c index ae4772e..9b3c914 100644 --- a/arch/arm/mach-at91/gpio.c +++ b/arch/arm/mach-at91/gpio.c @@ -19,12 +19,20 @@ #include <linux/list.h> #include <linux/module.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <mach/at91_pio.h> #include <mach/gpio.h> #include <asm/gpio.h> +#ifdef CONFIG_IPIPE +#include <asm/irq.h> + +#ifdef __IPIPE_FEATURE_PIC_MUTE +DEFINE_PER_CPU(__ipipe_irqbits_t, __ipipe_muted_irqs); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ +#endif /* CONFIG_IPIPE */ #include "generic.h" @@ -375,6 +383,10 @@ static int gpio_irq_type(unsigned pin, unsigned type) static struct irq_chip gpio_irqchip = { .name = "GPIO", +#ifdef CONFIG_IPIPE + .ack = gpio_irq_mask, + .mask_ack = gpio_irq_mask, +#endif /* CONFIG_IPIPE */ .mask = gpio_irq_mask, .unmask = gpio_irq_unmask, .set_type = gpio_irq_type, @@ -422,7 +434,7 @@ static void gpio_irq_handler(unsigned irq, struct irq_desc *desc) gpio_irq_mask(pin); } else - generic_handle_irq(pin); + ipipe_handle_irq_cond(pin); } pin++; gpio++; @@ -433,6 +445,38 @@ static void gpio_irq_handler(unsigned irq, struct irq_desc *desc) /* now it may re-trigger */ } +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) +void ipipe_mute_pic(void) +{ + unsigned long unmasked, muted; + unsigned i; + + for (i = 0; i < gpio_banks; i++) { + unmasked = __raw_readl(gpio_chip[i].regbase + PIO_IMR); + muted = unmasked & __ipipe_irqbits[i + 1]; + __raw_get_cpu_var(__ipipe_muted_irqs)[i + 1] = muted; + __raw_writel(muted, gpio_chip[i].regbase + PIO_IDR); + } + + unmasked = at91_sys_read(AT91_AIC_IMR); + muted = unmasked & __ipipe_irqbits[0]; + __raw_get_cpu_var(__ipipe_muted_irqs)[0] = muted; + at91_sys_write(AT91_AIC_IDCR, muted); +} + +void ipipe_unmute_pic(void) +{ + unsigned i; + + at91_sys_write(AT91_AIC_IECR, __raw_get_cpu_var(__ipipe_muted_irqs)[0]); + for (i = 0; i < gpio_banks; i++) { + unsigned long muted; + muted = __raw_get_cpu_var(__ipipe_muted_irqs)[i + 1]; + __raw_writel(muted, gpio_chip[i].regbase + PIO_IER); + } +} +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ + /*--------------------------------------------------------------------------*/ #ifdef CONFIG_DEBUG_FS @@ -509,6 +553,10 @@ void __init at91_gpio_irq_setup(void) unsigned pioc, pin; struct at91_gpio_chip *this, *prev; +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) + memset(__ipipe_irqbits, ~0, sizeof(__ipipe_irqbits)); +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ + for (pioc = 0, pin = PIN_BASE, this = gpio_chip, prev = NULL; pioc++ < gpio_banks; prev = this, this++) { @@ -538,6 +586,9 @@ void __init at91_gpio_irq_setup(void) set_irq_chip_data(id, this); set_irq_chained_handler(id, gpio_irq_handler); +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) + __ipipe_irqbits[0] &= ~(1 << id); +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ } pr_info("AT91: %d gpio irqs in %d banks\n", pin - PIN_BASE, gpio_banks); } diff --git a/arch/arm/mach-at91/include/mach/hardware.h b/arch/arm/mach-at91/include/mach/hardware.h index a0df8b0..05ff930 100644 --- a/arch/arm/mach-at91/include/mach/hardware.h +++ b/arch/arm/mach-at91/include/mach/hardware.h @@ -63,6 +63,25 @@ #define AT91_VA_BASE_SYS AT91_IO_P2V(AT91_BASE_SYS) #define AT91_VA_BASE_EMAC AT91_IO_P2V(AT91RM9200_BASE_EMAC) +#ifdef CONFIG_IPIPE +#if defined(CONFIG_ARCH_AT91RM9200) +#define AT91_BASE_TCB0 AT91RM9200_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9G20) +#define AT91_BASE_TCB0 AT91SAM9260_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9261) +#define AT91_BASE_TCB0 AT91SAM9261_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9263) +#define AT91_BASE_TCB0 AT91SAM9263_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9RL) +#define AT91_BASE_TCB0 AT91SAM9RL_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91X40) +#define AT91_BASE_TCB0 (AT91_BASE_SYS + AT91_TC) +#else +#error "AT91 processor unsupported by Adeos" +#endif +#define AT91_VA_BASE_TCB0 AT91_IO_P2V(AT91_BASE_TCB0) +#endif + /* Internal SRAM is mapped below the IO devices */ #define AT91_SRAM_MAX SZ_1M #define AT91_VIRT_BASE (AT91_IO_VIRT_BASE - AT91_SRAM_MAX) diff --git a/arch/arm/mach-at91/include/mach/irqs.h b/arch/arm/mach-at91/include/mach/irqs.h index 36bd55f..cedaab2 100644 --- a/arch/arm/mach-at91/include/mach/irqs.h +++ b/arch/arm/mach-at91/include/mach/irqs.h @@ -45,4 +45,6 @@ /* FIQ is AIC source 0. */ #define FIQ_START AT91_ID_FIQ +#define __IPIPE_FEATURE_PIC_MUTE + #endif diff --git a/arch/arm/mach-at91/include/mach/timex.h b/arch/arm/mach-at91/include/mach/timex.h index 31ac2d9..29a70e2 100644 --- a/arch/arm/mach-at91/include/mach/timex.h +++ b/arch/arm/mach-at91/include/mach/timex.h @@ -82,6 +82,6 @@ #define AT91X40_MASTER_CLOCK 40000000 #define CLOCK_TICK_RATE (AT91X40_MASTER_CLOCK) -#endif +#endif /* arch specific */ #endif diff --git a/arch/arm/mach-at91/irq.c b/arch/arm/mach-at91/irq.c index da3494a..d375427 100644 --- a/arch/arm/mach-at91/irq.c +++ b/arch/arm/mach-at91/irq.c @@ -121,6 +121,9 @@ static struct irq_chip at91_aic_chip = { .name = "AIC", .ack = at91_aic_mask_irq, .mask = at91_aic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = at91_aic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = at91_aic_unmask_irq, .set_type = at91_aic_set_type, .set_wake = at91_aic_set_wake, diff --git a/arch/arm/mach-integrator/core.c b/arch/arm/mach-integrator/core.c index a0f60e5..84c99a7 100644 --- a/arch/arm/mach-integrator/core.c +++ b/arch/arm/mach-integrator/core.c @@ -2,6 +2,7 @@ * linux/arch/arm/mach-integrator/core.c * * Copyright (C) 2000-2003 Deep Blue Solutions Ltd + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2, as @@ -230,53 +231,62 @@ EXPORT_SYMBOL(cm_control); /* * How long is the timer interval? */ -#define TIMER_INTERVAL (TICKS_PER_uSEC * mSEC_10) -#if TIMER_INTERVAL >= 0x100000 -#define TICKS2USECS(x) (256 * (x) / TICKS_PER_uSEC) -#elif TIMER_INTERVAL >= 0x10000 -#define TICKS2USECS(x) (16 * (x) / TICKS_PER_uSEC) -#else #define TICKS2USECS(x) ((x) / TICKS_PER_uSEC) -#endif static unsigned long timer_reload; +static unsigned long timer_interval; +static unsigned long timer_lxlost; +static int tscok; + +#ifdef CONFIG_IPIPE +int __ipipe_mach_timerint = IRQ_TIMERINT1; +static unsigned long long __ipipe_mach_tsc; +static IPIPE_DEFINE_SPINLOCK(timer_lock); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} +#endif /* - * Returns number of ms since last clock interrupt. Note that interrupts - * will have been disabled by do_gettimeoffset() + * Called with IRQ disabled from do_gettimeofday(). */ -unsigned long integrator_gettimeoffset(void) +static inline unsigned long integrator_getticksoffset(void) { - unsigned long ticks1, ticks2, status; + unsigned long ticks; - /* - * Get the current number of ticks. Note that there is a race - * condition between us reading the timer and checking for - * an interrupt. We get around this by ensuring that the - * counter has not reloaded between our two reads. - */ - ticks2 = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - do { - ticks1 = ticks2; - status = __raw_readl(VA_IC_BASE + IRQ_RAW_STATUS); - ticks2 = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - } while (ticks2 > ticks1); + if (!tscok) + return 0; - /* - * Number of ticks since last interrupt. - */ - ticks1 = timer_reload - ticks2; + ticks = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - /* - * Interrupt pending? If so, we've reloaded once already. - */ - if (status & (1 << IRQ_TIMERINT1)) - ticks1 += timer_reload; + if (ticks > timer_reload) + ticks = 0xffff + timer_reload - ticks; + else + ticks = timer_reload - ticks; + if (timer_interval < 0x10000) + return ticks; + else if (timer_interval < 0x100000) + return ticks * 16; + else + return ticks * 256; +} + +/* + * Returns number of ms since last clock interrupt. Note that interrupts + * will have been disabled by do_gettimeoffset() + */ +unsigned long integrator_gettimeoffset(void) +{ /* * Convert the ticks to usecs */ - return TICKS2USECS(ticks1); + return TICKS2USECS(timer_lxlost + integrator_getticksoffset()); } /* @@ -285,11 +295,22 @@ unsigned long integrator_gettimeoffset(void) static irqreturn_t integrator_timer_interrupt(int irq, void *dev_id) { + timer_lxlost = 0; + +#ifdef CONFIG_IPIPE /* - * clear the interrupt + * If Linux is the only domain, ack the timer and reprogram it */ - writel(1, TIMER1_VA_BASE + TIMER_INTCLR); + if (!__ipipe_mach_timerstolen) { + __ipipe_mach_tsc += integrator_getticksoffset(); +#else + writel(1, TIMER1_VA_BASE + TIMER_INTCLR); +#endif + writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); +#ifdef CONFIG_IPIPE + } +#endif timer_tick(); return IRQ_HANDLED; @@ -301,24 +322,30 @@ static struct irqaction integrator_timer_irq = { .handler = integrator_timer_interrupt, }; -/* - * Set up timer interrupt, and return the current time in seconds. - */ -void __init integrator_time_init(unsigned long reload, unsigned int ctrl) +static inline void set_dec(unsigned long reload) { - unsigned int timer_ctrl = TIMER_CTRL_ENABLE | TIMER_CTRL_PERIODIC; + unsigned int ctrl = TIMER_CTRL_ENABLE | TIMER_CTRL_IE; timer_reload = reload; - timer_ctrl |= ctrl; + timer_interval = reload; - if (timer_reload > 0x100000) { + if (timer_reload >= 0x100000) { timer_reload >>= 8; - timer_ctrl |= TIMER_CTRL_DIV256; - } else if (timer_reload > 0x010000) { + ctrl |= TIMER_CTRL_DIV256; + } else if (timer_reload >= 0x010000) { timer_reload >>= 4; - timer_ctrl |= TIMER_CTRL_DIV16; + ctrl |= TIMER_CTRL_DIV16; } + writel(ctrl, TIMER1_VA_BASE + TIMER_CTRL); + writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); +} + +/* + * Set up timer interrupt, and return the current time in seconds. + */ +void __init integrator_time_init(unsigned long reload, unsigned int ctrl) +{ /* * Initialise to a known state (all timers off) */ @@ -326,12 +353,60 @@ void __init integrator_time_init(unsigned long reload, unsigned int ctrl) writel(0, TIMER1_VA_BASE + TIMER_CTRL); writel(0, TIMER2_VA_BASE + TIMER_CTRL); - writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); - writel(timer_reload, TIMER1_VA_BASE + TIMER_VALUE); - writel(timer_ctrl, TIMER1_VA_BASE + TIMER_CTRL); + set_dec(reload); /* * Make irqs happen for the system timer */ setup_irq(IRQ_TIMERINT1, &integrator_timer_irq); + + tscok = 1; +} + +#ifdef CONFIG_IPIPE + +void __ipipe_mach_acktimer(void) +{ + writel(1, TIMER1_VA_BASE + TIMER_INTCLR); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + unsigned long long result; + unsigned long flags; + + local_irq_save_hw_notrace(flags); + spin_lock(&timer_lock); + result = __ipipe_mach_tsc + integrator_getticksoffset(); + spin_unlock(&timer_lock); + local_irq_restore_hw_notrace(flags); + return result; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +void __ipipe_mach_set_dec(unsigned long reload) +{ + unsigned long ticks; + unsigned long flags; + + spin_lock_irqsave(&timer_lock, flags); + ticks = integrator_getticksoffset(); + __ipipe_mach_tsc += ticks; + timer_lxlost += ticks; + + set_dec(reload); + spin_unlock_irqrestore(&timer_lock, flags); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return readl(TIMER1_VA_BASE + TIMER_VALUE); } +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-integrator/include/mach/entry-macro.S b/arch/arm/mach-integrator/include/mach/entry-macro.S index 7649c57..ab3623d 100644 --- a/arch/arm/mach-integrator/include/mach/entry-macro.S +++ b/arch/arm/mach-integrator/include/mach/entry-macro.S @@ -28,7 +28,11 @@ teq \irqstat, #0 ldreq \irqstat, [\base, #(INTEGRATOR_HDR_IC_OFFSET+IRQ_STATUS)] moveq \irqnr, #IRQ_CIC_START - +#ifdef CONFIG_IPIPE + tst \irqstat, #0x00000040 @ check IRQ_TIMERINT1 first + movne \irqnr, #6 + bne 1003f +#endif /* CONFIG_IPIPE */ 1001: tst \irqstat, #15 bne 1002f add \irqnr, \irqnr, #4 diff --git a/arch/arm/mach-integrator/include/mach/irqs.h b/arch/arm/mach-integrator/include/mach/irqs.h index 1fbe6d1..dea6390 100644 --- a/arch/arm/mach-integrator/include/mach/irqs.h +++ b/arch/arm/mach-integrator/include/mach/irqs.h @@ -79,4 +79,3 @@ #define IRQ_SIC_END 46 #define NR_IRQS 47 - diff --git a/arch/arm/mach-integrator/include/mach/platform.h b/arch/arm/mach-integrator/include/mach/platform.h index e00a262..3b39ce3 100644 --- a/arch/arm/mach-integrator/include/mach/platform.h +++ b/arch/arm/mach-integrator/include/mach/platform.h @@ -417,7 +417,7 @@ * Timer definitions * * Only use timer 1 & 2 - * (both run at 24MHz and will need the clock divider set to 16). + * (both run at 1MHZ on /CP and at 24MHz on /AP) * * Timer 0 runs at bus frequency and therefore could vary and currently * uHAL can't handle that. @@ -430,7 +430,12 @@ #define MAX_TIMER 2 #define MAX_PERIOD 699050 + +#ifdef CONFIG_ARCH_INTEGRATOR_CP +#define TICKS_PER_uSEC 1 +#else #define TICKS_PER_uSEC 24 +#endif /* * These are useconds NOT ticks. diff --git a/arch/arm/mach-integrator/include/mach/timex.h b/arch/arm/mach-integrator/include/mach/timex.h index 1dcb420..a04653a 100644 --- a/arch/arm/mach-integrator/include/mach/timex.h +++ b/arch/arm/mach-integrator/include/mach/timex.h @@ -21,6 +21,6 @@ */ /* - * ?? + * Timer rate */ -#define CLOCK_TICK_RATE (50000000 / 16) +#define CLOCK_TICK_RATE (1000000) diff --git a/arch/arm/mach-integrator/integrator_cp.c b/arch/arm/mach-integrator/integrator_cp.c index 3f35293..91d2eaa 100644 --- a/arch/arm/mach-integrator/integrator_cp.c +++ b/arch/arm/mach-integrator/integrator_cp.c @@ -2,6 +2,7 @@ * linux/arch/arm/mach-integrator/integrator_cp.c * * Copyright (C) 2003 Deep Blue Solutions Ltd + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -21,6 +22,7 @@ #include <linux/amba/clcd.h> #include <linux/amba/mmci.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <asm/clkdev.h> #include <mach/clkdev.h> @@ -161,6 +163,9 @@ static struct irq_chip cic_chip = { .name = "CIC", .ack = cic_mask_irq, .mask = cic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = cic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = cic_unmask_irq, }; @@ -180,6 +185,9 @@ static struct irq_chip pic_chip = { .name = "PIC", .ack = pic_mask_irq, .mask = pic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = pic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = pic_unmask_irq, }; @@ -199,6 +207,9 @@ static struct irq_chip sic_chip = { .name = "SIC", .ack = sic_mask_irq, .mask = sic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = sic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = sic_unmask_irq, }; @@ -218,7 +229,7 @@ sic_handle_irq(unsigned int irq, struct irq_desc *desc) irq += IRQ_SIC_START; - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } while (status); } @@ -571,9 +582,14 @@ static void __init intcp_init(void) #define TIMER_CTRL_IE (1 << 5) /* Interrupt Enable */ +#ifdef CONFIG_IPIPE +unsigned int __ipipe_mach_ticks_per_jiffy = 1000000 * TICKS_PER_uSEC / HZ; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); +#endif + static void __init intcp_timer_init(void) { - integrator_time_init(1000000 / HZ, TIMER_CTRL_IE); + integrator_time_init(1000000 * TICKS_PER_uSEC / HZ, TIMER_CTRL_IE); } static struct sys_timer cp_timer = { diff --git a/arch/arm/mach-ixp4xx/common.c b/arch/arm/mach-ixp4xx/common.c index 3bbf40f..cfb1f44 100644 --- a/arch/arm/mach-ixp4xx/common.c +++ b/arch/arm/mach-ixp4xx/common.c @@ -45,6 +45,65 @@ static void __init ixp4xx_clocksource_init(void); static void __init ixp4xx_clockevent_init(void); static struct clock_event_device clockevent_ixp4xx; +#ifdef CONFIG_IPIPE +#include <linux/ipipe.h> + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif + +int __ipipe_mach_timerint = IRQ_IXP4XX_TIMER1; +int __ipipe_mach_timerstolen = 0; +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_timerint); +EXPORT_SYMBOL(__ipipe_mach_timerstolen); +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int ixp4xx_timer_initialized; + +#define ONE_SHOT_ENABLE (IXP4XX_OST_ENABLE|IXP4XX_OST_ONE_SHOT) + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = + (unsigned *) + (IXP4XX_TIMER_BASE_PHYS + IXP4XX_OSTS_OFFSET); + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + /************************************************************************* * IXP4xx chipset I/O mapping *************************************************************************/ @@ -269,8 +328,12 @@ static irqreturn_t ixp4xx_timer_interrupt(int irq, void *dev_id) { struct clock_event_device *evt = dev_id; +#ifndef CONFIG_IPIPE /* Clear Pending Interrupt by writing '1' to it */ *IXP4XX_OSST = IXP4XX_OSST_TIMER_1_PEND; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ evt->event_handler(evt); @@ -295,6 +358,14 @@ void __init ixp4xx_timer_init(void) /* Reset time-stamp counter */ *IXP4XX_OSTS = 0; +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + ixp4xx_timer_initialized = 1; +#endif /* Connect the interrupt handler and enable the interrupt */ setup_irq(IRQ_IXP4XX_TIMER1, &ixp4xx_timer_irq); @@ -470,6 +541,13 @@ static void ixp4xx_set_mode(enum clock_event_mode mode, *IXP4XX_OSRT1 = osrt | opts; } +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_ixp4xx.name); +} +#endif /* CONFIG_IPIPE */ + static struct clock_event_device clockevent_ixp4xx = { .name = "ixp4xx timer1", .features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT, @@ -491,3 +569,98 @@ static void __init ixp4xx_clockevent_init(void) clockevents_register_device(&clockevent_ixp4xx); } + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + /* Clear Pending Interrupt by writing '1' to it */ + *IXP4XX_OSST = IXP4XX_OSST_TIMER_1_PEND; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = *IXP4XX_OSTS; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(ixp4xx_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = *IXP4XX_OSTS; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + return result.full; + } + + return 0; +} + +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + * + * The timer is aperiodic (most of the time) when running Xenomai, so + * __ipipe_mach_set_dec is called for each timer tick and programs the + * timer hardware for the next tick. + * + */ +#define MIN_DELAY 333 /* 5 usec with the 66.66 MHz system clock */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + unsigned long flags; + if (delay > MIN_DELAY) { + local_irq_save_hw(flags); + *IXP4XX_OSRT1 = delay | ONE_SHOT_ENABLE; + local_irq_restore_hw(flags); + } else { + ipipe_trigger_irq(IRQ_IXP4XX_TIMER1); + } +} + +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +/* + * This returns the number of clock ticks remaining. + */ +unsigned long __ipipe_mach_get_dec(void) +{ + return(*IXP4XX_OST1); /* remaining */ +} + +EXPORT_SYMBOL(__ipipe_mach_get_dec); + +void __ipipe_mach_release_timer(void) +{ + unsigned long flags; + + local_irq_save_hw(flags); + ixp4xx_set_mode(clockevent_ixp4xx.mode, &clockevent_ixp4xx); + if (clockevent_ixp4xx.mode == CLOCK_EVT_MODE_ONESHOT) + ixp4xx_set_next_event(LATCH, &clockevent_ixp4xx); + local_irq_restore_hw(flags); +} + +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx25/devices.c b/arch/arm/mach-mx25/devices.c index 9fdeea1..89f74e8 100644 --- a/arch/arm/mach-mx25/devices.c +++ b/arch/arm/mach-mx25/devices.c @@ -19,6 +19,7 @@ #include <linux/platform_device.h> #include <linux/dma-mapping.h> #include <linux/gpio.h> +#include <mach/common.h> #include <mach/mx25.h> #include <mach/irqs.h> @@ -438,3 +439,14 @@ struct platform_device mx25_fec_device = { .num_resources = ARRAY_SIZE(mx25_fec_resources), .resource = mx25_fec_resources, }; + +#ifdef CONFIG_IPIPE +static int post_cpu_init(void) +{ + ipipe_mach_allow_hwtimer_uaccess(MX25_AIPS1_BASE_ADDR_VIRT, + MX25_AIPS2_BASE_ADDR_VIRT); + return 0; +} + +postcore_initcall(post_cpu_init); +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx3/devices.c b/arch/arm/mach-mx3/devices.c index 6adb586..9040c13 100644 --- a/arch/arm/mach-mx3/devices.c +++ b/arch/arm/mach-mx3/devices.c @@ -22,6 +22,7 @@ #include <linux/platform_device.h> #include <linux/serial.h> #include <linux/gpio.h> +#include <mach/common.h> #include <mach/hardware.h> #include <mach/irqs.h> #include <mach/common.h> @@ -603,3 +604,15 @@ static int mx3_devices_init(void) } subsys_initcall(mx3_devices_init); + +#ifdef CONFIG_IPIPE +static int post_cpu_init(void) +{ + if (cpu_is_mx31()) + ipipe_mach_allow_hwtimer_uaccess(AIPS1_BASE_ADDR_VIRT, + AIPS2_BASE_ADDR_VIRT); + return 0; +} + +postcore_initcall(post_cpu_init); +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx3/mx31ads.c b/arch/arm/mach-mx3/mx31ads.c index 938c549..b7c2139 100644 --- a/arch/arm/mach-mx3/mx31ads.c +++ b/arch/arm/mach-mx3/mx31ads.c @@ -25,6 +25,7 @@ #include <linux/gpio.h> #include <linux/i2c.h> #include <linux/irq.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach-types.h> @@ -134,7 +135,7 @@ static void mx31ads_expio_irq_handler(u32 irq, struct irq_desc *desc) if ((int_valid & 1) == 0) continue; - generic_handle_irq(expio_irq); + ipipe_handle_irq_cond(expio_irq); } } diff --git a/arch/arm/mach-mx3/mx31pdk.c b/arch/arm/mach-mx3/mx31pdk.c index 18715f1..aa47303 100644 --- a/arch/arm/mach-mx3/mx31pdk.c +++ b/arch/arm/mach-mx3/mx31pdk.c @@ -23,6 +23,7 @@ #include <linux/gpio.h> #include <linux/smsc911x.h> #include <linux/platform_device.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach-types.h> @@ -108,7 +109,7 @@ static void mx31pdk_expio_irq_handler(uint32_t irq, struct irq_desc *desc) for (; int_valid != 0; int_valid >>= 1, expio_irq++) { if ((int_valid & 1) == 0) continue; - generic_handle_irq(expio_irq); + ipipe_handle_irq_cond(expio_irq); } } diff --git a/arch/arm/mach-omap2/board-igep0020.c b/arch/arm/mach-omap2/board-igep0020.c index 117b8fd..a13dd43 100644 --- a/arch/arm/mach-omap2/board-igep0020.c +++ b/arch/arm/mach-omap2/board-igep0020.c @@ -164,6 +164,18 @@ static struct twl4030_usb_data igep2_usb_data = { .usb_mode = T2_USB_MODE_ULPI, }; +static struct ehci_hcd_omap_platform_data ehci_pdata __initconst = { + + .port_mode[0] = EHCI_HCD_OMAP_MODE_PHY, + .port_mode[1] = EHCI_HCD_OMAP_MODE_UNKNOWN, + .port_mode[2] = EHCI_HCD_OMAP_MODE_UNKNOWN, + + .phy_reset = true, + .reset_gpio_port[0] = 24, + .reset_gpio_port[1] = -EINVAL, + .reset_gpio_port[2] = -EINVAL +}; + static void __init igep2_init_irq(void) { omap_board_config = igep2_config; @@ -217,6 +229,7 @@ static void __init igep2_init(void) igep2_i2c_init(); omap_serial_init(); usb_musb_init(); + usb_ehci_init(&ehci_pdata); igep2_init_smsc911x(); diff --git a/arch/arm/mach-omap2/irq.c b/arch/arm/mach-omap2/irq.c index 26aeef5..5ed22fa 100644 --- a/arch/arm/mach-omap2/irq.c +++ b/arch/arm/mach-omap2/irq.c @@ -148,6 +148,7 @@ static struct irq_chip omap_irq_chip = { .name = "INTC", .ack = omap_mask_ack_irq, .mask = omap_mask_irq, + .mask_ack = omap_mask_ack_irq, .unmask = omap_unmask_irq, }; @@ -167,8 +168,14 @@ static void __init omap_irq_bank_init_one(struct omap_irq_bank *bank) while (!(intc_bank_read_reg(bank, INTC_SYSSTATUS) & 0x1)) /* Wait for reset to complete */; +#ifndef CONFIG_IPIPE /* Enable autoidle */ intc_bank_write_reg(1 << 0, bank, INTC_SYSCONFIG); +#else /* !CONFIG_IPIPE */ + /* Disable autoidle */ + intc_bank_write_reg(0, bank, INTC_SYSCONFIG); + intc_bank_write_reg(0x2, bank, INTC_IDLE); +#endif /* CONFIG_IPIPE */ } int omap_irq_pending(void) diff --git a/arch/arm/mach-omap2/timer-gp.c b/arch/arm/mach-omap2/timer-gp.c index cd04dea..389df09 100644 --- a/arch/arm/mach-omap2/timer-gp.c +++ b/arch/arm/mach-omap2/timer-gp.c @@ -35,6 +35,7 @@ #include <linux/irq.h> #include <linux/clocksource.h> #include <linux/clockchips.h> +#include <linux/ipipe.h> #include <asm/mach/time.h> #include <plat/dmtimer.h> @@ -48,13 +49,74 @@ static struct clock_event_device clockevent_gpt; static u8 __initdata gptimer_id = 1; static u8 __initdata inited; struct omap_dm_timer *gptimer_wakeup; +#ifndef CONFIG_OMAP_32K_TIMER +static struct omap_dm_timer *gpt_clocksource; +#endif /* !CONFIG_OMAP_32K_TIMER */ + +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int omap2_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *) + omap_dm_timer_get_phys_counter_addr(gpt_clocksource); + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ static irqreturn_t omap2_gp_timer_interrupt(int irq, void *dev_id) { - struct omap_dm_timer *gpt = (struct omap_dm_timer *)dev_id; struct clock_event_device *evt = &clockevent_gpt; +#ifndef CONFIG_IPIPE + struct omap_dm_timer *gpt = (struct omap_dm_timer *)dev_id; omap_dm_timer_write_status(gpt, OMAP_TIMER_INT_OVERFLOW); +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ evt->event_handler(evt); return IRQ_HANDLED; @@ -126,6 +188,13 @@ int __init omap2_gp_clockevent_set_gptimer(u8 id) return 0; } +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_gpt.name); +} +#endif /* CONFIG_IPIPE */ + static void __init omap2_gp_clockevent_init(void) { u32 tick_rate; @@ -158,6 +227,12 @@ static void __init omap2_gp_clockevent_init(void) gptimer_id, tick_rate); omap2_gp_timer_irq.dev_id = (void *)gptimer; + +#ifdef CONFIG_IPIPE + __ipipe_mach_timerint = omap_dm_timer_get_irq(gptimer); + __ipipe_mach_ticks_per_jiffy = (tick_rate + HZ / 2) / HZ; +#endif /* CONFIG_IPIPE */ + setup_irq(omap_dm_timer_get_irq(gptimer), &omap2_gp_timer_irq); omap_dm_timer_set_int_enable(gptimer, OMAP_TIMER_INT_OVERFLOW); @@ -168,7 +243,6 @@ static void __init omap2_gp_clockevent_init(void) clockevent_gpt.min_delta_ns = clockevent_delta2ns(3, &clockevent_gpt); /* Timer internal resynch latency. */ - clockevent_gpt.cpumask = cpumask_of(0); clockevents_register_device(&clockevent_gpt); } @@ -176,10 +250,10 @@ static void __init omap2_gp_clockevent_init(void) /* Clocksource code */ #ifdef CONFIG_OMAP_32K_TIMER -/* +/* * When 32k-timer is enabled, don't use GPTimer for clocksource * instead, just leave default clocksource which uses the 32k - * sync counter. See clocksource setup in see plat-omap/common.c. + * sync counter. See clocksource setup in see plat-omap/common.c. */ static inline void __init omap2_gp_clocksource_init(void) {} @@ -187,7 +261,6 @@ static inline void __init omap2_gp_clocksource_init(void) {} /* * clocksource */ -static struct omap_dm_timer *gpt_clocksource; static cycle_t clocksource_read_cycles(struct clocksource *cs) { return (cycle_t)omap_dm_timer_read_counter(gpt_clocksource); @@ -225,6 +298,16 @@ static void __init omap2_gp_clocksource_init(void) clocksource_gpt.mult = clocksource_khz2mult(tick_rate/1000, clocksource_gpt.shift); + +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + omap2_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + if (clocksource_register(&clocksource_gpt)) printk(err2, clocksource_gpt.name); } @@ -245,3 +328,79 @@ static void __init omap2_gp_timer_init(void) struct sys_timer omap_timer = { .init = omap2_gp_timer_init, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + omap_dm_timer_write_status(gptimer, OMAP_TIMER_INT_OVERFLOW); + omap_dm_timer_read_status(gptimer); +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + if (likely(omap2_timer_initialized)) { + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = omap_dm_timer_read_counter(gpt_clocksource); + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); + } +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(omap2_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n" : + "=r"(result.full) : "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = omap_dm_timer_read_counter(gpt_clocksource); + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > 3) + omap_dm_timer_set_load_start(gptimer, 0, 0xffffffff - delay); + else + ipipe_trigger_irq(__ipipe_mach_timerint); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + struct clock_event_device *ckdev = &clockevent_gpt; + ckdev->set_mode(ckdev->mode, ckdev); + if (ckdev->mode == CLOCK_EVT_MODE_ONESHOT) + ckdev->set_next_event(__ipipe_mach_ticks_per_jiffy, ckdev); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return 0xffffffff - omap_dm_timer_read_counter(gptimer); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-pxa/irq.c b/arch/arm/mach-pxa/irq.c index 1beb40f..3b27f7b 100644 --- a/arch/arm/mach-pxa/irq.c +++ b/arch/arm/mach-pxa/irq.c @@ -51,6 +51,9 @@ static struct irq_chip pxa_internal_irq_chip = { .name = "SC", .ack = pxa_mask_irq, .mask = pxa_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = pxa_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = pxa_unmask_irq, }; diff --git a/arch/arm/mach-pxa/leds-idp.c b/arch/arm/mach-pxa/leds-idp.c index 8b9c171..144cdc1 100644 --- a/arch/arm/mach-pxa/leds-idp.c +++ b/arch/arm/mach-pxa/leds-idp.c @@ -13,6 +13,7 @@ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/leds-lubbock.c b/arch/arm/mach-pxa/leds-lubbock.c index e26d5ef..706cced 100644 --- a/arch/arm/mach-pxa/leds-lubbock.c +++ b/arch/arm/mach-pxa/leds-lubbock.c @@ -12,6 +12,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/leds-mainstone.c b/arch/arm/mach-pxa/leds-mainstone.c index db4af5e..c204cd4 100644 --- a/arch/arm/mach-pxa/leds-mainstone.c +++ b/arch/arm/mach-pxa/leds-mainstone.c @@ -11,6 +11,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/lpd270.c b/arch/arm/mach-pxa/lpd270.c index 1373c22..f931637 100644 --- a/arch/arm/mach-pxa/lpd270.c +++ b/arch/arm/mach-pxa/lpd270.c @@ -24,6 +24,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/pwm_backlight.h> +#include <linux/ipipe.h> #include <asm/types.h> #include <asm/setup.h> @@ -124,7 +125,7 @@ static void lpd270_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear useless edge notification */ if (likely(pending)) { irq = LPD270_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); pending = __raw_readw(LPD270_INT_STATUS) & lpd270_irq_enabled; diff --git a/arch/arm/mach-pxa/lubbock.c b/arch/arm/mach-pxa/lubbock.c index 98ee7e5..5dbef64 100644 --- a/arch/arm/mach-pxa/lubbock.c +++ b/arch/arm/mach-pxa/lubbock.c @@ -22,6 +22,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/smc91x.h> +#include <linux/ipipe.h> #include <linux/spi/spi.h> #include <linux/spi/ads7846.h> @@ -161,7 +162,7 @@ static void lubbock_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear our parent irq */ if (likely(pending)) { irq = LUBBOCK_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = LUB_IRQ_SET_CLR & lubbock_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/mainstone.c b/arch/arm/mach-pxa/mainstone.c index 851ee0f..9c1428a 100644 --- a/arch/arm/mach-pxa/mainstone.c +++ b/arch/arm/mach-pxa/mainstone.c @@ -27,6 +27,7 @@ #include <linux/gpio_keys.h> #include <linux/pwm_backlight.h> #include <linux/smc91x.h> +#include <linux/ipipe.h> #include <asm/types.h> #include <asm/setup.h> @@ -165,7 +166,7 @@ static void mainstone_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear useless edge notification */ if (likely(pending)) { irq = MAINSTONE_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = MST_INTSETCLR & mainstone_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/pcm990-baseboard.c b/arch/arm/mach-pxa/pcm990-baseboard.c index d5255ae..cf0ec2a 100644 --- a/arch/arm/mach-pxa/pcm990-baseboard.c +++ b/arch/arm/mach-pxa/pcm990-baseboard.c @@ -24,6 +24,7 @@ #include <linux/platform_device.h> #include <linux/i2c.h> #include <linux/pwm_backlight.h> +#include <linux/ipipe.h> #include <media/soc_camera.h> @@ -263,7 +264,7 @@ static void pcm990_irq_handler(unsigned int irq, struct irq_desc *desc) GPIO_bit(PCM990_CTRL_INT_IRQ_GPIO); if (likely(pending)) { irq = PCM027_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = (~PCM990_INTSETCLR) & pcm990_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/time.c b/arch/arm/mach-pxa/time.c index 750c448..2b7e7f8 100644 --- a/arch/arm/mach-pxa/time.c +++ b/arch/arm/mach-pxa/time.c @@ -24,6 +24,60 @@ #include <asm/mach/time.h> #include <mach/regs-ost.h> +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint = IRQ_OST0; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int pxa_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *) 0x40A00010; + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + /* * This is PXA's sched_clock implementation. This has a resolution * of at least 308 ns and a maximum value of 208 days. @@ -66,8 +120,12 @@ pxa_ost0_interrupt(int irq, void *dev_id) struct clock_event_device *c = dev_id; /* Disarm the compare/match, signal the event. */ +#ifndef CONFIG_IPIPE OIER &= ~OIER_E0; OSSR = OSSR_M0; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ c->event_handler(c); return IRQ_HANDLED; @@ -79,8 +137,8 @@ pxa_osmr0_set_next_event(unsigned long delta, struct clock_event_device *dev) unsigned long flags, next, oscr; raw_local_irq_save(flags); - OIER |= OIER_E0; next = OSCR + delta; + OIER |= OIER_E0; OSMR0 = next; oscr = OSCR; raw_local_irq_restore(flags); @@ -125,6 +183,13 @@ static struct clock_event_device ckevt_pxa_osmr0 = { .set_mode = pxa_osmr0_set_mode, }; +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, ckevt_pxa_osmr0.name); +} +#endif /* CONFIG_IPIPE */ + static cycle_t pxa_read_oscr(struct clocksource *cs) { return OSCR; @@ -168,6 +233,15 @@ static void __init pxa_timer_init(void) setup_irq(IRQ_OST0, &pxa_ost0_irq); +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + pxa_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + clocksource_register(&cksrc_pxa_oscr0); clockevents_register_device(&ckevt_pxa_osmr0); } @@ -213,3 +287,81 @@ struct sys_timer pxa_timer = { .suspend = pxa_timer_suspend, .resume = pxa_timer_resume, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + OSSR = OSSR_M0; /* Clear match on timer 0 */ + OIER &= ~OIER_E0; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = OSCR; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(pxa_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = OSCR; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > MIN_OSCR_DELTA) { + unsigned long flags; + + local_irq_save_hw(flags); + OSMR0 = delay + OSCR; + OIER |= OIER_E0; + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(IRQ_OST0); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + pxa_osmr0_set_mode(ckevt_pxa_osmr0.mode, &ckevt_pxa_osmr0); + if (ckevt_pxa_osmr0.mode == CLOCK_EVT_MODE_ONESHOT) + pxa_osmr0_set_next_event(LATCH, &ckevt_pxa_osmr0); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return OSMR0 - OSCR; +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-pxa/viper.c b/arch/arm/mach-pxa/viper.c index 89f258c..51e1d64 100644 --- a/arch/arm/mach-pxa/viper.c +++ b/arch/arm/mach-pxa/viper.c @@ -41,6 +41,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/mtd/physmap.h> +#include <linux/ipipe.h> #include <mach/pxa25x.h> #include <mach/audio.h> @@ -285,7 +286,7 @@ static void viper_irq_handler(unsigned int irq, struct irq_desc *desc) if (likely(pending)) { irq = viper_bit_to_irq(__ffs(pending)); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = viper_irq_pending(); } while (pending); diff --git a/arch/arm/mach-s3c2410/include/mach/irqs.h b/arch/arm/mach-s3c2410/include/mach/irqs.h index 6c12c63..06ccc4e 100644 --- a/arch/arm/mach-s3c2410/include/mach/irqs.h +++ b/arch/arm/mach-s3c2410/include/mach/irqs.h @@ -3,6 +3,8 @@ * Copyright (c) 2003-2005 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. diff --git a/arch/arm/mach-s3c2440/irq.c b/arch/arm/mach-s3c2440/irq.c index 0c049b9..23a5c90 100644 --- a/arch/arm/mach-s3c2440/irq.c +++ b/arch/arm/mach-s3c2440/irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -25,6 +27,7 @@ #include <linux/ioport.h> #include <linux/sysdev.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -57,10 +60,10 @@ static void s3c_irq_demux_wdtac97(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_S3C2440_WDT); + ipipe_handle_irq_cond(IRQ_S3C2440_WDT); } if (subsrc & 2) { - generic_handle_irq(IRQ_S3C2440_AC97); + ipipe_handle_irq_cond(IRQ_S3C2440_AC97); } } } diff --git a/arch/arm/mach-sa1100/irq.c b/arch/arm/mach-sa1100/irq.c index 3093d46..04a56d2 100644 --- a/arch/arm/mach-sa1100/irq.c +++ b/arch/arm/mach-sa1100/irq.c @@ -15,6 +15,7 @@ #include <linux/irq.h> #include <linux/ioport.h> #include <linux/sysdev.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach/irq.h> @@ -125,7 +126,7 @@ sa1100_high_gpio_handler(unsigned int irq, struct irq_desc *desc) mask >>= 11; do { if (mask & 1) - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); mask >>= 1; irq++; } while (mask); @@ -217,6 +218,9 @@ static struct irq_chip sa1100_normal_chip = { .name = "SC", .ack = sa1100_mask_irq, .mask = sa1100_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = sa1100_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = sa1100_unmask_irq, .set_wake = sa1100_set_wake, }; diff --git a/arch/arm/mach-sa1100/leds-assabet.c b/arch/arm/mach-sa1100/leds-assabet.c index 64e9b4b..0d4e5a4 100644 --- a/arch/arm/mach-sa1100/leds-assabet.c +++ b/arch/arm/mach-sa1100/leds-assabet.c @@ -10,6 +10,7 @@ * - Red - on if system is not idle */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-badge4.c b/arch/arm/mach-sa1100/leds-badge4.c index cf1e384..e41140f 100644 --- a/arch/arm/mach-sa1100/leds-badge4.c +++ b/arch/arm/mach-sa1100/leds-badge4.c @@ -11,6 +11,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-cerf.c b/arch/arm/mach-sa1100/leds-cerf.c index 259b48e..3ab9eff 100644 --- a/arch/arm/mach-sa1100/leds-cerf.c +++ b/arch/arm/mach-sa1100/leds-cerf.c @@ -4,6 +4,7 @@ * Author: ??? */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-hackkit.c b/arch/arm/mach-sa1100/leds-hackkit.c index 2bce137..0a4fe91 100644 --- a/arch/arm/mach-sa1100/leds-hackkit.c +++ b/arch/arm/mach-sa1100/leds-hackkit.c @@ -10,6 +10,7 @@ * as cpu led, the green one is used as timer led. */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-lart.c b/arch/arm/mach-sa1100/leds-lart.c index 0505a1f..f03128d 100644 --- a/arch/arm/mach-sa1100/leds-lart.c +++ b/arch/arm/mach-sa1100/leds-lart.c @@ -10,6 +10,7 @@ * pace of the LED. */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-simpad.c b/arch/arm/mach-sa1100/leds-simpad.c index d50f4ee..2c154e3 100644 --- a/arch/arm/mach-sa1100/leds-simpad.c +++ b/arch/arm/mach-sa1100/leds-simpad.c @@ -4,6 +4,7 @@ * Author: Juergen Messerer <juergen.messerer@siemens.ch> */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/time.c b/arch/arm/mach-sa1100/time.c index b9cbb56..90c62f1 100644 --- a/arch/arm/mach-sa1100/time.c +++ b/arch/arm/mach-sa1100/time.c @@ -14,19 +14,78 @@ #include <linux/irq.h> #include <linux/timex.h> #include <linux/clockchips.h> +#include <linux/ipipe.h> #include <asm/mach/time.h> #include <mach/hardware.h> #define MIN_OSCR_DELTA 2 +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint = IRQ_OST0; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int sa1100_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *)0x90000010; + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + static irqreturn_t sa1100_ost0_interrupt(int irq, void *dev_id) { struct clock_event_device *c = dev_id; /* Disarm the compare/match, signal the event. */ +#ifndef CONFIG_IPIPE OIER &= ~OIER_E0; OSSR = OSSR_M0; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ c->event_handler(c); return IRQ_HANDLED; @@ -77,7 +136,14 @@ static struct clock_event_device ckevt_sa1100_osmr0 = { .set_mode = sa1100_osmr0_set_mode, }; -static cycle_t sa1100_read_oscr(struct clocksource *s) +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, ckevt_sa1100_osmr0.name); +} +#endif /* CONFIG_IPIPE */ + +static cycle_t sa1100_read_oscr(struct clocksource *cs) { return OSCR; } @@ -116,6 +182,15 @@ static void __init sa1100_timer_init(void) setup_irq(IRQ_OST0, &sa1100_timer_irq); +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + sa1100_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + clocksource_register(&cksrc_sa1100_oscr); clockevents_register_device(&ckevt_sa1100_osmr0); } @@ -156,3 +231,80 @@ struct sys_timer sa1100_timer = { .suspend = sa1100_timer_suspend, .resume = sa1100_timer_resume, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + OSSR = OSSR_M0; /* Clear match on timer 0 */ + OIER &= ~OIER_E0; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = OSCR; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(sa1100_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = OSCR; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > MIN_OSCR_DELTA) { + unsigned long flags; + + local_irq_save_hw(flags); + OSMR0 = delay + OSCR; + OIER |= OIER_E0; + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(IRQ_OST0); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + sa1100_osmr0_set_mode(ckevt_sa1100_osmr0.mode, &ckevt_sa1100_osmr0); + if (ckevt_sa1100_osmr0.mode == CLOCK_EVT_MODE_ONESHOT) + sa1100_osmr0_set_next_event(LATCH, &ckevt_sa1100_osmr0); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return OSMR0 - OSCR; +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index baf6384..9c6421c 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -781,3 +781,83 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARM_FCSE + bool "Fast Context Switch Extension (EXPERIMENTAL)" + depends on EXPERIMENTAL && !SMP && (CPU_32v4 || CPU_32v4T || CPU_32v5) + help + The Fast Context Switch Extension (FCSE for short) is an extension of + some ARM processors which allows to switch contexts between processes + without flushing cache and saves a few tens of microseconds in the + worst case. + + Enabling this option makes linux use the FCSE. + + We propose two modes: + - the guaranteed mode: we guarantee that there will never be any cache + flush when switching context, but this means that there can not be + more than 95 running processes in the system, each with a virtual + memory space smaller than 32MB, and that the shared memory + mappings do not use cache; + - the best effort mode: we allow some cache flushes to happen from + time to time, but do not limit the number of processes or the + virtual memory space available for each process, and the shared + memory mappings use cache. + +if ARM_FCSE + +choice + prompt "FCSE mode" + default ARM_FCSE_BEST_EFFORT + help + This option allow setting which FCSE mode will be used. + +config ARM_FCSE_GUARANTEED + bool "guaranteed" + help + Select guaranteed mode. + +config ARM_FCSE_BEST_EFFORT + bool "best effort" + help + Select best-effort mode. + +endchoice + +config ARM_FCSE_DYNPID + bool "Dynamic FCSE pid allocation" + depends on ARM_FCSE_BEST_EFFORT && !IPIPE + help + When this option is enabled, the kernel may decide to change the FCSE + pid of a process when switching context. This helps reducing the + number of cache flushes, but adds some overhead to the context + switches. + +config ARM_FCSE_PREEMPT_FLUSH + bool "Preemptible cache flushes" + help + When FCSE is enabled, some cache flushes happen with preemption + disabled, with this option, we make them preemptible, this results in + better response time, but may result in the need to restart a cache + flush if it was preempted, so in a global higher cpu consumption. + +config ARM_FCSE_MESSAGES + bool "help messages" + help + When FCSE is enabled in best-effort mode, due to the VM space + reduction, a too large stack size limit may result in processes + exceeding the 32MB limit too easily. A too small stack size may result + in stack overflows. Enabling this option will print messages in these + situations to assist you in tuning the stack size limit. + + In guaranteed mode, this option will cause message to be printed if + one of the hard limits (95 proceses, 32 MB VM space) is exceeded. + +config ARM_FCSE_DEBUG + bool "FCSE debug" + select ARM_FCSE_MESSAGES + help + This option enables some internal debug checks. It has a high + overhead, and is only useful for debugging the FCSE code. + +endif diff --git a/arch/arm/mm/alignment.c b/arch/arm/mm/alignment.c index 62820ed..510a94f 100644 --- a/arch/arm/mm/alignment.c +++ b/arch/arm/mm/alignment.c @@ -726,6 +726,9 @@ do_alignment(unsigned long addr, unsigned int fsr, struct pt_regs *regs) int isize = 4; int thumb2_32b = 0; + if (ipipe_trap_notify(IPIPE_TRAP_ALIGNMENT,regs)) + return 0; + instrptr = instruction_pointer(regs); fs = get_fs(); diff --git a/arch/arm/mm/copypage-v4mc.c b/arch/arm/mm/copypage-v4mc.c index 7370a71..3a7928a 100644 --- a/arch/arm/mm/copypage-v4mc.c +++ b/arch/arm/mm/copypage-v4mc.c @@ -44,7 +44,7 @@ static DEFINE_SPINLOCK(minicache_lock); * instruction. If your processor does not supply this, you have to write your * own copy_user_highpage that does the right thing. */ -static void __naked +static void notrace __naked mc_copy_user_page(void *from, void *to) { asm volatile( diff --git a/arch/arm/mm/copypage-xscale.c b/arch/arm/mm/copypage-xscale.c index 76824d3..db642ce 100644 --- a/arch/arm/mm/copypage-xscale.c +++ b/arch/arm/mm/copypage-xscale.c @@ -42,7 +42,7 @@ static DEFINE_SPINLOCK(minicache_lock); * Dcache aliasing issue. The writes will be forwarded to the write buffer, * and merged as appropriate. */ -static void __naked +static void notrace __naked mc_copy_user_page(void *from, void *to) { /* diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index 10e0680..4413c0e 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -73,6 +73,10 @@ void show_pte(struct mm_struct *mm, unsigned long addr) if (!mm) mm = &init_mm; +#ifdef CONFIG_ARM_FCSE + printk(KERN_ALERT "fcse pid: %ld, 0x%08lx\n", + mm->context.fcse.pid >> FCSE_PID_SHIFT, mm->context.fcse.pid); +#endif /* CONFIG_ARM_FCSE */ printk(KERN_ALERT "pgd = %p\n", mm->pgd); pgd = pgd_offset(mm, addr); printk(KERN_ALERT "[%08lx] *pgd=%08lx", addr, pgd_val(*pgd)); @@ -161,11 +165,21 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr, if (user_debug & UDBG_SEGV) { printk(KERN_DEBUG "%s: unhandled page fault (%d) at 0x%08lx, code 0x%03x\n", tsk->comm, sig, addr, fsr); +#ifdef CONFIG_ARM_FCSE_DYNPID + /* Disable preemption to avoid page tables changing under our + feet */ + preempt_disable(); +#endif /* CONFIG_ARM_FCSE_DYNPID */ show_pte(tsk->mm, addr); +#ifdef CONFIG_ARM_FCSE_DYNPID + preempt_enable(); +#endif /* CONFIG_ARM_FCSE_DYNPID */ show_regs(regs); } #endif + fcse_notify_segv(tsk->mm, addr, regs); + tsk->thread.address = addr; tsk->thread.error_code = fsr; tsk->thread.trap_no = 14; @@ -266,6 +280,9 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (notify_page_fault(regs, fsr)) return 0; + if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs)) + return 0; + tsk = current; mm = tsk->mm; @@ -386,6 +403,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr, if (addr < TASK_SIZE) return do_page_fault(addr, fsr, regs); + if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs)) + return 0; + index = pgd_index(addr); /* @@ -429,6 +449,10 @@ do_translation_fault(unsigned long addr, unsigned int fsr, static int do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + + if (ipipe_trap_notify(IPIPE_TRAP_SECTION,regs)) + return 0; + do_bad_area(addr, fsr, regs); return 0; } @@ -439,6 +463,9 @@ do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) static int do_bad(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + if (ipipe_trap_notify(IPIPE_TRAP_DABT,regs)) + return 0; + return 1; } @@ -514,6 +541,9 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (!inf->fn(addr, fsr & ~FSR_LNX_PF, regs)) return; + if (ipipe_trap_notify(IPIPE_TRAP_UNKNOWN,regs)) + return; + printk(KERN_ALERT "Unhandled fault: %s (0x%03x) at 0x%08lx\n", inf->name, fsr, addr); @@ -579,3 +609,42 @@ do_PrefetchAbort(unsigned long addr, unsigned int ifsr, struct pt_regs *regs) arm_notify_die("", regs, &info, ifsr, 0); } +#ifdef CONFIG_IPIPE +extern spinlock_t pgd_lock; +extern struct page *pgd_list; + +static void vmalloc_sync_one(pgd_t *pgd, unsigned long addr) +{ + unsigned int index = pgd_index(addr); + pgd_t *pgd_k; + pmd_t *pmd, *pmd_k; + + pgd += index; + pgd_k = init_mm.pgd + index; + + if (!pgd_present(*pgd)) + set_pgd(pgd, *pgd_k); + + pmd_k = pmd_offset(pgd_k, addr); + pmd = pmd_offset(pgd, addr); + + copy_pmd(pmd, pmd_k); +} + +void __ipipe_pin_range_globally(unsigned long start, unsigned long end) +{ + unsigned long next, addr = start; + + do { + unsigned long flags; + struct page *page; + + next = pgd_addr_end(addr, end); + spin_lock_irqsave(&pgd_lock, flags); + for (page = pgd_list; page; page = (struct page *)page->index) + vmalloc_sync_one(page_address(page), addr); + spin_unlock_irqrestore(&pgd_lock, flags); + + } while (addr = next, addr != end); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mm/ioremap.c b/arch/arm/mm/ioremap.c index 0ab75c6..32fdbe6 100644 --- a/arch/arm/mm/ioremap.c +++ b/arch/arm/mm/ioremap.c @@ -316,6 +316,7 @@ __arm_ioremap_pfn(unsigned long pfn, unsigned long offset, size_t size, } flush_cache_vmap(addr, addr + size); + __ipipe_pin_range_globally(addr, addr + size); return (void __iomem *) (offset + addr); } EXPORT_SYMBOL(__arm_ioremap_pfn); diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index f5abc51..fb22056 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -29,7 +29,8 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - unsigned long start_addr; + unsigned long start_addr = addr; + #ifdef CONFIG_CPU_V6 unsigned int cache_type; int do_align = 0, aliasing = 0; @@ -50,6 +51,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, #define aliasing 0 #endif +#ifdef CONFIG_ARM_FCSE + start_addr = addr; +#endif /* CONFIG_ARM_FCSE */ /* * We enforce the MAP_FIXED case. */ @@ -57,7 +61,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, if (aliasing && flags & MAP_SHARED && (addr - (pgoff << PAGE_SHIFT)) & (SHMLBA - 1)) return -EINVAL; - return addr; + goto found_addr; } if (len > TASK_SIZE) @@ -72,13 +76,13 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, vma = find_vma(mm, addr); if (TASK_SIZE - len >= addr && (!vma || addr + len <= vma->vm_start)) - return addr; + goto found_addr; } if (len > mm->cached_hole_size) { - start_addr = addr = mm->free_area_cache; + start_addr = addr = mm->free_area_cache; } else { - start_addr = addr = TASK_UNMAPPED_BASE; - mm->cached_hole_size = 0; + start_addr = addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; } full_search: @@ -106,14 +110,31 @@ full_search: * Remember the place where we stopped the search: */ mm->free_area_cache = addr + len; - return addr; + goto found_addr; } if (addr + mm->cached_hole_size < vma->vm_start) - mm->cached_hole_size = vma->vm_start - addr; + mm->cached_hole_size = vma->vm_start - addr; addr = vma->vm_end; if (do_align) addr = COLOUR_ALIGN(addr, pgoff); } + + found_addr: +#ifdef CONFIG_ARM_FCSE + { + unsigned long new_addr = fcse_check_mmap_addr(mm, start_addr, + addr, len, flags); + if (new_addr != addr) { + addr = new_addr; + if (!(addr & ~PAGE_MASK)) { + start_addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; + goto full_search; + } + } + } +#endif /* CONFIG_ARM_FCSE */ + return addr; } diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 2690146..16c48a3 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -18,6 +18,41 @@ #define FIRST_KERNEL_PGD_NR (FIRST_USER_PGD_NR + USER_PTRS_PER_PGD) +#ifdef CONFIG_IPIPE +/* Copied from arch/i386/mm/pgdtable.c, maintains the list of pgds for the + implementation of ipipe_pin_range_globally in arch/arm/mm/fault.c. */ +DEFINE_SPINLOCK(pgd_lock); +struct page *pgd_list; + +#define pgd_list_lock(flags) spin_lock_irqsave(&pgd_lock, flags) +#define pgd_list_unlock(flags) spin_unlock_irqrestore(&pgd_lock, flags) + +static inline void pgd_list_add(pgd_t *pgd) +{ + struct page *page = virt_to_page(pgd); + page->index = (unsigned long)pgd_list; + if (pgd_list) + set_page_private(pgd_list, (unsigned long)&page->index); + pgd_list = page; + set_page_private(page, (unsigned long)&pgd_list); +} + +static inline void pgd_list_del(pgd_t *pgd) +{ + struct page *next, **pprev, *page = virt_to_page(pgd); + next = (struct page *)page->index; + pprev = (struct page **)page_private(page); + *pprev = next; + if (next) + set_page_private(next, (unsigned long)pprev); +} +#else /* !CONFIG_IPIPE */ +#define pgd_list_lock(flags) ((void) (flags)) +#define pgd_list_unlock(flags) ((void) (flags)) +#define pgd_list_add(pgd) do { } while (0) +#define pgd_list_del(pgd) do { } while (0) +#endif /* !CONFIG_IPIPE */ + /* * need to get a 16k page for level 1 */ @@ -26,6 +61,7 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) pgd_t *new_pgd, *init_pgd; pmd_t *new_pmd, *init_pmd; pte_t *new_pte, *init_pte; + unsigned long flags; new_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, 2); if (!new_pgd) @@ -37,12 +73,20 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) * Copy over the kernel and IO PGD entries */ init_pgd = pgd_offset_k(0); + pgd_list_lock(flags); memcpy(new_pgd + FIRST_KERNEL_PGD_NR, init_pgd + FIRST_KERNEL_PGD_NR, (PTRS_PER_PGD - FIRST_KERNEL_PGD_NR) * sizeof(pgd_t)); + pgd_list_add(new_pgd); + pgd_list_unlock(flags); clean_dcache_area(new_pgd, PTRS_PER_PGD * sizeof(pgd_t)); if (!vectors_high()) { +#ifdef CONFIG_ARM_FCSE + /* FCSE does not work without high vectors. */ + BUG(); +#endif /* CONFIG_ARM_FCSE */ + /* * On ARM, first page must always be allocated since it * contains the machine vectors. @@ -74,6 +118,7 @@ no_pgd: void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) { + unsigned long flags; pmd_t *pmd; pgtable_t pte; @@ -81,7 +126,7 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) return; /* pgd is always present and good */ - pmd = pmd_off(pgd, 0); + pmd = pmd_off(pgd + pgd_index(fcse_va_to_mva(mm, 0)), 0); if (pmd_none(*pmd)) goto free; if (pmd_bad(*pmd)) { @@ -95,5 +140,8 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) pte_free(mm, pte); pmd_free(mm, pmd); free: + pgd_list_lock(flags); + pgd_list_del(pgd); + pgd_list_unlock(flags); free_pages((unsigned long) pgd, 2); } diff --git a/arch/arm/mm/proc-arm920.S b/arch/arm/mm/proc-arm920.S index 471669e..e474ecc 100644 --- a/arch/arm/mm/proc-arm920.S +++ b/arch/arm/mm/proc-arm920.S @@ -322,6 +322,11 @@ ENTRY(cpu_arm920_dcache_clean_area) ENTRY(cpu_arm920_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 3f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -339,6 +344,10 @@ ENTRY(cpu_arm920_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* !CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +3: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-arm926.S b/arch/arm/mm/proc-arm926.S index 64db6e2..ce7aa93 100644 --- a/arch/arm/mm/proc-arm926.S +++ b/arch/arm/mm/proc-arm926.S @@ -338,6 +338,11 @@ ENTRY(cpu_arm926_dcache_clean_area) ENTRY(cpu_arm926_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -347,6 +352,10 @@ ENTRY(cpu_arm926_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-feroceon.S b/arch/arm/mm/proc-feroceon.S index dbc3938..d8da8d6 100644 --- a/arch/arm/mm/proc-feroceon.S +++ b/arch/arm/mm/proc-feroceon.S @@ -422,6 +422,12 @@ ENTRY(cpu_feroceon_dcache_clean_area) .align 5 ENTRY(cpu_feroceon_switch_mm) #ifdef CONFIG_MMU +#ifndef CONFIG_ARM_FCSE_GUARANTEED +#ifdef CONFIG_ARM_FCSE + cmp r2, #0 + mov r2, lr + beq 2f +#else /* !CONFIG_ARM_FCSE */ /* * Note: we wish to call __flush_whole_cache but we need to preserve * lr to do so. The only way without touching main memory is to @@ -429,12 +435,19 @@ ENTRY(cpu_feroceon_switch_mm) * compensate locally for the skipped ops if it is not set. */ mov r2, lr @ abuse r2 to preserve lr +#endif /* !CONFIG_ARM_FCSE */ bl __flush_whole_cache @ if r2 contains the VM_EXEC bit then the next 2 ops are done already tst r2, #VM_EXEC mcreq p15, 0, ip, c7, c5, 0 @ invalidate I cache mcreq p15, 0, ip, c7, c10, 4 @ drain WB +#ifdef CONFIG_ARM_FCSE +2: +#endif +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + mov r2, lr +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs mov pc, r2 diff --git a/arch/arm/mm/proc-xscale.S b/arch/arm/mm/proc-xscale.S index 93df472..f794a86 100644 --- a/arch/arm/mm/proc-xscale.S +++ b/arch/arm/mm/proc-xscale.S @@ -416,9 +416,18 @@ ENTRY(cpu_xscale_dcache_clean_area) */ .align 5 ENTRY(cpu_xscale_switch_mm) +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED clean_d_cache r1, r2 mcr p15, 0, ip, c7, c5, 0 @ Invalidate I cache & BTB mcr p15, 0, ip, c7, c10, 4 @ Drain Write (& Fill) Buffer +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs cpwait_ret lr, ip diff --git a/arch/arm/plat-mxc/cpu.c b/arch/arm/plat-mxc/cpu.c index 386e0d5..0bb140e 100644 --- a/arch/arm/plat-mxc/cpu.c +++ b/arch/arm/plat-mxc/cpu.c @@ -1,5 +1,5 @@ - #include <linux/module.h> +#include <linux/io.h> unsigned int __mxc_cpu_type; EXPORT_SYMBOL(__mxc_cpu_type); @@ -9,3 +9,29 @@ void mxc_set_cpu_type(unsigned int type) __mxc_cpu_type = type; } +#ifdef CONFIG_IPIPE +void ipipe_mach_allow_hwtimer_uaccess(unsigned long aips1, unsigned long aips2) +{ + volatile unsigned long aips_reg; + + /* + * S/W workaround: Clear the off platform peripheral modules + * Supervisor Protect bit for SDMA to access them. + */ + __raw_writel(0x0, aips1 + 0x40); + __raw_writel(0x0, aips1 + 0x44); + __raw_writel(0x0, aips1 + 0x48); + __raw_writel(0x0, aips1 + 0x4C); + aips_reg = __raw_readl(aips1 + 0x50); + aips_reg &= 0x00FFFFFF; + __raw_writel(aips_reg, aips1 + 0x50); + + __raw_writel(0x0, aips2 + 0x40); + __raw_writel(0x0, aips2 + 0x44); + __raw_writel(0x0, aips2 + 0x48); + __raw_writel(0x0, aips2 + 0x4C); + aips_reg = __raw_readl(aips2 + 0x50); + aips_reg &= 0x00FFFFFF; + __raw_writel(aips_reg, aips2 + 0x50); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-mxc/gpio.c b/arch/arm/plat-mxc/gpio.c index d65ebe3..2eedd79 100644 --- a/arch/arm/plat-mxc/gpio.c +++ b/arch/arm/plat-mxc/gpio.c @@ -23,6 +23,7 @@ #include <linux/io.h> #include <linux/irq.h> #include <linux/gpio.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm-generic/bug.h> @@ -174,8 +175,7 @@ static void mxc_gpio_irq_handler(struct mxc_gpio_port *port, u32 irq_stat) if (port->both_edges & (1 << (gpio & 31))) mxc_flip_edge(port, gpio); - irq_desc[gpio_irq_no].handle_irq(gpio_irq_no, - &irq_desc[gpio_irq_no]); + ipipe_handle_irq_cond(gpio_irq_no); } } diff --git a/arch/arm/plat-mxc/include/mach/common.h b/arch/arm/plat-mxc/include/mach/common.h index 4bf1068..1cf801f 100644 --- a/arch/arm/plat-mxc/include/mach/common.h +++ b/arch/arm/plat-mxc/include/mach/common.h @@ -45,4 +45,8 @@ extern void mxc91231_power_off(void); extern void mxc91231_arch_reset(int, const char *); extern void mxc91231_prepare_idle(void); +#ifdef CONFIG_IPIPE +void ipipe_mach_allow_hwtimer_uaccess(unsigned long aips1, unsigned long aips2); +#endif + #endif diff --git a/arch/arm/plat-mxc/time.c b/arch/arm/plat-mxc/time.c index 844567e..bc60cac 100644 --- a/arch/arm/plat-mxc/time.c +++ b/arch/arm/plat-mxc/time.c @@ -25,6 +25,7 @@ #include <linux/irq.h> #include <linux/clockchips.h> #include <linux/clk.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach/time.h> @@ -57,6 +58,72 @@ #define MX3_TCN 0x24 #define MX3_TCMP 0x10 +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int mxc_timer_initialized; +static unsigned mxc_min_delay; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + if (cpu_is_mx1()) { +#ifdef CONFIG_ARCH_MX1 + info->u.fr.counter = (unsigned *) (TIM1_BASE_ADDR + MX1_2_TCN); +#endif + } else if (cpu_is_mx2()) { +#ifdef CONFIG_ARCH_MX2 + info->u.fr.counter = (unsigned *) (GPT1_BASE_ADDR + MX1_2_TCN); +#endif + } else if (cpu_is_mx3()) { +#ifdef CONFIG_ARCH_MX3 + info->u.fr.counter = (unsigned *) (GPT1_BASE_ADDR + MX3_TCN); +#endif + } + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); +#endif /* CONFIG_IPIPE */ + static struct clock_event_device clockevent_mxc; static enum clock_event_mode clockevent_mode = CLOCK_EVT_MODE_UNUSED; @@ -120,6 +187,9 @@ static int __init mxc_clocksource_init(struct clk *timer_clk) if (cpu_is_mx3() || cpu_is_mx25()) clocksource_mxc.read = mx3_get_cycles; +#ifdef CONFIG_IPIPE + __ipipe_mach_ticks_per_jiffy = (c + HZ/2) / HZ; +#endif /* CONFIG_IPIPE */ clocksource_mxc.mult = clocksource_hz2mult(c, clocksource_mxc.shift); clocksource_register(&clocksource_mxc); @@ -238,7 +308,11 @@ static irqreturn_t mxc_timer_interrupt(int irq, void *dev_id) else tstat = __raw_readl(timer_base + MX1_2_TSTAT); +#ifndef CONFIG_IPIPE gpt_irq_acknowledge(); +#else /* !CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* !CONFIG_IPIPE */ evt->event_handler(evt); @@ -309,4 +383,116 @@ void __init mxc_timer_init(struct clk *timer_clk, void __iomem *base, int irq) /* Make irqs happen */ setup_irq(irq, &mxc_timer_irq); + +#ifdef CONFIG_IPIPE + __ipipe_mach_timerint = irq; +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + mxc_min_delay = ((ipipe_cpu_freq() + 500000) / 1000000) ?: 1; + mxc_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ +} + +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_mxc.name); +} + +void __ipipe_mach_acktimer(void) +{ + gpt_irq_acknowledge(); +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + if (!cpu_is_mx3()) + stamp = __raw_readl(timer_base + MX1_2_TCN); + else + stamp = __raw_readl(timer_base + MX3_TCN); + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(mxc_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + if (!cpu_is_mx3()) { + __asm__ ("ldmia %1, %M0\n" + : "=r"(result.full), "+&r"(local_tsc) + : "m"(*local_tsc)); + barrier(); + stamp = __raw_readl(timer_base + MX1_2_TCN); + } else { + __asm__ ("ldmia %1, %M0\n" + : "=r"(result.full), "+&r"(local_tsc) + : "m"(*local_tsc)); + barrier(); + stamp = __raw_readl(timer_base + MX3_TCN); + } + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > mxc_min_delay) { + unsigned long flags; + + local_irq_save_hw(flags); + if (!cpu_is_mx3()) + mx1_2_set_next_event(delay, NULL); + else + mx3_set_next_event(delay, NULL); + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(__ipipe_mach_timerint); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + mxc_set_mode(clockevent_mxc.mode, &clockevent_mxc); + if (clockevent_mxc.mode == CLOCK_EVT_MODE_ONESHOT) + clockevent_mxc.set_next_event(LATCH, &clockevent_mxc); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + if (!cpu_is_mx3()) + return __raw_readl(timer_base + MX1_2_TCMP) + - __raw_readl(timer_base + MX1_2_TCN); + else + return __raw_readl(timer_base + MX3_TCMP) + - __raw_readl(timer_base + MX3_TCN); } +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-omap/Kconfig b/arch/arm/plat-omap/Kconfig index e2ea04a..7756c92 100644 --- a/arch/arm/plat-omap/Kconfig +++ b/arch/arm/plat-omap/Kconfig @@ -114,6 +114,7 @@ config OMAP_MPU_TIMER timer provides more intra-tick resolution than the 32KHz timer, but consumes more power. +if !IPIPE config OMAP_32K_TIMER bool "Use 32KHz timer" depends on ARCH_OMAP16XX || ARCH_OMAP24XX || ARCH_OMAP34XX || ARCH_OMAP4 @@ -123,6 +124,7 @@ config OMAP_32K_TIMER support for no tick during idle. The 32KHz timer provides less intra-tick resolution than OMAP_MPU_TIMER. The 32KHz timer is currently only available for OMAP16XX, 24XX, 34XX and OMAP4. +endif endchoice diff --git a/arch/arm/plat-omap/dmtimer.c b/arch/arm/plat-omap/dmtimer.c index 08ccf89..4d1abd7 100644 --- a/arch/arm/plat-omap/dmtimer.c +++ b/arch/arm/plat-omap/dmtimer.c @@ -475,6 +475,13 @@ int omap_dm_timer_get_irq(struct omap_dm_timer *timer) } EXPORT_SYMBOL_GPL(omap_dm_timer_get_irq); +#ifdef CONFIG_IPIPE +unsigned long omap_dm_timer_get_phys_counter_addr(struct omap_dm_timer *timer) +{ + return timer->phys_base + (OMAP_TIMER_COUNTER_REG & 0xff); +} +#endif /* CONFIG_IPIPE */ + #if defined(CONFIG_ARCH_OMAP1) /** diff --git a/arch/arm/plat-omap/gpio.c b/arch/arm/plat-omap/gpio.c index d2422c7..589a4bb 100644 --- a/arch/arm/plat-omap/gpio.c +++ b/arch/arm/plat-omap/gpio.c @@ -20,7 +20,9 @@ #include <linux/sysdev.h> #include <linux/err.h> #include <linux/clk.h> +#include <linux/irq.h> /* For irq_desc */ #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -1345,8 +1347,7 @@ static void gpio_irq_handler(unsigned int irq, struct irq_desc *desc) if (bank->toggle_mask & (1 << gpio_index)) _toggle_gpio_edge_triggering(bank, gpio_index); #endif - - generic_handle_irq(gpio_irq); + ipipe_handle_irq_cond(gpio_irq); } } /* if bank has any level sensitive GPIO pin interrupt diff --git a/arch/arm/plat-omap/include/plat/dmtimer.h b/arch/arm/plat-omap/include/plat/dmtimer.h index 20f1054..42a0b8d 100644 --- a/arch/arm/plat-omap/include/plat/dmtimer.h +++ b/arch/arm/plat-omap/include/plat/dmtimer.h @@ -56,6 +56,9 @@ void omap_dm_timer_enable(struct omap_dm_timer *timer); void omap_dm_timer_disable(struct omap_dm_timer *timer); int omap_dm_timer_get_irq(struct omap_dm_timer *timer); +#ifdef CONFIG_IPIPE +unsigned long omap_dm_timer_get_phys_counter_addr(struct omap_dm_timer *timer); +#endif /* CONFIG_IPIPE */ u32 omap_dm_timer_modify_idlect_mask(u32 inputmask); struct clk *omap_dm_timer_get_fclk(struct omap_dm_timer *timer); diff --git a/arch/arm/plat-pxa/gpio.c b/arch/arm/plat-pxa/gpio.c index 98548c6..e732e45 100644 --- a/arch/arm/plat-pxa/gpio.c +++ b/arch/arm/plat-pxa/gpio.c @@ -17,6 +17,7 @@ #include <linux/io.h> #include <linux/sysdev.h> #include <linux/slab.h> +#include <linux/ipipe.h> #include <mach/gpio.h> @@ -220,7 +221,7 @@ static void pxa_gpio_demux_handler(unsigned int irq, struct irq_desc *desc) while (n < BITS_PER_LONG) { loop = 1; - generic_handle_irq(gpio_to_irq(gpio_base + n)); + ipipe_handle_irq_cond(gpio_to_irq(gpio_base + n)); n = find_next_bit(&gedr, BITS_PER_LONG, n + 1); } } @@ -285,7 +286,7 @@ void __init pxa_init_gpio(int mux_irq, int start, int end, set_wake_t fn) for (irq = gpio_to_irq(start); irq <= gpio_to_irq(end); irq++) { set_irq_chip(irq, &pxa_muxed_gpio_chip); - set_irq_handler(irq, handle_edge_irq); + set_irq_handler(irq, handle_level_irq); set_irq_flags(irq, IRQF_VALID | IRQF_PROBE); } diff --git a/arch/arm/plat-s3c/time.c b/arch/arm/plat-s3c/time.c index 3b27b29..976fe03 100644 --- a/arch/arm/plat-s3c/time.c +++ b/arch/arm/plat-s3c/time.c @@ -3,6 +3,8 @@ * Copyright (C) 2003-2005 Simtec Electronics * Ben Dooks, <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -27,6 +29,7 @@ #include <linux/clk.h> #include <linux/io.h> #include <linux/platform_device.h> +#include <linux/module.h> #include <asm/system.h> #include <asm/leds.h> @@ -42,7 +45,6 @@ #include <plat/clock.h> #include <plat/cpu.h> -static unsigned long timer_startval; static unsigned long timer_usec_ticks; #ifndef TICK_MAX @@ -61,6 +63,39 @@ static unsigned long timer_usec_ticks; * Original patch by Dimitry Andric, updated by Ben Dooks */ +static unsigned long last_free_running_tcnt = 0; +static unsigned long free_running_tcon = 0; +static unsigned long timer_lxlost = 0; + +#ifdef CONFIG_IPIPE + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ + +unsigned int __ipipe_mach_ticks_per_jiffy; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +int __ipipe_mach_timerint = IRQ_TIMER4; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +static unsigned long long *tsc; +static unsigned *last_cnt; +static unsigned long timer_ackval = 1UL << (IRQ_TIMER4 - IRQ_EINT0); +static IPIPE_DEFINE_SPINLOCK(timer_lock); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_DECREMENTER; + info->u.dec.counter = (unsigned *)0x51000038; + info->u.dec.mask = 0xffff; + info->u.dec.last_cnt = last_cnt; + info->u.dec.tsc = tsc; +} +#endif /* CONFIG_IPIPE */ /* timer_mask_usec_ticks * @@ -91,38 +126,45 @@ static inline unsigned long timer_ticks_to_usec(unsigned long ticks) return res >> TIMER_USEC_SHIFT; } -/*** - * Returns microsecond since last clock interrupt. Note that interrupts - * will have been disabled by do_gettimeoffset() - * IRQs are disabled before entering here from do_gettimeofday() - */ - -static unsigned long s3c2410_gettimeoffset (void) +static inline unsigned long timer_freerunning_getvalue(void) { - unsigned long tdone; - unsigned long tval; - - /* work out how many ticks have gone since last timer interrupt */ + return __raw_readl(S3C2410_TCNTO(3)); +} - tval = __raw_readl(S3C2410_TCNTO(4)); - tdone = timer_startval - tval; +static inline unsigned long timer_freerunning_getticksoffset(unsigned long tval) +{ + long tdone; - /* check to see if there is an interrupt pending */ + tdone = last_free_running_tcnt - tval; + if (tdone < 0) + tdone += 0x10000; - if (s3c24xx_ostimer_pending()) { - /* re-read the timer, and try and fix up for the missed - * interrupt. Note, the interrupt may go off before the - * timer has re-loaded from wrapping. - */ + return tdone; +} - tval = __raw_readl(S3C2410_TCNTO(4)); - tdone = timer_startval - tval; +static inline unsigned long getticksoffset(void) +{ + return timer_freerunning_getticksoffset(timer_freerunning_getvalue()); +} - if (tval != 0) - tdone += timer_startval; - } +#ifdef CONFIG_IPIPE +static inline unsigned long getticksoffset_tscupdate(void) +{ + unsigned long tval; + unsigned long ticks; + + tval = timer_freerunning_getvalue(); + ticks = timer_freerunning_getticksoffset(tval); + last_free_running_tcnt = tval; + *tsc += ticks; + *last_cnt = last_free_running_tcnt; + return ticks; +} +#endif /* CONFIG_IPIPE */ - return timer_ticks_to_usec(tdone); +static unsigned long s3c2410_gettimeoffset (void) +{ + return timer_ticks_to_usec(timer_lxlost + getticksoffset()); } @@ -132,6 +174,13 @@ static unsigned long s3c2410_gettimeoffset (void) static irqreturn_t s3c2410_timer_interrupt(int irq, void *dev_id) { +#ifdef CONFIG_IPIPE + timer_lxlost = 0; + + if (!__ipipe_mach_timerstolen) + getticksoffset_tscupdate(); +#endif /* CONFIG_IPIPE */ + timer_tick(); return IRQ_HANDLED; } @@ -153,10 +202,10 @@ static struct clk *tdiv; static struct clk *timerclk; /* - * Set up timer interrupt, and return the current time in seconds. + * Set up timer interrupt. * - * Currently we only use timer4, as it is the only timer which has no - * other function that can be exploited externally + * Currently we use timer4 as event timer and timer3 as tick counter which + * permanently counts ticks without interrupt generation. */ static void s3c2410_timer_setup (void) { @@ -164,6 +213,7 @@ static void s3c2410_timer_setup (void) unsigned long tcnt; unsigned long tcfg1; unsigned long tcfg0; + unsigned long intmask; tcnt = TICK_MAX; /* default value for tcnt */ @@ -175,8 +225,8 @@ static void s3c2410_timer_setup (void) tcnt = 12000000 / HZ; tcfg1 = __raw_readl(S3C2410_TCFG1); - tcfg1 &= ~S3C2410_TCFG1_MUX4_MASK; - tcfg1 |= S3C2410_TCFG1_MUX4_TCLK1; + tcfg1 &= ~(S3C2410_TCFG1_MUX4_MASK | S3C2410_TCFG1_MUX3_MASK); + tcfg1 |= (S3C2410_TCFG1_MUX4_TCLK1 | S3C2410_TCFG1_MUX3_TCLK1); __raw_writel(tcfg1, S3C2410_TCFG1); } else { unsigned long pclk; @@ -210,6 +260,14 @@ static void s3c2410_timer_setup (void) tcfg0 = __raw_readl(S3C2410_TCFG0); tcfg1 = __raw_readl(S3C2410_TCFG1); +#ifdef CONFIG_IPIPE + tsc = (unsigned long long *)__ipipe_tsc_area; + last_cnt = (unsigned *)(tsc + 1); /* means: +8 bytes */ + barrier(); + + __ipipe_mach_ticks_per_jiffy = tcnt; +#endif /* CONFIG_IPIPE */ + /* timers reload after counting zero, so reduce the count by 1 */ tcnt--; @@ -226,23 +284,37 @@ static void s3c2410_timer_setup (void) __raw_writel(tcfg1, S3C2410_TCFG1); __raw_writel(tcfg0, S3C2410_TCFG0); - timer_startval = tcnt; - __raw_writel(tcnt, S3C2410_TCNTB(4)); - - /* ensure timer is stopped... */ + /* ensure timers are stopped... */ + tcon &= ~(0x3f<<17); + __raw_writel(tcon, S3C2410_TCON); - tcon &= ~(7<<20); - tcon |= S3C2410_TCON_T4RELOAD; - tcon |= S3C2410_TCON_T4MANUALUPD; + /* Mask timer3 interrupt. */ + intmask = __raw_readl(S3C2410_INTMSK); + intmask |= 1UL << (IRQ_TIMER3 - IRQ_EINT0); + __raw_writel(intmask, S3C2410_INTMSK); - __raw_writel(tcon, S3C2410_TCON); + /* Set timer values */ __raw_writel(tcnt, S3C2410_TCNTB(4)); __raw_writel(tcnt, S3C2410_TCMPB(4)); + __raw_writel(0xffff, S3C2410_TCNTB(3)); + __raw_writel(0xffff, S3C2410_TCMPB(3)); + + /* Set base tcon value for later programming of timer 4 by Xenomai. */ + free_running_tcon = tcon | S3C2410_TCON_T3RELOAD | S3C2410_TCON_T3START; + + /* Set auto reloads for both timers. */ + tcon |= S3C2410_TCON_T3RELOAD | S3C2410_TCON_T4RELOAD; - /* start the timer running */ - tcon |= S3C2410_TCON_T4START; - tcon &= ~S3C2410_TCON_T4MANUALUPD; + /* Manual update */ + __raw_writel(tcon | S3C2410_TCON_T3MANUALUPD + | S3C2410_TCON_T4MANUALUPD, S3C2410_TCON); + + tcon |= S3C2410_TCON_T3START | S3C2410_TCON_T4START; + /* Start timers.*/ __raw_writel(tcon, S3C2410_TCON); + + /* Save start value of timer 3 as begining of first period. */ + last_free_running_tcnt = 0xffff; } static void __init s3c2410_timer_resources(void) @@ -283,3 +355,58 @@ struct sys_timer s3c24xx_timer = { .offset = s3c2410_gettimeoffset, .resume = s3c2410_timer_setup }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + __raw_writel(timer_ackval, S3C2410_SRCPND); + __raw_writel(timer_ackval, S3C2410_INTPND); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + unsigned long long result; + unsigned long flags; + + local_irq_save_hw_notrace(flags); + spin_lock(&timer_lock); + result = *tsc + getticksoffset(); + spin_unlock(&timer_lock); + local_irq_restore_hw_notrace(flags); + return result; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +static inline void set_dec(unsigned long reload) +{ + __raw_writel(reload, S3C2410_TCNTB(4)); + /* Manual update */ + __raw_writel(free_running_tcon | S3C2410_TCON_T4MANUALUPD, S3C2410_TCON); + /* Start timer */ + __raw_writel(free_running_tcon | S3C2410_TCON_T4START, S3C2410_TCON); +} + +void __ipipe_mach_set_dec(unsigned long reload) +{ + unsigned long flags; + + spin_lock_irqsave(&timer_lock, flags); + timer_lxlost += getticksoffset_tscupdate(); + set_dec(reload); + spin_unlock_irqrestore(&timer_lock, flags); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + free_running_tcon |= S3C2410_TCON_T4RELOAD; + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy - 1); + free_running_tcon &= ~S3C2410_TCON_T4RELOAD; +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return __raw_readl(S3C2410_TCNTO(4)); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-s3c24xx/irq.c b/arch/arm/plat-s3c24xx/irq.c index ad0d44e..8444c8b 100644 --- a/arch/arm/plat-s3c24xx/irq.c +++ b/arch/arm/plat-s3c24xx/irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -23,6 +25,7 @@ #include <linux/interrupt.h> #include <linux/ioport.h> #include <linux/sysdev.h> +#include <linux/ipipe.h> #include <asm/irq.h> #include <asm/mach/irq.h> @@ -87,6 +90,9 @@ struct irq_chip s3c_irq_level_chip = { .name = "s3c-level", .ack = s3c_irq_maskack, .mask = s3c_irq_mask, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_maskack, +#endif /* CONFIG_IPIPE */ .unmask = s3c_irq_unmask, .set_wake = s3c_irq_wake }; @@ -283,6 +289,9 @@ static struct irq_chip s3c_irq_uart0 = { .mask = s3c_irq_uart0_mask, .unmask = s3c_irq_uart0_unmask, .ack = s3c_irq_uart0_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart0_ack, +#endif /* CONFIG_IPIPE */ }; /* UART1 */ @@ -310,6 +319,9 @@ static struct irq_chip s3c_irq_uart1 = { .mask = s3c_irq_uart1_mask, .unmask = s3c_irq_uart1_unmask, .ack = s3c_irq_uart1_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart1_ack, +#endif /* CONFIG_IPIPE */ }; /* UART2 */ @@ -337,6 +349,9 @@ static struct irq_chip s3c_irq_uart2 = { .mask = s3c_irq_uart2_mask, .unmask = s3c_irq_uart2_unmask, .ack = s3c_irq_uart2_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart2_ack, +#endif /* CONFIG_IPIPE */ }; /* ADC and Touchscreen */ @@ -385,10 +400,10 @@ static void s3c_irq_demux_adc(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_TC); + ipipe_handle_irq_cond(IRQ_TC); } if (subsrc & 2) { - generic_handle_irq(IRQ_ADC); + ipipe_handle_irq_cond(IRQ_ADC); } } } @@ -413,13 +428,13 @@ static void s3c_irq_demux_uart(unsigned int start) if (subsrc != 0) { if (subsrc & 1) - generic_handle_irq(start); + ipipe_handle_irq_cond(start); if (subsrc & 2) - generic_handle_irq(start+1); + ipipe_handle_irq_cond(start+1); if (subsrc & 4) - generic_handle_irq(start+2); + ipipe_handle_irq_cond(start+2); } } @@ -466,7 +481,7 @@ s3c_irq_demux_extint8(unsigned int irq, eintpnd &= ~(1<<irq); irq += (IRQ_EINT4 - 4); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } } @@ -489,7 +504,7 @@ s3c_irq_demux_extint4t7(unsigned int irq, irq += (IRQ_EINT4 - 4); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } } diff --git a/arch/arm/plat-s3c24xx/s3c244x-irq.c b/arch/arm/plat-s3c24xx/s3c244x-irq.c index a75c0c2..5dd2dd1 100644 --- a/arch/arm/plat-s3c24xx/s3c244x-irq.c +++ b/arch/arm/plat-s3c24xx/s3c244x-irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -25,6 +27,7 @@ #include <linux/ioport.h> #include <linux/sysdev.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -57,10 +60,10 @@ static void s3c_irq_demux_cam(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_S3C2440_CAM_C); + ipipe_handle_irq_cond(IRQ_S3C2440_CAM_C); } if (subsrc & 2) { - generic_handle_irq(IRQ_S3C2440_CAM_P); + ipipe_handle_irq_cond(IRQ_S3C2440_CAM_P); } } } @@ -89,6 +92,9 @@ static struct irq_chip s3c_irq_cam = { .mask = s3c_irq_cam_mask, .unmask = s3c_irq_cam_unmask, .ack = s3c_irq_cam_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_cam_ack, +#endif /* CONFIG_IPIPE */ }; static int s3c244x_irq_add(struct sys_device *sysdev) diff --git a/arch/arm/vfp/entry.S b/arch/arm/vfp/entry.S index 4fa9903..e46bae3 100644 --- a/arch/arm/vfp/entry.S +++ b/arch/arm/vfp/entry.S @@ -25,7 +25,6 @@ ENTRY(do_vfp) add r11, r4, #1 @ increment it str r11, [r10, #TI_PREEMPT] #endif - enable_irq ldr r4, .LCvfp ldr r11, [r10, #TI_CPU] @ CPU number add r10, r10, #TI_VFPSTATE @ r10 = workspace @@ -33,6 +32,7 @@ ENTRY(do_vfp) ENDPROC(do_vfp) ENTRY(vfp_null_entry) + enable_irq #ifdef CONFIG_PREEMPT get_thread_info r10 ldr r4, [r10, #TI_PREEMPT] @ get preempt count @@ -51,6 +51,7 @@ ENDPROC(vfp_null_entry) __INIT ENTRY(vfp_testing_entry) + enable_irq #ifdef CONFIG_PREEMPT get_thread_info r10 ldr r4, [r10, #TI_PREEMPT] @ get preempt count diff --git a/arch/arm/vfp/vfphw.S b/arch/arm/vfp/vfphw.S index 66dc2d0..b1b9c31 100644 --- a/arch/arm/vfp/vfphw.S +++ b/arch/arm/vfp/vfphw.S @@ -19,7 +19,7 @@ #include "../kernel/entry-header.S" .macro DBGSTR, str -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} add r0, pc, #4 bl printk @@ -31,7 +31,7 @@ .endm .macro DBGSTR1, str, arg -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} mov r1, \arg add r0, pc, #4 @@ -44,7 +44,7 @@ .endm .macro DBGSTR3, str, arg1, arg2, arg3 -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} mov r3, \arg3 mov r2, \arg2 @@ -87,6 +87,11 @@ ENTRY(vfp_support_entry) @ still there. In this case, we do @ not want to drop a pending exception. + enable_irq +#ifdef CONFIG_IPIPE + disable_irq + ldr r4, [r3, r11, lsl #2] @ reload last_VFP_context pointer +#endif VFPFMXR FPEXC, r5 @ enable VFP, disable any pending @ exceptions, so we can get at the @ rest of it @@ -139,6 +144,7 @@ check_for_exception: @ out before setting an FPEXC that @ stops us reading stuff VFPFMXR FPEXC, r1 @ restore FPEXC last + enable_irq_cond sub r2, r2, #4 str r2, [sp, #S_PC] @ retry the instruction #ifdef CONFIG_PREEMPT @@ -164,6 +170,7 @@ look_for_VFP_exceptions: @ Fall into hand on to next handler - appropriate coproc instr @ not recognised by VFP + enable_irq_cond DBGSTR "not VFP" #ifdef CONFIG_PREEMPT get_thread_info r10 diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index a63c4be..1fdff52 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -44,6 +44,7 @@ unsigned int VFP_arch; static void vfp_thread_flush(struct thread_info *thread) { union vfp_state *vfp = &thread->vfpstate; + unsigned long flags; unsigned int cpu; memset(vfp, 0, sizeof(union vfp_state)); @@ -56,22 +57,23 @@ static void vfp_thread_flush(struct thread_info *thread) * that the modification of last_VFP_context[] and hardware disable * are done for the same CPU and without preemption. */ - cpu = get_cpu(); + cpu = ipipe_get_cpu(flags); if (last_VFP_context[cpu] == vfp) last_VFP_context[cpu] = NULL; fmxr(FPEXC, fmrx(FPEXC) & ~FPEXC_EN); - put_cpu(); + ipipe_put_cpu(flags); } static void vfp_thread_exit(struct thread_info *thread) { /* release case: Per-thread VFP cleanup. */ union vfp_state *vfp = &thread->vfpstate; - unsigned int cpu = get_cpu(); + unsigned long flags; + unsigned int cpu = ipipe_get_cpu(flags); if (last_VFP_context[cpu] == vfp) last_VFP_context[cpu] = NULL; - put_cpu(); + ipipe_put_cpu(flags); } /* @@ -100,10 +102,13 @@ static void vfp_thread_exit(struct thread_info *thread) static int vfp_notifier(struct notifier_block *self, unsigned long cmd, void *v) { struct thread_info *thread = v; + unsigned long flags; if (likely(cmd == THREAD_NOTIFY_SWITCH)) { - u32 fpexc = fmrx(FPEXC); + u32 fpexc; + local_irq_save_hw_cond(flags); + fpexc = fmrx(FPEXC); #ifdef CONFIG_SMP unsigned int cpu = thread->cpu; @@ -130,6 +135,7 @@ static int vfp_notifier(struct notifier_block *self, unsigned long cmd, void *v) * old state. */ fmxr(FPEXC, fpexc & ~FPEXC_EN); + local_irq_restore_hw_cond(flags); return NOTIFY_DONE; } @@ -266,7 +272,7 @@ static u32 vfp_emulate_instruction(u32 inst, u32 fpscr, struct pt_regs *regs) */ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) { - u32 fpscr, orig_fpscr, fpsid, exceptions; + u32 fpscr, orig_fpscr, fpsid, exceptions, next_trigger = 0; pr_debug("VFP: bounce: trigger %08x fpexc %08x\n", trigger, fpexc); @@ -296,6 +302,7 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) /* * Synchronous exception, emulate the trigger instruction */ + local_irq_enable_hw_cond(); goto emulate; } @@ -308,13 +315,24 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) trigger = fmrx(FPINST); regs->ARM_pc -= 4; #endif - } else if (!(fpexc & FPEXC_DEX)) { + if (fpexc & FPEXC_FP2V) { + /* + * The barrier() here prevents fpinst2 being read + * before the condition above. + */ + barrier(); + next_trigger = fmrx(FPINST2); + } + } + local_irq_enable_hw_cond(); + + if (!(fpexc & (FPEXC_EX | FPEXC_DEX))) { /* * Illegal combination of bits. It can be caused by an * unallocated VFP instruction but with FPSCR.IXE set and not * on VFP subarch 1. */ - vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs); + vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs); goto exit; } @@ -348,18 +366,14 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) if (fpexc ^ (FPEXC_EX | FPEXC_FP2V)) goto exit; - /* - * The barrier() here prevents fpinst2 being read - * before the condition above. - */ - barrier(); - trigger = fmrx(FPINST2); + trigger = next_trigger; emulate: exceptions = vfp_emulate_instruction(trigger, orig_fpscr, regs); if (exceptions) vfp_raise_exceptions(exceptions, trigger, orig_fpscr, regs); exit: + local_irq_enable_hw_cond(); preempt_enable(); } diff --git a/drivers/i2c/busses/i2c-omap.c b/drivers/i2c/busses/i2c-omap.c index 0037e31..8d608a1 100644 --- a/drivers/i2c/busses/i2c-omap.c +++ b/drivers/i2c/busses/i2c-omap.c @@ -478,6 +478,10 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, if (msg->len == 0) return -EINVAL; +#ifdef CONFIG_IPIPE + disable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ + omap_i2c_write_reg(dev, OMAP_I2C_SA_REG, msg->addr); /* REVISIT: Could the STB bit of I2C_CON be used with probing? */ @@ -521,6 +525,9 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, /* Let the user know if i2c is in a bad state */ if (time_after(jiffies, delay)) { +#ifdef CONFIG_IPIPE + enable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ dev_err(dev->dev, "controller timed out " "waiting for start condition to finish\n"); return -ETIMEDOUT; @@ -532,6 +539,9 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, w &= ~OMAP_I2C_CON_STT; omap_i2c_write_reg(dev, OMAP_I2C_CON_REG, w); } +#ifdef CONFIG_IPIPE + enable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ /* * REVISIT: We should abort the transfer on signals, but the bus goes diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index e3551d2..d8f7317 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -46,7 +46,7 @@ config ATMEL_TCLIB config ATMEL_TCB_CLKSRC bool "TC Block Clocksource" - depends on ATMEL_TCLIB && GENERIC_TIME + depends on ATMEL_TCLIB && GENERIC_TIME && !IPIPE default y help Select this to get a high precision clocksource based on a diff --git a/drivers/pci/htirq.c b/drivers/pci/htirq.c index 737a1c4..15e81de 100644 --- a/drivers/pci/htirq.c +++ b/drivers/pci/htirq.c @@ -21,7 +21,7 @@ * With multiple simultaneous hypertransport irq devices it might pay * to make this more fine grained. But start with simple, stupid, and correct. */ -static DEFINE_SPINLOCK(ht_irq_lock); +static IPIPE_DEFINE_SPINLOCK(ht_irq_lock); struct ht_irq_cfg { struct pci_dev *dev; diff --git a/drivers/serial/8250.c b/drivers/serial/8250.c index e9b15c3..f93771a 100644 --- a/drivers/serial/8250.c +++ b/drivers/serial/8250.c @@ -3020,6 +3020,51 @@ static int serial8250_resume(struct platform_device *dev) return 0; } +#if defined(CONFIG_IPIPE_DEBUG) && defined(CONFIG_SERIAL_8250_CONSOLE) + +#include <stdarg.h> + +void __weak __ipipe_serial_debug(const char *fmt, ...) +{ + struct uart_8250_port *up = &serial8250_ports[0]; + unsigned int ier, count; + unsigned long flags; + char buf[128]; + va_list ap; + + va_start(ap, fmt); + vsprintf(buf, fmt, ap); + va_end(ap); + count = strlen(buf); + + touch_nmi_watchdog(); + + local_irq_save_hw(flags); + + /* + * First save the IER then disable the interrupts + */ + ier = serial_in(up, UART_IER); + + if (up->capabilities & UART_CAP_UUE) + serial_out(up, UART_IER, UART_IER_UUE); + else + serial_out(up, UART_IER, 0); + + uart_console_write(&up->port, buf, count, serial8250_console_putchar); + + /* + * Finally, wait for transmitter to become empty + * and restore the IER + */ + wait_for_xmitr(up, BOTH_EMPTY); + serial_out(up, UART_IER, ier); + + local_irq_restore_hw(flags); +} + +#endif + static struct platform_driver serial8250_isa_driver = { .probe = serial8250_probe, .remove = __devexit_p(serial8250_remove), diff --git a/fs/exec.c b/fs/exec.c index cce6bbd..1f2021d 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -714,6 +714,7 @@ static int exec_mmap(struct mm_struct *mm) { struct task_struct *tsk; struct mm_struct * old_mm, *active_mm; + unsigned long flags; /* Notify parent that we're no longer interested in the old VM */ tsk = current; @@ -736,8 +737,10 @@ static int exec_mmap(struct mm_struct *mm) task_lock(tsk); active_mm = tsk->active_mm; tsk->mm = mm; + ipipe_mm_switch_protect(flags); tsk->active_mm = mm; activate_mm(active_mm, mm); + ipipe_mm_switch_unprotect(flags); task_unlock(tsk); arch_pick_mmap_layout(mm); if (old_mm) { diff --git a/fs/proc/array.c b/fs/proc/array.c index 13b5d07..e2c2b55 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -144,6 +144,10 @@ static const char *task_state_array[] = { "x (dead)", /* 64 */ "K (wakekill)", /* 128 */ "W (waking)", /* 256 */ +#ifdef CONFIG_IPIPE + "A (atomic switch)", /* 512 */ + "N (wakeup disabled)", /* 1024 */ +#endif }; static inline const char *get_task_state(struct task_struct *tsk) diff --git a/include/asm-generic/atomic.h b/include/asm-generic/atomic.h index c99c64d..5d01b93 100644 --- a/include/asm-generic/atomic.h +++ b/include/asm-generic/atomic.h @@ -60,11 +60,11 @@ static inline int atomic_add_return(int i, atomic_t *v) unsigned long flags; int temp; - local_irq_save(flags); + local_irq_save_hw(flags); temp = v->counter; temp += i; v->counter = temp; - local_irq_restore(flags); + local_irq_restore_hw(flags); return temp; } @@ -82,11 +82,11 @@ static inline int atomic_sub_return(int i, atomic_t *v) unsigned long flags; int temp; - local_irq_save(flags); + local_irq_save_hw(flags); temp = v->counter; temp -= i; v->counter = temp; - local_irq_restore(flags); + local_irq_restore_hw(flags); return temp; } @@ -139,9 +139,9 @@ static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) unsigned long flags; mask = ~mask; - local_irq_save(flags); + local_irq_save_hw(flags); *addr &= mask; - local_irq_restore(flags); + local_irq_restore_hw(flags); } #define atomic_xchg(ptr, v) (xchg(&(ptr)->counter, (v))) diff --git a/include/asm-generic/bitops/atomic.h b/include/asm-generic/bitops/atomic.h index ecc44a8..5caf6e9 100644 --- a/include/asm-generic/bitops/atomic.h +++ b/include/asm-generic/bitops/atomic.h @@ -21,20 +21,20 @@ extern arch_spinlock_t __atomic_hash[ATOMIC_HASH_SIZE] __lock_aligned; * this is the substitute */ #define _atomic_spin_lock_irqsave(l,f) do { \ arch_spinlock_t *s = ATOMIC_HASH(l); \ - local_irq_save(f); \ + local_irq_save_hw(f); \ arch_spin_lock(s); \ } while(0) #define _atomic_spin_unlock_irqrestore(l,f) do { \ arch_spinlock_t *s = ATOMIC_HASH(l); \ arch_spin_unlock(s); \ - local_irq_restore(f); \ + local_irq_restore_hw(f); \ } while(0) #else -# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save(f); } while (0) -# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore(f); } while (0) +# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save_hw(f); } while (0) +# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore_hw(f); } while (0) #endif /* diff --git a/include/asm-generic/cmpxchg-local.h b/include/asm-generic/cmpxchg-local.h index b2ba2fc..ed01ab9 100644 --- a/include/asm-generic/cmpxchg-local.h +++ b/include/asm-generic/cmpxchg-local.h @@ -20,7 +20,7 @@ static inline unsigned long __cmpxchg_local_generic(volatile void *ptr, if (size == 8 && sizeof(unsigned long) != 8) wrong_size_cmpxchg(ptr); - local_irq_save(flags); + local_irq_save_hw(flags); switch (size) { case 1: prev = *(u8 *)ptr; if (prev == old) @@ -41,7 +41,7 @@ static inline unsigned long __cmpxchg_local_generic(volatile void *ptr, default: wrong_size_cmpxchg(ptr); } - local_irq_restore(flags); + local_irq_restore_hw(flags); return prev; } @@ -54,11 +54,11 @@ static inline u64 __cmpxchg64_local_generic(volatile void *ptr, u64 prev; unsigned long flags; - local_irq_save(flags); + local_irq_save_hw(flags); prev = *(u64 *)ptr; if (prev == old) *(u64 *)ptr = new; - local_irq_restore(flags); + local_irq_restore_hw(flags); return prev; } diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h index 8087b90..d5c9897 100644 --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -59,6 +59,20 @@ extern unsigned long __per_cpu_offset[NR_CPUS]; #define this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) #define __this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset) +#ifdef CONFIG_IPIPE +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) +extern int __ipipe_check_percpu_access(void); +#define __ipipe_local_cpu_offset \ + ({ \ + WARN_ON_ONCE(__ipipe_check_percpu_access()); \ + __my_cpu_offset; \ + }) +#else +#define __ipipe_local_cpu_offset __my_cpu_offset +#endif +#define __ipipe_get_cpu_var(var) \ + (*SHIFT_PERCPU_PTR(&per_cpu_var(var), __ipipe_local_cpu_offset)) +#endif /* CONFIG_IPIPE */ #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA extern void setup_per_cpu_areas(void); @@ -69,6 +83,7 @@ extern void setup_per_cpu_areas(void); #define per_cpu(var, cpu) (*((void)(cpu), &per_cpu_var(var))) #define __get_cpu_var(var) per_cpu_var(var) #define __raw_get_cpu_var(var) per_cpu_var(var) +#define __ipipe_get_cpu_var(var) __raw_get_cpu_var(var) #define this_cpu_ptr(ptr) per_cpu_ptr(ptr, 0) #define __this_cpu_ptr(ptr) this_cpu_ptr(ptr) diff --git a/include/asm-generic/resource.h b/include/asm-generic/resource.h index 587566f..66d60cd 100644 --- a/include/asm-generic/resource.h +++ b/include/asm-generic/resource.h @@ -58,6 +58,14 @@ #endif /* + * Limit the stack by to some sane default: root can always + * increase this limit if needed.. 8MB seems reasonable. + */ +#ifndef _STK_LIM +# define _STK_LIM (8*1024*1024) +#endif + +/* * RLIMIT_STACK default maximum - some architectures override it: */ #ifndef _STK_LIM_MAX diff --git a/include/linux/autoconf.h b/include/linux/autoconf.h new file mode 100644 index 0000000..9c633d6 --- /dev/null +++ b/include/linux/autoconf.h @@ -0,0 +1,552 @@ +/* + * Automatically generated C config: don't edit + * Linux kernel version: 2.6.32.7 + * Thu Feb 4 10:20:41 2010 + */ +#define AUTOCONF_INCLUDED +#define CONFIG_ACPI_AC 1 +#define CONFIG_ACPI_SYSFS_POWER 1 +#define CONFIG_DEBUG_SPINLOCK_SLEEP 1 +#define CONFIG_XENO_OPT_NATIVE_MPS 1 +#define CONFIG_FRAME_WARN 2048 +#define CONFIG_ARCH_DEFCONFIG "arch/x86/configs/x86_64_defconfig" +#define CONFIG_XENO_OPT_PRIOCPL 1 +#define CONFIG_SND_HDA_CODEC_CIRRUS 1 +#define CONFIG_TASK_XACCT 1 +#define CONFIG_POSIX_MQUEUE_SYSCTL 1 +#define CONFIG_ARCH_MAY_HAVE_PC_FDC 1 +#define CONFIG_CRYPTO_BLKCIPHER2 1 +#define CONFIG_CRYPTO_MD5 1 +#define CONFIG_X86_MCE_THRESHOLD 1 +#define CONFIG_PROC_KCORE 1 +#define CONFIG_XENO_OPT_TIMER_LIST 1 +#define CONFIG_CRYPTO_WORKQUEUE 1 +#define CONFIG_FTRACE_MCOUNT_RECORD 1 +#define CONFIG_CRYPTO_CBC 1 +#define CONFIG_TRACING 1 +#define CONFIG_DEVKMEM 1 +#define CONFIG_HAVE_FUNCTION_TRACER 1 +#define CONFIG_X86_PAT 1 +#define CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ 12 +#define CONFIG_XENO_OPT_NUCLEUS 1 +#define CONFIG_USE_GENERIC_SMP_HELPERS 1 +#define CONFIG_HAVE_KERNEL_BZIP2 1 +#define CONFIG_SND_SEQ_HRTIMER_DEFAULT 1 +#define CONFIG_SERIAL_8250_SHARE_IRQ 1 +#define CONFIG_PNPACPI 1 +#define CONFIG_TIMERFD 1 +#define CONFIG_ENABLE_MUST_CHECK 1 +#define CONFIG_EVENTFD 1 +#define CONFIG_X86_LOCAL_APIC 1 +#define CONFIG_NOP_TRACER 1 +#define CONFIG_BLK_DEV_SD 1 +#define CONFIG_HAS_DMA 1 +#define CONFIG_THERMAL 1 +#define CONFIG_X86_CPU 1 +#define CONFIG_EXT3_FS_XATTR 1 +#define CONFIG_NFS_COMMON 1 +#define CONFIG_X86_TSC 1 +#define CONFIG_HAVE_CPUMASK_OF_CPU_MAP 1 +#define CONFIG_SERIO_SERPORT 1 +#define CONFIG_SOUND_OSS_CORE_PRECLAIM 1 +#define CONFIG_HW_CONSOLE 1 +#define CONFIG_XENOMAI 1 +#define CONFIG_HPET_TIMER 1 +#define CONFIG_VGA_ARB 1 +#define CONFIG_CONSOLE_TRANSLATIONS 1 +#define CONFIG_ATA_SFF 1 +#define CONFIG_CHR_DEV_SG 1 +#define CONFIG_XENO_OPT_SYS_HEAPSZ 256 +#define CONFIG_KEYBOARD_ATKBD 1 +#define CONFIG_RCU_FANOUT 64 +#define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS 1 +#define CONFIG_HAS_IOPORT 1 +#define CONFIG_HAVE_ARCH_KGDB 1 +#define CONFIG_USB_EHCI_TT_NEWSCHED 1 +#define CONFIG_COMPAT_BINFMT_ELF 1 +#define CONFIG_PHYS_ADDR_T_64BIT 1 +#define CONFIG_XENO_OPT_PIPE 1 +#define CONFIG_MODULES 1 +#define CONFIG_HAVE_SETUP_PER_CPU_AREA 1 +#define CONFIG_OUTPUT_FORMAT "elf64-x86-64" +#define CONFIG_DYNAMIC_FTRACE 1 +#define CONFIG_USB_ARCH_HAS_EHCI 1 +#define CONFIG_CPU_SUP_INTEL 1 +#define CONFIG_PCI_QUIRKS 1 +#define CONFIG_INPUT_MISC 1 +#define CONFIG_MOUSE_PS2_TRACKPOINT 1 +#define CONFIG_BLK_DEV_SR 1 +#define CONFIG_EXT3_FS 1 +#define CONFIG_VT_CONSOLE 1 +#define CONFIG_CRYPTO_DES 1 +#define CONFIG_GENERIC_TRACER 1 +#define CONFIG_SND_MIXER_OSS 1 +#define CONFIG_PREEMPT 1 +#define CONFIG_ACPI 1 +#define CONFIG_USB_HID 1 +#define CONFIG_RTC_LIB 1 +#define CONFIG_SPARSEMEM_VMEMMAP 1 +#define CONFIG_GENERIC_IRQ_PROBE 1 +#define CONFIG_USER_STACKTRACE_SUPPORT 1 +#define CONFIG_CRYPTO_ALGAPI2 1 +#define CONFIG_SND_PCI 1 +#define CONFIG_DEFCONFIG_LIST "/lib/modules/$UNAME_RELEASE/.config" +#define CONFIG_X86_TRAMPOLINE 1 +#define CONFIG_DETECT_SOFTLOCKUP 1 +#define CONFIG_SLAB 1 +#define CONFIG_HAVE_LATENCYTOP_SUPPORT 1 +#define CONFIG_HOTPLUG 1 +#define CONFIG_USB_ARCH_HAS_OHCI 1 +#define CONFIG_SND_PCM 1 +#define CONFIG_DEVPORT 1 +#define CONFIG_SWIOTLB 1 +#define CONFIG_PCI_MSI 1 +#define CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK 1 +#define CONFIG_SPARSEMEM_VMEMMAP_ENABLE 1 +#define CONFIG_COMPAT_FOR_U64_ALIGNMENT 1 +#define CONFIG_SPARSEMEM_EXTREME 1 +#define CONFIG_SUNRPC_GSS 1 +#define CONFIG_PHYSICAL_ALIGN 0x1000000 +#define CONFIG_ARCH_PROC_KCORE_TEXT 1 +#define CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ 1024 +#define CONFIG_DEBUG_SPINLOCK 1 +#define CONFIG_VFAT_FS 1 +#define CONFIG_SERIAL_8250_PCI 1 +#define CONFIG_XENO_DRIVERS_SWITCHTEST 1 +#define CONFIG_IO_DELAY_TYPE_UDELAY 2 +#define CONFIG_R8169 1 +#define CONFIG_EMBEDDED 1 +#define CONFIG_GENERIC_ACL 1 +#define CONFIG_PROC_FS 1 +#define CONFIG_ARCH_HAS_CACHE_LINE_SIZE 1 +#define CONFIG_XENO_OPT_TIMING_VIRTICK 1000 +#define CONFIG_X86_CMPXCHG 1 +#define CONFIG_XENO_OPT_NATIVE_HEAP 1 +#define CONFIG_INET 1 +#define CONFIG_ARCH_SPARSEMEM_ENABLE 1 +#define CONFIG_RT_MUTEXES 1 +#define CONFIG_HPET 1 +#define CONFIG_SYSVIPC 1 +#define CONFIG_WLAN 1 +#define CONFIG_SCSI 1 +#define CONFIG_TCP_CONG_CUBIC 1 +#define CONFIG_X86_L1_CACHE_SHIFT 6 +#define CONFIG_SERIAL_8250_RSA 1 +#define CONFIG_ATA_VERBOSE_ERROR 1 +#define CONFIG_HAVE_IOREMAP_PROT 1 +#define CONFIG_SND_HDA_CODEC_REALTEK 1 +#define CONFIG_ACPI_PROCFS_POWER 1 +#define CONFIG_XENO_OPT_NATIVE_COND 1 +#define CONFIG_PROC_PAGE_MONITOR 1 +#define CONFIG_XENO_OPT_RTDM_FILDES 128 +#define CONFIG_IOSCHED_CFQ 1 +#define CONFIG_X86_IO_APIC 1 +#define CONFIG_PAGEFLAGS_EXTENDED 1 +#define CONFIG_NR_CPUS 8 +#define CONFIG_X86_EXTENDED_PLATFORM 1 +#define CONFIG_ARCH_HAS_DEFAULT_IDLE 1 +#define CONFIG_GENERIC_BUG 1 +#define CONFIG_SERIAL_8250_PNP 1 +#define CONFIG_CRYPTO_MANAGER 1 +#define CONFIG_X86_L1_CACHE_BYTES 64 +#define CONFIG_SWAP 1 +#define CONFIG_HAVE_MLOCKED_PAGE_BIT 1 +#define CONFIG_IPIPE_DEBUG 1 +#define CONFIG_EXT3_FS_POSIX_ACL 1 +#define CONFIG_AC97_BUS 1 +#define CONFIG_CRC32 1 +#define CONFIG_X86_MPPARSE 1 +#define CONFIG_DEFAULT_CFQ 1 +#define CONFIG_XENO_OPT_NATIVE_PIPE 1 +#define CONFIG_FAT_DEFAULT_IOCHARSET "iso8859-15" +#define CONFIG_INPUT_KEYBOARD 1 +#define CONFIG_EXTRA_FIRMWARE "" +#define CONFIG_PACKET_MMAP 1 +#define CONFIG_HAVE_DMA_ATTRS 1 +#define CONFIG_SND_SEQUENCER_OSS 1 +#define CONFIG_UNIX 1 +#define CONFIG_CPU_IDLE_GOV_MENU 1 +#define CONFIG_GENERIC_CPU 1 +#define CONFIG_XENO_OPT_RTDM_PERIOD 0 +#define CONFIG_ARCH_SELECT_MEMORY_MODEL 1 +#define CONFIG_MTRR 1 +#define CONFIG_PCI_DOMAINS 1 +#define CONFIG_ISA_DMA_API 1 +#define CONFIG_SERIAL_CORE 1 +#define CONFIG_SIGNALFD 1 +#define CONFIG_ACPI_PROCFS 1 +#define CONFIG_SCHED_OMIT_FRAME_POINTER 1 +#define CONFIG_RING_BUFFER 1 +#define CONFIG_UID16 1 +#define CONFIG_LOCK_KERNEL 1 +#define CONFIG_HAVE_SYSCALL_TRACEPOINTS 1 +#define CONFIG_64BIT 1 +#define CONFIG_USB_EHCI_ROOT_HUB_TT 1 +#define CONFIG_SCSI_WAIT_SCAN_MODULE 1 +#define CONFIG_PHYSICAL_START 0x200000 +#define CONFIG_IKCONFIG 1 +#define CONFIG_MOUSE_PS2_ALPS 1 +#define CONFIG_IP_FIB_HASH 1 +#define CONFIG_DEFAULT_MMAP_MIN_ADDR 4096 +#define CONFIG_XENO_FASTSYNCH 1 +#define CONFIG_TRACEPOINTS 1 +#define CONFIG_ARCH_USES_PG_UNCACHED 1 +#define CONFIG_ANON_INODES 1 +#define CONFIG_GENERIC_FIND_LAST_BIT 1 +#define CONFIG_SLABINFO 1 +#define CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST 1 +#define CONFIG_VGA_CONSOLE 1 +#define CONFIG_PCIEPORTBUS 1 +#define CONFIG_NLS_DEFAULT "iso8859-1" +#define CONFIG_SND_RAWMIDI 1 +#define CONFIG_ACPI_FAN 1 +#define CONFIG_ATA_ACPI 1 +#define CONFIG_SPLIT_PTLOCK_CPUS 4 +#define CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ 1 +#define CONFIG_SYSVIPC_COMPAT 1 +#define CONFIG_X86_SUPPORTS_MEMORY_FAILURE 1 +#define CONFIG_HAVE_DMA_API_DEBUG 1 +#define CONFIG_ARCH_WANT_FRAME_POINTERS 1 +#define CONFIG_ARCH_SUPPORTS_MSI 1 +#define CONFIG_HAVE_IDE 1 +#define CONFIG_IO_DELAY_TYPE_0X80 0 +#define CONFIG_LOCKD_V4 1 +#define CONFIG_SCHED_MC 1 +#define CONFIG_VM_EVENT_COUNTERS 1 +#define CONFIG_ACPI_BUTTON 1 +#define CONFIG_HAVE_FTRACE_NMI_ENTER 1 +#define CONFIG_CRYPTO_RNG2 1 +#define CONFIG_GENERIC_FIND_NEXT_BIT 1 +#define CONFIG_X86_CMOV 1 +#define CONFIG_XENO_OPT_TIMING_SCHEDLAT 0 +#define CONFIG_KERNEL_GZIP 1 +#define CONFIG_GENERIC_TIME_VSYSCALL 1 +#define CONFIG_UNUSED_SYMBOLS 1 +#define CONFIG_SYSCTL_SYSCALL 1 +#define CONFIG_MTRR_SANITIZER 1 +#define CONFIG_USB_STORAGE 1 +#define CONFIG_SND_HRTIMER 1 +#define CONFIG_SERIAL_8250_RUNTIME_UARTS 4 +#define CONFIG_COMPAT_VDSO 1 +#define CONFIG_ELF_CORE 1 +#define CONFIG_DEBUG_FS 1 +#define CONFIG_ARCH_SUSPEND_POSSIBLE 1 +#define CONFIG_SERIAL_8250_CONSOLE 1 +#define CONFIG_X86_DEBUGCTLMSR 1 +#define CONFIG_MAGIC_SYSRQ 1 +#define CONFIG_SND_DRIVERS 1 +#define CONFIG_SCSI_DMA 1 +#define CONFIG_DEFAULT_IOSCHED "cfq" +#define CONFIG_CRYPTO_HASH 1 +#define CONFIG_UNIX98_PTYS 1 +#define CONFIG_IP_PNP_DHCP 1 +#define CONFIG_ACPI_SBS 1 +#define CONFIG_X86_THERMAL_VECTOR 1 +#define CONFIG_XENO_SKIN_RTDM 1 +#define CONFIG_SOUND_OSS_CORE 1 +#define CONFIG_CPU_IDLE 1 +#define CONFIG_SND_MPU401_UART 1 +#define CONFIG_SND_VMASTER 1 +#define CONFIG_SYN_COOKIES 1 +#define CONFIG_X86_INTERNODE_CACHE_BYTES 64 +#define CONFIG_DMIID 1 +#define CONFIG_MOUSE_PS2_LIFEBOOK 1 +#define CONFIG_SCSI_MULTI_LUN 1 +#define CONFIG_GENERIC_ISA_DMA 1 +#define CONFIG_DEFAULT_IO_DELAY_TYPE 0 +#define CONFIG_PNP_DEBUG_MESSAGES 1 +#define CONFIG_BLOCK 1 +#define CONFIG_SND_OSSEMUL 1 +#define CONFIG_GENERIC_CLOCKEVENTS_BUILD 1 +#define CONFIG_GENERIC_HWEIGHT 1 +#define CONFIG_CRYPTO_MANAGER2 1 +#define CONFIG_LOCKD 1 +#define CONFIG_ZONE_DMA 1 +#define CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE 0 +#define CONFIG_ENABLE_WARN_DEPRECATED 1 +#define CONFIG_TIMER_STATS 1 +#define CONFIG_ATA_PIIX 1 +#define CONFIG_KALLSYMS 1 +#define CONFIG_ARCH_HIBERNATION_POSSIBLE 1 +#define CONFIG_HID_SUPPORT 1 +#define CONFIG_HAVE_KVM 1 +#define CONFIG_FSNOTIFY 1 +#define CONFIG_SYSVIPC_SYSCTL 1 +#define CONFIG_HAVE_UNSTABLE_SCHED_CLOCK 1 +#define CONFIG_X86_VERBOSE_BOOTUP 1 +#define CONFIG_NLS_UTF8 1 +#define CONFIG_SND_HDA_CODEC_CA0110 1 +#define CONFIG_FIRMWARE_IN_KERNEL 1 +#define CONFIG_TASK_IO_ACCOUNTING 1 +#define CONFIG_PROC_SYSCTL 1 +#define CONFIG_CONTEXT_SWITCH_TRACER 1 +#define CONFIG_IO_DELAY_TYPE_0XED 1 +#define CONFIG_IPIPE 1 +#define CONFIG_SND_PCM_OSS 1 +#define CONFIG_INPUT_MOUSE 1 +#define CONFIG_FIRMWARE_MEMMAP 1 +#define CONFIG_SERIAL_8250_MANY_PORTS 1 +#define CONFIG_TREE_RCU 1 +#define CONFIG_XENO_OPT_STATS 1 +#define CONFIG_SHMEM 1 +#define CONFIG_TASK_DELAY_ACCT 1 +#define CONFIG_ARCH_HAS_CPU_RELAX 1 +#define CONFIG_INET_LRO 1 +#define CONFIG_EPOLL 1 +#define CONFIG_CRYPTO_AEAD2 1 +#define CONFIG_RELAY 1 +#define CONFIG_XENO_OPT_NATIVE_ALARM 1 +#define CONFIG_RPCSEC_GSS_KRB5 1 +#define CONFIG_NLS_CODEPAGE_850 1 +#define CONFIG_NLS_CODEPAGE_437 1 +#define CONFIG_STACKTRACE_SUPPORT 1 +#define CONFIG_SERIO 1 +#define CONFIG_UEVENT_HELPER_PATH "/sbin/hotplug" +#define CONFIG_XENO_HW_FPU 1 +#define CONFIG_HAVE_KRETPROBES 1 +#define CONFIG_FUNCTION_TRACER 1 +#define CONFIG_FILE_LOCKING 1 +#define CONFIG_USB_SUPPORT 1 +#define CONFIG_SND_VERBOSE_PROCFS 1 +#define CONFIG_HUGETLB_PAGE 1 +#define CONFIG_IA32_AOUT 1 +#define CONFIG_DEBUG_KERNEL 1 +#define CONFIG_HAVE_ARCH_KMEMCHECK 1 +#define CONFIG_SOUND 1 +#define CONFIG_ROOT_NFS 1 +#define CONFIG_TMPFS 1 +#define CONFIG_RTC_INTF_DEV_UIE_EMUL 1 +#define CONFIG_SCHED_HRTICK 1 +#define CONFIG_GENERIC_TIME 1 +#define CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST 1 +#define CONFIG_HAVE_FTRACE_MCOUNT_RECORD 1 +#define CONFIG_FUTEX 1 +#define CONFIG_SERIO_LIBPS2 1 +#define CONFIG_ARCH_SPARSEMEM_DEFAULT 1 +#define CONFIG_BLOCK_COMPAT 1 +#define CONFIG_EXPERIMENTAL 1 +#define CONFIG_GENERIC_CLOCKEVENTS_BROADCAST 1 +#define CONFIG_X86_64 1 +#define CONFIG_USB_EHCI_HCD 1 +#define CONFIG_INPUT 1 +#define CONFIG_VIRT_TO_BUS 1 +#define CONFIG_ACPI_PROC_EVENT 1 +#define CONFIG_LOCALVERSION "" +#define CONFIG_USB_DEVICEFS 1 +#define CONFIG_SERIO_PCIPS2 1 +#define CONFIG_ZLIB_INFLATE 1 +#define CONFIG_XENO_OPT_NATIVE_EVENT 1 +#define CONFIG_INIT_ENV_ARG_LIMIT 32 +#define CONFIG_IPIPE_DELAYED_ATOMICSW 1 +#define CONFIG_CRYPTO_HASH2 1 +#define CONFIG_IO_DELAY_TYPE_NONE 3 +#define CONFIG_ARCH_PHYS_ADDR_T_64BIT 1 +#define CONFIG_XENO_OPT_NATIVE_QUEUE 1 +#define CONFIG_ACPI_BATTERY 1 +#define CONFIG_PCSPKR_PLATFORM 1 +#define CONFIG_HZ_250 1 +#define CONFIG_XENO_HW_SMI_DETECT 1 +#define CONFIG_SYSFS 1 +#define CONFIG_SND_RAWMIDI_SEQ 1 +#define CONFIG_X86_CMPXCHG64 1 +#define CONFIG_HAVE_KPROBES 1 +#define CONFIG_IPIPE_COMPAT 1 +#define CONFIG_IOSCHED_NOOP 1 +#define CONFIG_STOP_MACHINE 1 +#define CONFIG_HIGH_RES_TIMERS 1 +#define CONFIG_HAVE_DYNAMIC_FTRACE 1 +#define CONFIG_ACPI_DOCK 1 +#define CONFIG_PM 1 +#define CONFIG_GENERIC_CLOCKEVENTS 1 +#define CONFIG_SERIAL_8250_EXTENDED 1 +#define CONFIG_NO_HZ 1 +#define CONFIG_LOCKDEP_SUPPORT 1 +#define CONFIG_MSDOS_PARTITION 1 +#define CONFIG_USB_UHCI_HCD 1 +#define CONFIG_XENO_OPT_POSIX_PERIOD 0 +#define CONFIG_XENO_OPT_PERVASIVE 1 +#define CONFIG_HAVE_ARCH_TRACEHOOK 1 +#define CONFIG_X86_PM_TIMER 1 +#define CONFIG_HZ 250 +#define CONFIG_GENERIC_FIND_FIRST_BIT 1 +#define CONFIG_HUGETLBFS 1 +#define CONFIG_XENO_OPT_NATIVE_MUTEX 1 +#define CONFIG_TRACING_SUPPORT 1 +#define CONFIG_ACPI_BLACKLIST_YEAR 0 +#define CONFIG_XENO_GENERIC_STACKPOOL 1 +#define CONFIG_IOMMU_HELPER 1 +#define CONFIG_XENO_OPT_SYS_STACKPOOLSZ 128 +#define CONFIG_SSB_POSSIBLE 1 +#define CONFIG_X86_MINIMUM_CPU_FAMILY 64 +#define CONFIG_USB_LIBUSUAL 1 +#define CONFIG_VT 1 +#define CONFIG_TICK_ONESHOT 1 +#define CONFIG_TMPFS_POSIX_ACL 1 +#define CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE 1 +#define CONFIG_GENERIC_PENDING_IRQ 1 +#define CONFIG_RWSEM_GENERIC_SPINLOCK 1 +#define CONFIG_BRANCH_PROFILE_NONE 1 +#define CONFIG_SPARSEMEM 1 +#define CONFIG_BASE_FULL 1 +#define CONFIG_PREVENT_FIRMWARE_BUILD 1 +#define CONFIG_SYSFS_DEPRECATED_V2 1 +#define CONFIG_CPU_SUP_CENTAUR 1 +#define CONFIG_GENERIC_CMOS_UPDATE 1 +#define CONFIG_GENERIC_CALIBRATE_DELAY 1 +#define CONFIG_ARCH_HAS_CPU_IDLE_WAIT 1 +#define CONFIG_NFS_ACL_SUPPORT 1 +#define CONFIG_HAS_IOMEM 1 +#define CONFIG_XENO_OPT_PIPELINE_HEAD 1 +#define CONFIG_FW_LOADER 1 +#define CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE 0 +#define CONFIG_IO_DELAY_0X80 1 +#define CONFIG_FTRACE 1 +#define CONFIG_MODULE_FORCE_UNLOAD 1 +#define CONFIG_RTC_INTF_DEV 1 +#define CONFIG_PACKET 1 +#define CONFIG_BSD_PROCESS_ACCT_V3 1 +#define CONFIG_CONSTRUCTORS 1 +#define CONFIG_DEBUG_BUGVERBOSE 1 +#define CONFIG_FS_POSIX_ACL 1 +#define CONFIG_SERIAL_CORE_CONSOLE 1 +#define CONFIG_X86_WP_WORKS_OK 1 +#define CONFIG_GENERIC_HARDIRQS 1 +#define CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC 1 +#define CONFIG_HPET_MMAP 1 +#define CONFIG_CPU_IDLE_GOV_LADDER 1 +#define CONFIG_CRAMFS 1 +#define CONFIG_MOUSE_PS2_LOGIPS2PP 1 +#define CONFIG_BOUNCE 1 +#define CONFIG_SERIO_CT82C710 1 +#define CONFIG_MOUSE_PS2 1 +#define CONFIG_SCSI_PROC_FS 1 +#define CONFIG_RTC_CLASS 1 +#define CONFIG_CRYPTO_PCOMP 1 +#define CONFIG_HAVE_MMIOTRACE_SUPPORT 1 +#define CONFIG_INOTIFY_USER 1 +#define CONFIG_XENO_SKIN_POSIX 1 +#define CONFIG_IPIPE_DEBUG_CONTEXT 1 +#define CONFIG_X86_HT 1 +#define CONFIG_INPUT_PCSPKR 1 +#define CONFIG_TASKSTATS 1 +#define CONFIG_TRACE_IRQFLAGS_SUPPORT 1 +#define CONFIG_COMPAT 1 +#define CONFIG_POSIX_MQUEUE 1 +#define CONFIG_SND_TIMER 1 +#define CONFIG_RTC_DRV_CMOS 1 +#define CONFIG_X86 1 +#define CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT 0 +#define CONFIG_SELECT_MEMORY_MODEL 1 +#define CONFIG_SYSFS_DEPRECATED 1 +#define CONFIG_GENERIC_BUG_RELATIVE_POINTERS 1 +#define CONFIG_SPARSEMEM_MANUAL 1 +#define CONFIG_SND_MPU401 1 +#define CONFIG_SERIO_RAW 1 +#define CONFIG_XENO_OPT_NATIVE_BUFFER 1 +#define CONFIG_EDD 1 +#define CONFIG_NFS_V3 1 +#define CONFIG_NFS_V4 1 +#define CONFIG_JBD 1 +#define CONFIG_USB_ARCH_HAS_HCD 1 +#define CONFIG_DEFAULT_TCP_CONG "cubic" +#define CONFIG_BSD_PROCESS_ACCT 1 +#define CONFIG_ZONE_DMA32 1 +#define CONFIG_GENERIC_IOMAP 1 +#define CONFIG_FAT_FS 1 +#define CONFIG_CRYPTO_BLKCIPHER 1 +#define CONFIG_XENO_OPT_NATIVE_PERIOD 0 +#define CONFIG_FTRACE_NMI_ENTER 1 +#define CONFIG_HID 1 +#define CONFIG_NLATTR 1 +#define CONFIG_X86_MCE_INTEL 1 +#define CONFIG_FAT_DEFAULT_CODEPAGE 437 +#define CONFIG_ATA 1 +#define CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT 1 +#define CONFIG_HAVE_KERNEL_LZMA 1 +#define CONFIG_PRINTK 1 +#define CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING 1 +#define CONFIG_POWER_SUPPLY 1 +#define CONFIG_AIO 1 +#define CONFIG_NLS_ISO8859_15 1 +#define CONFIG_MOUSE_PS2_SYNAPTICS 1 +#define CONFIG_DMI 1 +#define CONFIG_HAVE_FUNCTION_GRAPH_TRACER 1 +#define CONFIG_SUNRPC 1 +#define CONFIG_FS_MBCACHE 1 +#define CONFIG_SERIAL_8250_NR_UARTS 32 +#define CONFIG_DETECT_HUNG_TASK 1 +#define CONFIG_HAVE_MEMORY_PRESENT 1 +#define CONFIG_XENO_TESTING_MODULE_MODULE 1 +#define CONFIG_PCI 1 +#define CONFIG_IKCONFIG_PROC 1 +#define CONFIG_HAVE_KERNEL_GZIP 1 +#define CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG 1 +#define CONFIG_X86_64_SMP 1 +#define CONFIG_BUG 1 +#define CONFIG_NFS_FS 1 +#define CONFIG_MII 1 +#define CONFIG_PROCESSOR_SELECT 1 +#define CONFIG_NETWORK_FILESYSTEMS 1 +#define CONFIG_FIX_EARLYCON_MEM 1 +#define CONFIG_CRYPTO 1 +#define CONFIG_IPIPE_DOMAINS 4 +#define CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB 1 +#define CONFIG_SYSCTL 1 +#define CONFIG_DEBUG_PREEMPT 1 +#define CONFIG_MISC_FILESYSTEMS 1 +#define CONFIG_HAVE_OPROFILE 1 +#define CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK 1 +#define CONFIG_HAVE_PERF_EVENTS 1 +#define CONFIG_STACKTRACE 1 +#define CONFIG_SND 1 +#define CONFIG_HPET_EMULATE_RTC 1 +#define CONFIG_NETDEV_1000 1 +#define CONFIG_IP_PNP 1 +#define CONFIG_ZONE_DMA_FLAG 1 +#define CONFIG_NET 1 +#define CONFIG_MODULE_UNLOAD 1 +#define CONFIG_STANDALONE 1 +#define CONFIG_EVENT_TRACING 1 +#define CONFIG_NETDEVICES 1 +#define CONFIG_USB 1 +#define CONFIG_FRAME_POINTER 1 +#define CONFIG_SND_HDA_INTEL 1 +#define CONFIG_SND_DMA_SGBUF 1 +#define CONFIG_NLS 1 +#define CONFIG_CLOCKSOURCE_WATCHDOG 1 +#define CONFIG_CRYPTO_ALGAPI 1 +#define CONFIG_SERIAL_8250 1 +#define CONFIG_PNP 1 +#define CONFIG_SND_SEQUENCER 1 +#define CONFIG_MMU 1 +#define CONFIG_PCI_LEGACY 1 +#define CONFIG_SMP 1 +#define CONFIG_DEBUG_MUTEXES 1 +#define CONFIG_DEBUG_INFO 1 +#define CONFIG_EXT3_FS_SECURITY 1 +#define CONFIG_BASE_SMALL 0 +#define CONFIG_XENO_SKIN_NATIVE 1 +#define CONFIG_AUDIT_ARCH 1 +#define CONFIG_SND_AC97_CODEC 1 +#define CONFIG_SND_ENS1371 1 +#define CONFIG_BINFMT_ELF 1 +#define CONFIG_PCI_DIRECT 1 +#define CONFIG_XENO_OPT_NATIVE_SEM 1 +#define CONFIG_BINARY_PRINTF 1 +#define CONFIG_XENO_OPT_PIPE_NRDEV 32 +#define CONFIG_SND_SUPPORT_OLD_API 1 +#define CONFIG_SERIO_I8042 1 +#define CONFIG_XENO_DRIVERS_TIMERBENCH 1 +#define CONFIG_XENO_OPT_REGISTRY_NRSLOTS 512 +#define CONFIG_IPIPE_DEBUG_INTERNAL 1 +#define CONFIG_SND_PCM_OSS_PLUGINS 1 +#define CONFIG_PCIEAER 1 +#define CONFIG_HAVE_MLOCK 1 +#define CONFIG_X86_MCE 1 +#define CONFIG_XENO_OPT_SEM_HEAPSZ 12 +#define CONFIG_BITREVERSE 1 +#define CONFIG_LOG_BUF_SHIFT 16 +#define CONFIG_IA32_EMULATION 1 +#define CONFIG_ARCH_POPULATES_NODE_MAP 1 +#define CONFIG_NFS_V3_ACL 1 +#define CONFIG_DUMMY_CONSOLE 1 diff --git a/include/linux/bounds.h b/include/linux/bounds.h new file mode 100644 index 0000000..2ca0fee --- /dev/null +++ b/include/linux/bounds.h @@ -0,0 +1,13 @@ +#ifndef __LINUX_BOUNDS_H__ +#define __LINUX_BOUNDS_H__ +/* + * DO NOT MODIFY. + * + * This file was generated by Kbuild + * + */ + +#define NR_PAGEFLAGS 24 /* __NR_PAGEFLAGS # */ +#define MAX_NR_ZONES 4 /* __MAX_NR_ZONES # */ + +#endif diff --git a/include/linux/compile.h b/include/linux/compile.h new file mode 100644 index 0000000..70f55aa --- /dev/null +++ b/include/linux/compile.h @@ -0,0 +1,9 @@ +/* This file is auto generated, version 92 */ +/* SMP PREEMPT */ +#define UTS_MACHINE "x86_64" +#define UTS_VERSION "#92 SMP PREEMPT Thu Feb 4 10:29:12 CET 2010" +#define LINUX_COMPILE_TIME "10:29:12" +#define LINUX_COMPILE_BY "rpm" +#define LINUX_COMPILE_HOST "redshift" +#define LINUX_COMPILE_DOMAIN "xenomai.org" +#define LINUX_COMPILER "gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9) " diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index d5b3876..010aa8b 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -207,24 +207,28 @@ extern void irq_enter(void); */ extern void irq_exit(void); -#define nmi_enter() \ - do { \ - ftrace_nmi_enter(); \ - BUG_ON(in_nmi()); \ - add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ - lockdep_off(); \ - rcu_nmi_enter(); \ - trace_hardirq_enter(); \ +#define nmi_enter() \ + do { \ + if (likely(!ipipe_test_foreign_stack())) { \ + ftrace_nmi_enter(); \ + BUG_ON(in_nmi()); \ + add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ + lockdep_off(); \ + rcu_nmi_enter(); \ + trace_hardirq_enter(); \ + } \ } while (0) -#define nmi_exit() \ - do { \ - trace_hardirq_exit(); \ - rcu_nmi_exit(); \ - lockdep_on(); \ - BUG_ON(!in_nmi()); \ - sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ - ftrace_nmi_exit(); \ +#define nmi_exit() \ + do { \ + if (likely(!ipipe_test_foreign_stack())) { \ + trace_hardirq_exit(); \ + rcu_nmi_exit(); \ + lockdep_on(); \ + BUG_ON(!in_nmi()); \ + sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ + ftrace_nmi_exit(); \ + } \ } while (0) #endif /* LINUX_HARDIRQ_H */ diff --git a/include/linux/ipipe.h b/include/linux/ipipe.h new file mode 100644 index 0000000..59da296 --- /dev/null +++ b/include/linux/ipipe.h @@ -0,0 +1,684 @@ +/* -*- linux-c -*- + * include/linux/ipipe.h + * + * Copyright (C) 2002-2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_H +#define __LINUX_IPIPE_H + +#include <linux/spinlock.h> +#include <linux/cache.h> +#include <linux/percpu.h> +#include <linux/mutex.h> +#include <linux/linkage.h> +#include <linux/ipipe_base.h> +#include <asm/ipipe.h> + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + +#include <linux/cpumask.h> +#include <asm/system.h> + +static inline int ipipe_disable_context_check(int cpu) +{ + return xchg(&per_cpu(ipipe_percpu_context_check, cpu), 0); +} + +static inline void ipipe_restore_context_check(int cpu, int old_state) +{ + per_cpu(ipipe_percpu_context_check, cpu) = old_state; +} + +static inline void ipipe_context_check_off(void) +{ + int cpu; + for_each_online_cpu(cpu) + per_cpu(ipipe_percpu_context_check, cpu) = 0; +} + +#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +static inline int ipipe_disable_context_check(int cpu) +{ + return 0; +} + +static inline void ipipe_restore_context_check(int cpu, int old_state) { } + +static inline void ipipe_context_check_off(void) { } + +#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +#ifdef CONFIG_IPIPE + +#define IPIPE_VERSION_STRING IPIPE_ARCH_STRING +#define IPIPE_RELEASE_NUMBER ((IPIPE_MAJOR_NUMBER << 16) | \ + (IPIPE_MINOR_NUMBER << 8) | \ + (IPIPE_PATCH_NUMBER)) + +#ifndef BROKEN_BUILTIN_RETURN_ADDRESS +#define __BUILTIN_RETURN_ADDRESS0 ((unsigned long)__builtin_return_address(0)) +#define __BUILTIN_RETURN_ADDRESS1 ((unsigned long)__builtin_return_address(1)) +#endif /* !BUILTIN_RETURN_ADDRESS */ + +#define IPIPE_ROOT_PRIO 100 +#define IPIPE_ROOT_ID 0 +#define IPIPE_ROOT_NPTDKEYS 4 /* Must be <= BITS_PER_LONG */ + +#define IPIPE_RESET_TIMER 0x1 +#define IPIPE_GRAB_TIMER 0x2 + +/* Global domain flags */ +#define IPIPE_SPRINTK_FLAG 0 /* Synchronous printk() allowed */ +#define IPIPE_AHEAD_FLAG 1 /* Domain always heads the pipeline */ + +/* Interrupt control bits */ +#define IPIPE_HANDLE_FLAG 0 +#define IPIPE_PASS_FLAG 1 +#define IPIPE_ENABLE_FLAG 2 +#define IPIPE_DYNAMIC_FLAG IPIPE_HANDLE_FLAG +#define IPIPE_STICKY_FLAG 3 +#define IPIPE_SYSTEM_FLAG 4 +#define IPIPE_LOCK_FLAG 5 +#define IPIPE_WIRED_FLAG 6 +#define IPIPE_EXCLUSIVE_FLAG 7 + +#define IPIPE_HANDLE_MASK (1 << IPIPE_HANDLE_FLAG) +#define IPIPE_PASS_MASK (1 << IPIPE_PASS_FLAG) +#define IPIPE_ENABLE_MASK (1 << IPIPE_ENABLE_FLAG) +#define IPIPE_DYNAMIC_MASK IPIPE_HANDLE_MASK +#define IPIPE_STICKY_MASK (1 << IPIPE_STICKY_FLAG) +#define IPIPE_SYSTEM_MASK (1 << IPIPE_SYSTEM_FLAG) +#define IPIPE_LOCK_MASK (1 << IPIPE_LOCK_FLAG) +#define IPIPE_WIRED_MASK (1 << IPIPE_WIRED_FLAG) +#define IPIPE_EXCLUSIVE_MASK (1 << IPIPE_EXCLUSIVE_FLAG) + +#define IPIPE_DEFAULT_MASK (IPIPE_HANDLE_MASK|IPIPE_PASS_MASK) +#define IPIPE_STDROOT_MASK (IPIPE_HANDLE_MASK|IPIPE_PASS_MASK|IPIPE_SYSTEM_MASK) + +#define IPIPE_EVENT_SELF 0x80000000 + +#define IPIPE_NR_CPUS NR_CPUS + +/* This accessor assumes hw IRQs are off on SMP; allows assignment. */ +#define __ipipe_current_domain __ipipe_get_cpu_var(ipipe_percpu_domain) +/* This read-only accessor makes sure that hw IRQs are off on SMP. */ +#define ipipe_current_domain \ + ({ \ + struct ipipe_domain *__ipd__; \ + unsigned long __flags__; \ + local_irq_save_hw_smp(__flags__); \ + __ipd__ = __ipipe_current_domain; \ + local_irq_restore_hw_smp(__flags__); \ + __ipd__; \ + }) + +#define ipipe_virtual_irq_p(irq) ((irq) >= IPIPE_VIRQ_BASE && \ + (irq) < IPIPE_NR_IRQS) + +#define IPIPE_SAME_HANDLER ((ipipe_irq_handler_t)(-1)) + +struct irq_desc; + +typedef void (*ipipe_irq_ackfn_t)(unsigned irq, struct irq_desc *desc); + +typedef int (*ipipe_event_handler_t)(unsigned event, + struct ipipe_domain *from, + void *data); +struct ipipe_domain { + + int slot; /* Slot number in percpu domain data array. */ + struct list_head p_link; /* Link in pipeline */ + ipipe_event_handler_t evhand[IPIPE_NR_EVENTS]; /* Event handlers. */ + unsigned long long evself; /* Self-monitored event bits. */ + + struct ipipe_irqdesc { + unsigned long control; + ipipe_irq_ackfn_t acknowledge; + ipipe_irq_handler_t handler; + void *cookie; + } ____cacheline_aligned irqs[IPIPE_NR_IRQS]; + + int priority; + void *pdd; + unsigned long flags; + unsigned domid; + const char *name; + struct mutex mutex; +}; + +#define IPIPE_HEAD_PRIORITY (-1) /* For domains always heading the pipeline */ + +struct ipipe_domain_attr { + + unsigned domid; /* Domain identifier -- Magic value set by caller */ + const char *name; /* Domain name -- Warning: won't be dup'ed! */ + int priority; /* Priority in interrupt pipeline */ + void (*entry) (void); /* Domain entry point */ + void *pdd; /* Per-domain (opaque) data pointer */ +}; + +#define __ipipe_irq_cookie(ipd, irq) (ipd)->irqs[irq].cookie +#define __ipipe_irq_handler(ipd, irq) (ipd)->irqs[irq].handler +#define __ipipe_cpudata_irq_hits(ipd, cpu, irq) ipipe_percpudom(ipd, irqall, cpu)[irq] + +extern unsigned __ipipe_printk_virq; + +extern unsigned long __ipipe_virtual_irq_map; + +extern struct list_head __ipipe_pipeline; + +extern int __ipipe_event_monitors[]; + +/* Private interface */ + +void ipipe_init(void); + +#ifdef CONFIG_PROC_FS +void ipipe_init_proc(void); + +#ifdef CONFIG_IPIPE_TRACE +void __ipipe_init_tracer(void); +#else /* !CONFIG_IPIPE_TRACE */ +#define __ipipe_init_tracer() do { } while(0) +#endif /* CONFIG_IPIPE_TRACE */ + +#else /* !CONFIG_PROC_FS */ +#define ipipe_init_proc() do { } while(0) +#endif /* CONFIG_PROC_FS */ + +void __ipipe_init_stage(struct ipipe_domain *ipd); + +void __ipipe_cleanup_domain(struct ipipe_domain *ipd); + +void __ipipe_add_domain_proc(struct ipipe_domain *ipd); + +void __ipipe_remove_domain_proc(struct ipipe_domain *ipd); + +void __ipipe_flush_printk(unsigned irq, void *cookie); + +void __ipipe_walk_pipeline(struct list_head *pos); + +void __ipipe_pend_irq(unsigned irq, struct list_head *head); + +int __ipipe_dispatch_event(unsigned event, void *data); + +void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq); + +void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq); + +void __ipipe_sync_stage(int dovirt); + +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq); + +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq); + +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq); + +void __ipipe_pin_range_globally(unsigned long start, unsigned long end); + +/* Must be called hw IRQs off. */ +static inline void ipipe_irq_lock(unsigned irq) +{ + __ipipe_lock_irq(__ipipe_current_domain, ipipe_processor_id(), irq); +} + +/* Must be called hw IRQs off. */ +static inline void ipipe_irq_unlock(unsigned irq) +{ + __ipipe_unlock_irq(__ipipe_current_domain, irq); +} + +#ifndef __ipipe_sync_pipeline +#define __ipipe_sync_pipeline(dovirt) __ipipe_sync_stage(dovirt) +#endif + +#ifndef __ipipe_run_irqtail +#define __ipipe_run_irqtail() do { } while(0) +#endif + +#define __ipipe_pipeline_head_p(ipd) (&(ipd)->p_link == __ipipe_pipeline.next) + +#define __ipipe_ipending_p(p) ((p)->irqpend_himap != 0) + +/* + * Keep the following as a macro, so that client code could check for + * the support of the invariant pipeline head optimization. + */ +#define __ipipe_pipeline_head() \ + list_entry(__ipipe_pipeline.next, struct ipipe_domain, p_link) + +#define local_irq_enable_hw_cond() local_irq_enable_hw() +#define local_irq_disable_hw_cond() local_irq_disable_hw() +#define local_irq_save_hw_cond(flags) local_irq_save_hw(flags) +#define local_irq_restore_hw_cond(flags) local_irq_restore_hw(flags) + +#ifdef CONFIG_SMP +cpumask_t __ipipe_set_irq_affinity(unsigned irq, cpumask_t cpumask); +int __ipipe_send_ipi(unsigned ipi, cpumask_t cpumask); +#define local_irq_save_hw_smp(flags) local_irq_save_hw(flags) +#define local_irq_restore_hw_smp(flags) local_irq_restore_hw(flags) +#else /* !CONFIG_SMP */ +#define local_irq_save_hw_smp(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_smp(flags) do { } while(0) +#endif /* CONFIG_SMP */ + +#define local_irq_save_full(vflags, rflags) \ + do { \ + local_irq_save(vflags); \ + local_irq_save_hw(rflags); \ + } while(0) + +#define local_irq_restore_full(vflags, rflags) \ + do { \ + local_irq_restore_hw(rflags); \ + local_irq_restore(vflags); \ + } while(0) + +static inline void __local_irq_restore_nosync(unsigned long x) +{ + struct ipipe_percpu_domain_data *p = ipipe_root_cpudom_ptr(); + + if (raw_irqs_disabled_flags(x)) { + set_bit(IPIPE_STALL_FLAG, &p->status); + trace_hardirqs_off(); + } else { + trace_hardirqs_on(); + clear_bit(IPIPE_STALL_FLAG, &p->status); + } +} + +static inline void local_irq_restore_nosync(unsigned long x) +{ + unsigned long flags; + local_irq_save_hw_smp(flags); + __local_irq_restore_nosync(x); + local_irq_restore_hw_smp(flags); +} + +#define __ipipe_root_domain_p (__ipipe_current_domain == ipipe_root_domain) +#define ipipe_root_domain_p (ipipe_current_domain == ipipe_root_domain) + +static inline int __ipipe_event_monitored_p(int ev) +{ + if (__ipipe_event_monitors[ev] > 0) + return 1; + + return (ipipe_current_domain->evself & (1LL << ev)) != 0; +} + +#define ipipe_sigwake_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SIGWAKE)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SIGWAKE, p); \ +} while(0) + +#define ipipe_exit_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_EXIT)) \ + __ipipe_dispatch_event(IPIPE_EVENT_EXIT, p); \ +} while(0) + +#define ipipe_setsched_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SETSCHED)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SETSCHED, p); \ +} while(0) + +#define ipipe_schedule_notify(prev, next) \ +do { \ + if ((((prev)->flags|(next)->flags) & PF_EVNOTIFY) && \ + __ipipe_event_monitored_p(IPIPE_EVENT_SCHEDULE)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SCHEDULE,next); \ +} while(0) + +#define ipipe_trap_notify(ex, regs) \ +({ \ + unsigned long __flags__; \ + int __ret__ = 0; \ + local_irq_save_hw_smp(__flags__); \ + if ((test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)) || \ + ((current)->flags & PF_EVNOTIFY)) && \ + __ipipe_event_monitored_p(ex)) { \ + local_irq_restore_hw_smp(__flags__); \ + __ret__ = __ipipe_dispatch_event(ex, regs); \ + } else \ + local_irq_restore_hw_smp(__flags__); \ + __ret__; \ +}) + +static inline void ipipe_init_notify(struct task_struct *p) +{ + if (__ipipe_event_monitored_p(IPIPE_EVENT_INIT)) + __ipipe_dispatch_event(IPIPE_EVENT_INIT, p); +} + +struct mm_struct; + +static inline void ipipe_cleanup_notify(struct mm_struct *mm) +{ + if (__ipipe_event_monitored_p(IPIPE_EVENT_CLEANUP)) + __ipipe_dispatch_event(IPIPE_EVENT_CLEANUP, mm); +} + +/* Public interface */ + +int ipipe_register_domain(struct ipipe_domain *ipd, + struct ipipe_domain_attr *attr); + +int ipipe_unregister_domain(struct ipipe_domain *ipd); + +void ipipe_suspend_domain(void); + +int ipipe_virtualize_irq(struct ipipe_domain *ipd, + unsigned irq, + ipipe_irq_handler_t handler, + void *cookie, + ipipe_irq_ackfn_t acknowledge, + unsigned modemask); + +int ipipe_control_irq(unsigned irq, + unsigned clrmask, + unsigned setmask); + +unsigned ipipe_alloc_virq(void); + +int ipipe_free_virq(unsigned virq); + +int ipipe_trigger_irq(unsigned irq); + +static inline void __ipipe_propagate_irq(unsigned irq) +{ + struct list_head *next = __ipipe_current_domain->p_link.next; + if (next == &ipipe_root.p_link) { + /* Fast path: root must handle all interrupts. */ + __ipipe_set_irq_pending(&ipipe_root, irq); + return; + } + __ipipe_pend_irq(irq, next); +} + +static inline void __ipipe_schedule_irq(unsigned irq) +{ + __ipipe_pend_irq(irq, &__ipipe_current_domain->p_link); +} + +static inline void __ipipe_schedule_irq_head(unsigned irq) +{ + __ipipe_set_irq_pending(__ipipe_pipeline_head(), irq); +} + +static inline void __ipipe_schedule_irq_root(unsigned irq) +{ + __ipipe_set_irq_pending(&ipipe_root, irq); +} + +static inline void ipipe_propagate_irq(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_propagate_irq(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq_head(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq_head(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq_root(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq_root(irq); + local_irq_restore_hw(flags); +} + +void ipipe_stall_pipeline_from(struct ipipe_domain *ipd); + +unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd); + +unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd); + +static inline void ipipe_unstall_pipeline_from(struct ipipe_domain *ipd) +{ + ipipe_test_and_unstall_pipeline_from(ipd); +} + +void ipipe_restore_pipeline_from(struct ipipe_domain *ipd, + unsigned long x); + +static inline unsigned long ipipe_test_pipeline_from(struct ipipe_domain *ipd) +{ + return test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline void ipipe_stall_pipeline_head(void) +{ + local_irq_disable_hw(); + __set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status)); +} + +static inline unsigned long ipipe_test_and_stall_pipeline_head(void) +{ + local_irq_disable_hw(); + return __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status)); +} + +void ipipe_unstall_pipeline_head(void); + +void __ipipe_restore_pipeline_head(unsigned long x); + +static inline void ipipe_restore_pipeline_head(unsigned long x) +{ + /* On some archs, __test_and_set_bit() might return different + * truth value than test_bit(), so we test the exclusive OR of + * both statuses, assuming that the lowest bit is always set in + * the truth value (if this is wrong, the failed optimization will + * be caught in __ipipe_restore_pipeline_head() if + * CONFIG_DEBUG_KERNEL is set). */ + if ((x ^ test_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status))) & 1) + __ipipe_restore_pipeline_head(x); +} + +#define ipipe_unstall_pipeline() \ + ipipe_unstall_pipeline_from(ipipe_current_domain) + +#define ipipe_test_and_unstall_pipeline() \ + ipipe_test_and_unstall_pipeline_from(ipipe_current_domain) + +#define ipipe_test_pipeline() \ + ipipe_test_pipeline_from(ipipe_current_domain) + +#define ipipe_test_and_stall_pipeline() \ + ipipe_test_and_stall_pipeline_from(ipipe_current_domain) + +#define ipipe_stall_pipeline() \ + ipipe_stall_pipeline_from(ipipe_current_domain) + +#define ipipe_restore_pipeline(x) \ + ipipe_restore_pipeline_from(ipipe_current_domain, (x)) + +void ipipe_init_attr(struct ipipe_domain_attr *attr); + +int ipipe_get_sysinfo(struct ipipe_sysinfo *sysinfo); + +unsigned long ipipe_critical_enter(void (*syncfn) (void)); + +void ipipe_critical_exit(unsigned long flags); + +static inline void ipipe_set_printk_sync(struct ipipe_domain *ipd) +{ + set_bit(IPIPE_SPRINTK_FLAG, &ipd->flags); +} + +static inline void ipipe_set_printk_async(struct ipipe_domain *ipd) +{ + clear_bit(IPIPE_SPRINTK_FLAG, &ipd->flags); +} + +static inline void ipipe_set_foreign_stack(struct ipipe_domain *ipd) +{ + /* Must be called hw interrupts off. */ + __set_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline void ipipe_clear_foreign_stack(struct ipipe_domain *ipd) +{ + /* Must be called hw interrupts off. */ + __clear_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline int ipipe_test_foreign_stack(void) +{ + /* Must be called hw interrupts off. */ + return test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)); +} + +#ifndef ipipe_safe_current +#define ipipe_safe_current() \ +({ \ + struct task_struct *p; \ + unsigned long flags; \ + local_irq_save_hw_smp(flags); \ + p = ipipe_test_foreign_stack() ? &init_task : current; \ + local_irq_restore_hw_smp(flags); \ + p; \ +}) +#endif + +ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd, + unsigned event, + ipipe_event_handler_t handler); + +cpumask_t ipipe_set_irq_affinity(unsigned irq, + cpumask_t cpumask); + +int ipipe_send_ipi(unsigned ipi, + cpumask_t cpumask); + +int ipipe_setscheduler_root(struct task_struct *p, + int policy, + int prio); + +int ipipe_reenter_root(struct task_struct *prev, + int policy, + int prio); + +int ipipe_alloc_ptdkey(void); + +int ipipe_free_ptdkey(int key); + +int ipipe_set_ptd(int key, + void *value); + +void *ipipe_get_ptd(int key); + +int ipipe_disable_ondemand_mappings(struct task_struct *tsk); + +static inline void ipipe_nmi_enter(void) +{ + int cpu = ipipe_processor_id(); + + per_cpu(ipipe_nmi_saved_root, cpu) = ipipe_root_cpudom_var(status); + __set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + per_cpu(ipipe_saved_context_check_state, cpu) = + ipipe_disable_context_check(cpu); +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ +} + +static inline void ipipe_nmi_exit(void) +{ + int cpu = ipipe_processor_id(); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + ipipe_restore_context_check + (cpu, per_cpu(ipipe_saved_context_check_state, cpu)); +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ + + if (!test_bit(IPIPE_STALL_FLAG, &per_cpu(ipipe_nmi_saved_root, cpu))) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); +} + +#else /* !CONFIG_IPIPE */ + +#define ipipe_init() do { } while(0) +#define ipipe_suspend_domain() do { } while(0) +#define ipipe_sigwake_notify(p) do { } while(0) +#define ipipe_setsched_notify(p) do { } while(0) +#define ipipe_init_notify(p) do { } while(0) +#define ipipe_exit_notify(p) do { } while(0) +#define ipipe_cleanup_notify(mm) do { } while(0) +#define ipipe_trap_notify(t,r) 0 +#define ipipe_init_proc() do { } while(0) + +static inline void __ipipe_pin_range_globally(unsigned long start, + unsigned long end) +{ +} + +static inline int ipipe_test_foreign_stack(void) +{ + return 0; +} + +#define local_irq_enable_hw_cond() do { } while(0) +#define local_irq_disable_hw_cond() do { } while(0) +#define local_irq_save_hw_cond(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_cond(flags) do { } while(0) +#define local_irq_save_hw_smp(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_smp(flags) do { } while(0) + +#define ipipe_irq_lock(irq) do { } while(0) +#define ipipe_irq_unlock(irq) do { } while(0) + +#define __ipipe_root_domain_p 1 +#define ipipe_root_domain_p 1 +#define ipipe_safe_current current +#define ipipe_processor_id() smp_processor_id() + +#define ipipe_nmi_enter() do { } while (0) +#define ipipe_nmi_exit() do { } while (0) + +#define local_irq_disable_head() local_irq_disable() + +#define local_irq_save_full(vflags, rflags) do { (void)(vflags); local_irq_save(rflags); } while(0) +#define local_irq_restore_full(vflags, rflags) do { (void)(vflags); local_irq_restore(rflags); } while(0) +#define local_irq_restore_nosync(vflags) local_irq_restore(vflags) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_H */ diff --git a/include/linux/ipipe_base.h b/include/linux/ipipe_base.h new file mode 100644 index 0000000..2b0705d --- /dev/null +++ b/include/linux/ipipe_base.h @@ -0,0 +1,133 @@ +/* -*- linux-c -*- + * include/linux/ipipe_base.h + * + * Copyright (C) 2002-2007 Philippe Gerum. + * 2007 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_BASE_H +#define __LINUX_IPIPE_BASE_H + +#ifdef CONFIG_IPIPE + +#include <asm/ipipe_base.h> +#include <asm/bitsperlong.h> + +#define __bpl_up(x) (((x)+(BITS_PER_LONG-1)) & ~(BITS_PER_LONG-1)) +/* Number of virtual IRQs (must be a multiple of BITS_PER_LONG) */ +#define IPIPE_NR_VIRQS BITS_PER_LONG +/* First virtual IRQ # (must be aligned on BITS_PER_LONG) */ +#define IPIPE_VIRQ_BASE __bpl_up(IPIPE_NR_XIRQS) +/* Total number of IRQ slots */ +#define IPIPE_NR_IRQS (IPIPE_VIRQ_BASE+IPIPE_NR_VIRQS) + +#define IPIPE_IRQ_LOMAPSZ (IPIPE_NR_IRQS / BITS_PER_LONG) +#if IPIPE_IRQ_LOMAPSZ > BITS_PER_LONG +/* + * We need a 3-level mapping. This allows us to handle up to 32k IRQ + * vectors on 32bit machines, 256k on 64bit ones. + */ +#define __IPIPE_3LEVEL_IRQMAP 1 +#define IPIPE_IRQ_MDMAPSZ (__bpl_up(IPIPE_IRQ_LOMAPSZ) / BITS_PER_LONG) +#else +/* + * 2-level mapping is enough. This allows us to handle up to 1024 IRQ + * vectors on 32bit machines, 4096 on 64bit ones. + */ +#define __IPIPE_2LEVEL_IRQMAP 1 +#endif + +#define IPIPE_IRQ_DOALL 0 +#define IPIPE_IRQ_DOVIRT 1 + +/* Per-cpu pipeline status */ +#define IPIPE_STALL_FLAG 0 /* Stalls a pipeline stage -- guaranteed at bit #0 */ +#define IPIPE_SYNC_FLAG 1 /* The interrupt syncer is running for the domain */ +#define IPIPE_NOSTACK_FLAG 2 /* Domain currently runs on a foreign stack */ + +#define IPIPE_STALL_MASK (1L << IPIPE_STALL_FLAG) +#define IPIPE_SYNC_MASK (1L << IPIPE_SYNC_FLAG) +#define IPIPE_NOSTACK_MASK (1L << IPIPE_NOSTACK_FLAG) + +typedef void (*ipipe_irq_handler_t)(unsigned int irq, + void *cookie); + +extern struct ipipe_domain ipipe_root; + +#define ipipe_root_domain (&ipipe_root) + +void __ipipe_unstall_root(void); + +void __ipipe_restore_root(unsigned long x); + +#define ipipe_preempt_disable(flags) \ + do { \ + local_irq_save_hw(flags); \ + if (__ipipe_root_domain_p) \ + preempt_disable(); \ + } while (0) + +#define ipipe_preempt_enable(flags) \ + do { \ + if (__ipipe_root_domain_p) { \ + preempt_enable_no_resched(); \ + local_irq_restore_hw(flags); \ + preempt_check_resched(); \ + } else \ + local_irq_restore_hw(flags); \ + } while (0) + +#define ipipe_get_cpu(flags) ({ ipipe_preempt_disable(flags); ipipe_processor_id(); }) +#define ipipe_put_cpu(flags) ipipe_preempt_enable(flags) + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT +void ipipe_check_context(struct ipipe_domain *border_ipd); +#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */ +static inline void ipipe_check_context(struct ipipe_domain *border_ipd) { } +#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +/* Generic features */ + +#ifdef CONFIG_GENERIC_CLOCKEVENTS +#define __IPIPE_FEATURE_REQUEST_TICKDEV 1 +#endif +#define __IPIPE_FEATURE_DELAYED_ATOMICSW 1 +#define __IPIPE_FEATURE_FASTPEND_IRQ 1 +#define __IPIPE_FEATURE_TRACE_EVENT 1 + +#else /* !CONFIG_IPIPE */ + +#define ipipe_preempt_disable(flags) \ + do { \ + preempt_disable(); \ + (void)(flags); \ + } while (0) +#define ipipe_preempt_enable(flags) preempt_enable() + +#define ipipe_get_cpu(flags) ({ (void)(flags); get_cpu(); }) +#define ipipe_put_cpu(flags) \ + do { \ + (void)(flags); \ + put_cpu(); \ + } while (0) + +#define ipipe_check_context(ipd) do { } while(0) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_BASE_H */ diff --git a/include/linux/ipipe_lock.h b/include/linux/ipipe_lock.h new file mode 100644 index 0000000..b4d0e4b --- /dev/null +++ b/include/linux/ipipe_lock.h @@ -0,0 +1,197 @@ +/* -*- linux-c -*- + * include/linux/ipipe_lock.h + * + * Copyright (C) 2009 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_LOCK_H +#define __LINUX_IPIPE_LOCK_H + +typedef struct { + arch_spinlock_t arch_lock; +} __ipipe_spinlock_t; + +#define ipipe_spinlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), __ipipe_spinlock_t *) + +#define std_spinlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), spinlock_t *) + +#define ipipe_spinlock(lock) ((__ipipe_spinlock_t *)(lock)) +#define std_spinlock(lock) ((spinlock_t *)(lock)) + +#define PICK_SPINLOCK_IRQSAVE(lock, flags) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + (flags) = __ipipe_spin_lock_irqsave(ipipe_spinlock(lock)); \ + else if (std_spinlock_p(lock)) \ + raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \ + else __bad_lock_type(); \ + } while (0) + +#define PICK_SPINTRYLOCK_IRQSAVE(lock, flags) \ + ({ \ + int __ret__; \ + if (ipipe_spinlock_p(lock)) \ + __ret__ = __ipipe_spin_trylock_irqsave(ipipe_spinlock(lock), &(flags)); \ + else if (std_spinlock_p(lock)) \ + __ret__ = raw_spin_trylock_irqsave(&std_spinlock(lock)->rlock, flags); \ + else __bad_lock_type(); \ + __ret__; \ + }) + +#define PICK_SPINUNLOCK_IRQRESTORE(lock, flags) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + __ipipe_spin_unlock_irqrestore(ipipe_spinlock(lock), flags); \ + else if (std_spinlock_p(lock)) \ + raw_spin_unlock_irqrestore(&std_spinlock(lock)->rlock, flags); \ + } while (0) + +#define PICK_SPINOP(op, lock) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + arch_spin##op(&ipipe_spinlock(lock)->arch_lock); \ + else if (std_spinlock_p(lock)) \ + raw_spin##op(&std_spinlock(lock)->rlock); \ + else __bad_lock_type(); \ + } while (0) + +#define PICK_SPINOP_RET(op, lock, type) \ + ({ \ + type __ret__; \ + if (ipipe_spinlock_p(lock)) \ + __ret__ = arch_spin##op(&ipipe_spinlock(lock)->arch_lock); \ + else if (std_spinlock_p(lock)) \ + __ret__ = raw_spin##op(&std_spinlock(lock)->rlock); \ + else { __ret__ = -1; __bad_lock_type(); } \ + __ret__; \ + }) + +#define arch_spin_lock_init(lock) \ + do { \ + IPIPE_DEFINE_SPINLOCK(__lock__); \ + *((ipipe_spinlock_t *)lock) = __lock__; \ + } while (0) + +#define arch_spin_lock_irq(lock) \ + do { \ + local_irq_disable_hw(); \ + arch_spin_lock(lock); \ + } while (0) + +#define arch_spin_unlock_irq(lock) \ + do { \ + arch_spin_unlock(lock); \ + local_irq_enable_hw(); \ + } while (0) + +typedef struct { + arch_rwlock_t arch_lock; +} __ipipe_rwlock_t; + +#define ipipe_rwlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), __ipipe_rwlock_t *) + +#define std_rwlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), rwlock_t *) + +#define ipipe_rwlock(lock) ((__ipipe_rwlock_t *)(lock)) +#define std_rwlock(lock) ((rwlock_t *)(lock)) + +#define PICK_RWOP(op, lock) \ + do { \ + if (ipipe_rwlock_p(lock)) \ + arch##op(&ipipe_rwlock(lock)->arch_lock); \ + else if (std_rwlock_p(lock)) \ + _raw##op(std_rwlock(lock)); \ + else __bad_lock_type(); \ + } while (0) + +extern int __bad_lock_type(void); + +#ifdef CONFIG_IPIPE + +#define ipipe_spinlock_t __ipipe_spinlock_t +#define IPIPE_DEFINE_SPINLOCK(x) ipipe_spinlock_t x = IPIPE_SPIN_LOCK_UNLOCKED +#define IPIPE_DECLARE_SPINLOCK(x) extern ipipe_spinlock_t x + +#define IPIPE_SPIN_LOCK_UNLOCKED \ + (__ipipe_spinlock_t) { .arch_lock = __ARCH_SPIN_LOCK_UNLOCKED } + +#define spin_lock_irqsave_cond(lock, flags) \ + spin_lock_irqsave(lock, flags) + +#define spin_unlock_irqrestore_cond(lock, flags) \ + spin_unlock_irqrestore(lock, flags) + +void __ipipe_spin_lock_irq(ipipe_spinlock_t *lock); + +void __ipipe_spin_unlock_irq(ipipe_spinlock_t *lock); + +unsigned long __ipipe_spin_lock_irqsave(ipipe_spinlock_t *lock); + +int __ipipe_spin_trylock_irqsave(ipipe_spinlock_t *lock, + unsigned long *x); + +void __ipipe_spin_unlock_irqrestore(ipipe_spinlock_t *lock, + unsigned long x); + +void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock); + +void __ipipe_spin_unlock_irqcomplete(unsigned long x); + +#define ipipe_rwlock_t __ipipe_rwlock_t +#define IPIPE_DEFINE_RWLOCK(x) ipipe_rwlock_t x = IPIPE_RW_LOCK_UNLOCKED +#define IPIPE_DECLARE_RWLOCK(x) extern ipipe_rwlock_t x + +#define IPIPE_RW_LOCK_UNLOCKED \ + (__ipipe_rwlock_t) { .arch_lock = __ARCH_RW_LOCK_UNLOCKED } + +#else /* !CONFIG_IPIPE */ + +#define ipipe_spinlock_t spinlock_t +#define IPIPE_DEFINE_SPINLOCK(x) DEFINE_SPINLOCK(x) +#define IPIPE_DECLARE_SPINLOCK(x) extern spinlock_t x +#define IPIPE_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED + +#define spin_lock_irqsave_cond(lock, flags) \ + do { \ + (void)(flags); \ + spin_lock(lock); \ + } while(0) + +#define spin_unlock_irqrestore_cond(lock, flags) \ + spin_unlock(lock) + +#define __ipipe_spin_lock_irq(lock) do { } while (0) +#define __ipipe_spin_unlock_irq(lock) do { } while (0) +#define __ipipe_spin_lock_irqsave(lock) 0 +#define __ipipe_spin_trylock_irqsave(lock, x) ({ (void)(x); 1; }) +#define __ipipe_spin_unlock_irqrestore(lock, x) do { (void)(x); } while (0) +#define __ipipe_spin_unlock_irqbegin(lock) do { } while (0) +#define __ipipe_spin_unlock_irqcomplete(x) do { (void)(x); } while (0) + +#define ipipe_rwlock_t rwlock_t +#define IPIPE_DEFINE_RWLOCK(x) DEFINE_RWLOCK(x) +#define IPIPE_DECLARE_RWLOCK(x) extern rwlock_t x +#define IPIPE_RW_LOCK_UNLOCKED RW_LOCK_UNLOCKED + +#endif /* !CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_LOCK_H */ diff --git a/include/linux/ipipe_percpu.h b/include/linux/ipipe_percpu.h new file mode 100644 index 0000000..f6727e3 --- /dev/null +++ b/include/linux/ipipe_percpu.h @@ -0,0 +1,89 @@ +/* -*- linux-c -*- + * include/linux/ipipe_percpu.h + * + * Copyright (C) 2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_PERCPU_H +#define __LINUX_IPIPE_PERCPU_H + +#include <asm/percpu.h> +#include <asm/ptrace.h> + +struct ipipe_domain; + +struct ipipe_percpu_domain_data { + unsigned long status; /* <= Must be first in struct. */ + unsigned long irqpend_himap; +#ifdef __IPIPE_3LEVEL_IRQMAP + unsigned long irqpend_mdmap[IPIPE_IRQ_MDMAPSZ]; +#endif + unsigned long irqpend_lomap[IPIPE_IRQ_LOMAPSZ]; + unsigned long irqheld_map[IPIPE_IRQ_LOMAPSZ]; + unsigned long irqall[IPIPE_NR_IRQS]; + u64 evsync; +}; + +/* + * CAREFUL: all accessors based on __raw_get_cpu_var() you may find in + * this file should be used only while hw interrupts are off, to + * prevent from CPU migration regardless of the running domain. + */ +#ifdef CONFIG_SMP +#define ipipe_percpudom_ptr(ipd, cpu) \ + (&per_cpu(ipipe_percpu_darray, cpu)[(ipd)->slot]) +#define ipipe_cpudom_ptr(ipd) \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[(ipd)->slot]) +#else +DECLARE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]); +#define ipipe_percpudom_ptr(ipd, cpu) \ + (per_cpu(ipipe_percpu_daddr, cpu)[(ipd)->slot]) +#define ipipe_cpudom_ptr(ipd) \ + (__ipipe_get_cpu_var(ipipe_percpu_daddr)[(ipd)->slot]) +#endif +#define ipipe_percpudom(ipd, var, cpu) (ipipe_percpudom_ptr(ipd, cpu)->var) +#define ipipe_cpudom_var(ipd, var) (ipipe_cpudom_ptr(ipd)->var) + +#define IPIPE_ROOT_SLOT 0 +#define IPIPE_HEAD_SLOT (CONFIG_IPIPE_DOMAINS - 1) + +DECLARE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]); + +DECLARE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain); + +DECLARE_PER_CPU(unsigned long, ipipe_nmi_saved_root); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT +DECLARE_PER_CPU(int, ipipe_percpu_context_check); +DECLARE_PER_CPU(int, ipipe_saved_context_check_state); +#endif + +#define ipipe_root_cpudom_ptr(var) \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT]) + +#define ipipe_root_cpudom_var(var) ipipe_root_cpudom_ptr()->var + +#define ipipe_this_cpudom_var(var) \ + ipipe_cpudom_var(__ipipe_current_domain, var) + +#define ipipe_head_cpudom_ptr() \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_HEAD_SLOT]) + +#define ipipe_head_cpudom_var(var) ipipe_head_cpudom_ptr()->var + +#endif /* !__LINUX_IPIPE_PERCPU_H */ diff --git a/include/linux/ipipe_tickdev.h b/include/linux/ipipe_tickdev.h new file mode 100644 index 0000000..4a1cb1b --- /dev/null +++ b/include/linux/ipipe_tickdev.h @@ -0,0 +1,58 @@ +/* -*- linux-c -*- + * include/linux/ipipe_tickdev.h + * + * Copyright (C) 2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_TICKDEV_H +#define __LINUX_IPIPE_TICKDEV_H + +#if defined(CONFIG_IPIPE) && defined(CONFIG_GENERIC_CLOCKEVENTS) + +#include <linux/clockchips.h> + +struct tick_device; + +struct ipipe_tick_device { + + void (*emul_set_mode)(enum clock_event_mode, + struct clock_event_device *cdev); + int (*emul_set_tick)(unsigned long delta, + struct clock_event_device *cdev); + void (*real_set_mode)(enum clock_event_mode mode, + struct clock_event_device *cdev); + int (*real_set_tick)(unsigned long delta, + struct clock_event_device *cdev); + struct tick_device *slave; + unsigned long real_max_delta_ns; + unsigned long real_mult; + int real_shift; +}; + +int ipipe_request_tickdev(const char *devname, + void (*emumode)(enum clock_event_mode mode, + struct clock_event_device *cdev), + int (*emutick)(unsigned long evt, + struct clock_event_device *cdev), + int cpu, unsigned long *tmfreq); + +void ipipe_release_tickdev(int cpu); + +#endif /* CONFIG_IPIPE && CONFIG_GENERIC_CLOCKEVENTS */ + +#endif /* !__LINUX_IPIPE_TICKDEV_H */ diff --git a/include/linux/ipipe_trace.h b/include/linux/ipipe_trace.h new file mode 100644 index 0000000..627b354 --- /dev/null +++ b/include/linux/ipipe_trace.h @@ -0,0 +1,72 @@ +/* -*- linux-c -*- + * include/linux/ipipe_trace.h + * + * Copyright (C) 2005 Luotao Fu. + * 2005-2007 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef _LINUX_IPIPE_TRACE_H +#define _LINUX_IPIPE_TRACE_H + +#ifdef CONFIG_IPIPE_TRACE + +#include <linux/types.h> + +void ipipe_trace_begin(unsigned long v); +void ipipe_trace_end(unsigned long v); +void ipipe_trace_freeze(unsigned long v); +void ipipe_trace_special(unsigned char special_id, unsigned long v); +void ipipe_trace_pid(pid_t pid, short prio); +void ipipe_trace_event(unsigned char id, unsigned long delay_tsc); +int ipipe_trace_max_reset(void); +int ipipe_trace_frozen_reset(void); + +#else /* !CONFIG_IPIPE_TRACE */ + +#define ipipe_trace_begin(v) do { (void)(v); } while(0) +#define ipipe_trace_end(v) do { (void)(v); } while(0) +#define ipipe_trace_freeze(v) do { (void)(v); } while(0) +#define ipipe_trace_special(id, v) do { (void)(id); (void)(v); } while(0) +#define ipipe_trace_pid(pid, prio) do { (void)(pid); (void)(prio); } while(0) +#define ipipe_trace_event(id, delay_tsc) do { (void)(id); (void)(delay_tsc); } while(0) +#define ipipe_trace_max_reset() do { } while(0) +#define ipipe_trace_froze_reset() do { } while(0) + +#endif /* !CONFIG_IPIPE_TRACE */ + +#ifdef CONFIG_IPIPE_TRACE_PANIC +void ipipe_trace_panic_freeze(void); +void ipipe_trace_panic_dump(void); +#else +static inline void ipipe_trace_panic_freeze(void) { } +static inline void ipipe_trace_panic_dump(void) { } +#endif + +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF +#define ipipe_trace_irq_entry(irq) ipipe_trace_begin(irq) +#define ipipe_trace_irq_exit(irq) ipipe_trace_end(irq) +#define ipipe_trace_irqsoff() ipipe_trace_begin(0x80000000UL) +#define ipipe_trace_irqson() ipipe_trace_end(0x80000000UL) +#else +#define ipipe_trace_irq_entry(irq) do { (void)(irq);} while(0) +#define ipipe_trace_irq_exit(irq) do { (void)(irq);} while(0) +#define ipipe_trace_irqsoff() do { } while(0) +#define ipipe_trace_irqson() do { } while(0) +#endif + +#endif /* !__LINUX_IPIPE_TRACE_H */ diff --git a/include/linux/irq.h b/include/linux/irq.h index 451481c..3dbfa17 100644 --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -124,6 +124,9 @@ struct irq_chip { void (*end)(unsigned int irq); int (*set_affinity)(unsigned int irq, const struct cpumask *dest); +#ifdef CONFIG_IPIPE + void (*move)(unsigned int irq); +#endif /* CONFIG_IPIPE */ int (*retrigger)(unsigned int irq); int (*set_type)(unsigned int irq, unsigned int flow_type); int (*set_wake)(unsigned int irq, unsigned int on); @@ -173,6 +176,12 @@ struct irq_2_iommu; * @name: flow handler name for /proc/interrupts output */ struct irq_desc { +#ifdef CONFIG_IPIPE + void (*ipipe_ack)(unsigned int irq, + struct irq_desc *desc); + void (*ipipe_end)(unsigned int irq, + struct irq_desc *desc); +#endif /* CONFIG_IPIPE */ unsigned int irq; struct timer_rand_state *timer_rand_state; unsigned int *kstat_irqs; @@ -346,6 +355,10 @@ extern void set_irq_chip_and_handler_name(unsigned int irq, struct irq_chip *chip, irq_flow_handler_t handle, const char *name); +extern irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, + int is_chained); + extern void __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, const char *name); @@ -357,6 +370,7 @@ static inline void __set_irq_handler_unlocked(int irq, struct irq_desc *desc; desc = irq_to_desc(irq); + handler = __fixup_irq_handler(desc, handler, 0); desc->handle_irq = handler; } diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 328bca6..4ed869d 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -14,6 +14,7 @@ #include <linux/compiler.h> #include <linux/bitops.h> #include <linux/log2.h> +#include <linux/ipipe_base.h> #include <linux/typecheck.h> #include <linux/dynamic_debug.h> #include <asm/byteorder.h> @@ -118,9 +119,12 @@ struct user; #ifdef CONFIG_PREEMPT_VOLUNTARY extern int _cond_resched(void); -# define might_resched() _cond_resched() +# define might_resched() do { \ + ipipe_check_context(ipipe_root_domain); \ + _cond_resched(); \ + } while (0) #else -# define might_resched() do { } while (0) +# define might_resched() ipipe_check_context(ipipe_root_domain) #endif #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 9ccf0e2..f500f1e 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -395,7 +395,7 @@ do { \ #endif /* CONFIG_LOCK_STAT */ -#ifdef CONFIG_LOCKDEP +#if defined (CONFIG_LOCKDEP) || defined(CONFIG_IPIPE) /* * On lockdep we dont want the hand-coded irq-enable of diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 2e681d9..130b7d5 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -9,13 +9,20 @@ #include <linux/thread_info.h> #include <linux/linkage.h> #include <linux/list.h> +#include <linux/ipipe_base.h> #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER) extern void add_preempt_count(int val); extern void sub_preempt_count(int val); #else -# define add_preempt_count(val) do { preempt_count() += (val); } while (0) -# define sub_preempt_count(val) do { preempt_count() -= (val); } while (0) +# define add_preempt_count(val) do { \ + ipipe_check_context(ipipe_root_domain); \ + preempt_count() += (val); \ + } while (0) +# define sub_preempt_count(val) do { \ + ipipe_check_context(ipipe_root_domain); \ + preempt_count() -= (val); \ + } while (0) #endif #define inc_preempt_count() add_preempt_count(1) diff --git a/include/linux/resource.h b/include/linux/resource.h index f1e914e..bc3dfa0 100644 --- a/include/linux/resource.h +++ b/include/linux/resource.h @@ -51,12 +51,6 @@ struct rlimit { #define PRIO_USER 2 /* - * Limit the stack by to some sane default: root can always - * increase this limit if needed.. 8MB seems reasonable. - */ -#define _STK_LIM (8*1024*1024) - -/* * GPG2 wants 64kB of mlocked memory, to make sure pass phrases * and other sensitive information are never written to disk. */ diff --git a/include/linux/rwlock.h b/include/linux/rwlock.h index 71e0b00..8119f0c 100644 --- a/include/linux/rwlock.h +++ b/include/linux/rwlock.h @@ -61,8 +61,8 @@ do { \ #define read_trylock(lock) __cond_lock(lock, _raw_read_trylock(lock)) #define write_trylock(lock) __cond_lock(lock, _raw_write_trylock(lock)) -#define write_lock(lock) _raw_write_lock(lock) -#define read_lock(lock) _raw_read_lock(lock) +#define write_lock(lock) PICK_RWOP(_write_lock, lock) +#define read_lock(lock) PICK_RWOP(_read_lock, lock) #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) @@ -96,8 +96,8 @@ do { \ #define read_lock_bh(lock) _raw_read_lock_bh(lock) #define write_lock_irq(lock) _raw_write_lock_irq(lock) #define write_lock_bh(lock) _raw_write_lock_bh(lock) -#define read_unlock(lock) _raw_read_unlock(lock) -#define write_unlock(lock) _raw_write_unlock(lock) +#define read_unlock(lock) PICK_RWOP(_read_unlock, lock) +#define write_unlock(lock) PICK_RWOP(_write_unlock, lock) #define read_unlock_irq(lock) _raw_read_unlock_irq(lock) #define write_unlock_irq(lock) _raw_write_unlock_irq(lock) diff --git a/include/linux/rwlock_api_smp.h b/include/linux/rwlock_api_smp.h index 9c9f049..62c8941 100644 --- a/include/linux/rwlock_api_smp.h +++ b/include/linux/rwlock_api_smp.h @@ -141,7 +141,9 @@ static inline int __raw_write_trylock(rwlock_t *lock) * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) static inline void __raw_read_lock(rwlock_t *lock) { diff --git a/include/linux/sched.h b/include/linux/sched.h index 78efe7c..c2eb9a4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -61,6 +61,7 @@ struct sched_param { #include <linux/errno.h> #include <linux/nodemask.h> #include <linux/mm_types.h> +#include <linux/ipipe.h> #include <asm/system.h> #include <asm/page.h> @@ -192,9 +193,17 @@ print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) #define TASK_DEAD 64 #define TASK_WAKEKILL 128 #define TASK_WAKING 256 +#ifdef CONFIG_IPIPE +#define TASK_ATOMICSWITCH 512 +#define TASK_NOWAKEUP 1024 +#define TASK_STATE_MAX 2048 +#define TASK_STATE_TO_CHAR_STR "RSDTtZXxKWAN" +#else /* !CONFIG_IPIPE */ +#define TASK_ATOMICSWITCH 0 +#define TASK_NOWAKEUP 0 #define TASK_STATE_MAX 512 - #define TASK_STATE_TO_CHAR_STR "RSDTtZXxKW" +#endif /* CONFIG_IPIPE */ extern char ___assert_task_state[1 - 2*!!( sizeof(TASK_STATE_TO_CHAR_STR)-1 != ilog2(TASK_STATE_MAX)+1)]; @@ -305,6 +314,15 @@ extern void trap_init(void); extern void update_process_times(int user); extern void scheduler_tick(void); +#ifdef CONFIG_IPIPE +void update_root_process_times(struct pt_regs *regs); +#else /* !CONFIG_IPIPE */ +static inline void update_root_process_times(struct pt_regs *regs) +{ + update_process_times(user_mode(regs)); +} +#endif /* CONFIG_IPIPE */ + extern void sched_show_task(struct task_struct *p); #ifdef CONFIG_DETECT_SOFTLOCKUP @@ -356,7 +374,7 @@ extern signed long schedule_timeout(signed long timeout); extern signed long schedule_timeout_interruptible(signed long timeout); extern signed long schedule_timeout_killable(signed long timeout); extern signed long schedule_timeout_uninterruptible(signed long timeout); -asmlinkage void schedule(void); +asmlinkage int schedule(void); extern int mutex_spin_on_owner(struct mutex *lock, struct thread_info *owner); struct nsproxy; @@ -486,6 +504,9 @@ extern int get_dumpable(struct mm_struct *mm); #endif /* leave room for more dump flags */ #define MMF_VM_MERGEABLE 16 /* KSM may merge identical pages */ +#ifdef CONFIG_IPIPE +#define MMF_VM_PINNED 31 /* ondemand load up and COW disabled */ +#endif #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK) @@ -1512,6 +1533,9 @@ struct task_struct { #endif atomic_t fs_excl; /* holding fs exclusive resources */ struct rcu_head rcu; +#ifdef CONFIG_IPIPE + void *ptd[IPIPE_ROOT_NPTDKEYS]; +#endif /* * cache last used pipe for splice @@ -1759,6 +1783,11 @@ extern void thread_group_times(struct task_struct *p, cputime_t *ut, cputime_t * #define PF_EXITING 0x00000004 /* getting shut down */ #define PF_EXITPIDONE 0x00000008 /* pi exit done on shut down */ #define PF_VCPU 0x00000010 /* I'm a virtual CPU */ +#ifdef CONFIG_IPIPE +#define PF_EVNOTIFY 0x00000020 /* Notify other domains about internal events */ +#else +#define PF_EVNOTIFY 0 +#endif /* CONFIG_IPIPE */ #define PF_FORKNOEXEC 0x00000040 /* forked but didn't exec */ #define PF_MCE_PROCESS 0x00000080 /* process policy on mce errors */ #define PF_SUPERPRIV 0x00000100 /* used super-user privileges */ diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 8608821..c93ec16 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -88,6 +88,8 @@ # include <linux/spinlock_up.h> #endif +#include <linux/ipipe_lock.h> + #ifdef CONFIG_DEBUG_SPINLOCK extern void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name, struct lock_class_key *key); @@ -270,16 +272,9 @@ static inline raw_spinlock_t *spinlock_check(spinlock_t *lock) return &lock->rlock; } -#define spin_lock_init(_lock) \ -do { \ - spinlock_check(_lock); \ - raw_spin_lock_init(&(_lock)->rlock); \ -} while (0) +#define spin_lock_init(_lock) PICK_SPINOP(_lock_init, _lock) -static inline void spin_lock(spinlock_t *lock) -{ - raw_spin_lock(&lock->rlock); -} +#define spin_lock(lock) PICK_SPINOP(_lock, lock) static inline void spin_lock_bh(spinlock_t *lock) { @@ -288,7 +283,7 @@ static inline void spin_lock_bh(spinlock_t *lock) static inline int spin_trylock(spinlock_t *lock) { - return raw_spin_trylock(&lock->rlock); + return PICK_SPINOP_RET(_trylock, lock, int); } #define spin_lock_nested(lock, subclass) \ @@ -301,40 +296,27 @@ do { \ raw_spin_lock_nest_lock(spinlock_check(lock), nest_lock); \ } while (0) -static inline void spin_lock_irq(spinlock_t *lock) -{ - raw_spin_lock_irq(&lock->rlock); -} +#define spin_lock_irq(lock) PICK_SPINOP(_lock_irq, lock) -#define spin_lock_irqsave(lock, flags) \ -do { \ - raw_spin_lock_irqsave(spinlock_check(lock), flags); \ -} while (0) +#define spin_lock_irqsave(lock, flags) \ + PICK_SPINLOCK_IRQSAVE(lock, flags) #define spin_lock_irqsave_nested(lock, flags, subclass) \ do { \ raw_spin_lock_irqsave_nested(spinlock_check(lock), flags, subclass); \ } while (0) -static inline void spin_unlock(spinlock_t *lock) -{ - raw_spin_unlock(&lock->rlock); -} +#define spin_unlock(lock) PICK_SPINOP(_unlock, lock) static inline void spin_unlock_bh(spinlock_t *lock) { raw_spin_unlock_bh(&lock->rlock); } -static inline void spin_unlock_irq(spinlock_t *lock) -{ - raw_spin_unlock_irq(&lock->rlock); -} +#define spin_unlock_irq(lock) PICK_SPINOP(_unlock_irq, lock) -static inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) -{ - raw_spin_unlock_irqrestore(&lock->rlock, flags); -} +#define spin_unlock_irqrestore(lock, flags) \ + PICK_SPINUNLOCK_IRQRESTORE(lock, flags) static inline int spin_trylock_bh(spinlock_t *lock) { @@ -347,9 +329,7 @@ static inline int spin_trylock_irq(spinlock_t *lock) } #define spin_trylock_irqsave(lock, flags) \ -({ \ - raw_spin_trylock_irqsave(spinlock_check(lock), flags); \ -}) + PICK_SPINTRYLOCK_IRQSAVE(lock, flags) static inline void spin_unlock_wait(spinlock_t *lock) { diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_smp.h index e253ccd..378e01e 100644 --- a/include/linux/spinlock_api_smp.h +++ b/include/linux/spinlock_api_smp.h @@ -99,7 +99,9 @@ static inline int __raw_spin_trylock(raw_spinlock_t *lock) * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock) { @@ -113,7 +115,7 @@ static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock) * do_raw_spin_lock_flags() code, because lockdep assumes * that interrupts are not re-enabled during lock-acquire: */ -#ifdef CONFIG_LOCKDEP +#if defined(CONFIG_LOCKDEP) || defined(CONFIG_IPIPE) LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); #else do_raw_spin_lock_flags(lock, &flags); diff --git a/include/linux/spinlock_up.h b/include/linux/spinlock_up.h index b14f6a9..e400972 100644 --- a/include/linux/spinlock_up.h +++ b/include/linux/spinlock_up.h @@ -49,13 +49,6 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock) /* * Read-write spinlocks. No debug version. */ -#define arch_read_lock(lock) do { (void)(lock); } while (0) -#define arch_write_lock(lock) do { (void)(lock); } while (0) -#define arch_read_trylock(lock) ({ (void)(lock); 1; }) -#define arch_write_trylock(lock) ({ (void)(lock); 1; }) -#define arch_read_unlock(lock) do { (void)(lock); } while (0) -#define arch_write_unlock(lock) do { (void)(lock); } while (0) - #else /* DEBUG_SPINLOCK */ #define arch_spin_is_locked(lock) ((void)(lock), 0) /* for sched.c and kernel_lock.c: */ @@ -65,6 +58,13 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock) # define arch_spin_trylock(lock) ({ (void)(lock); 1; }) #endif /* DEBUG_SPINLOCK */ +#define arch_read_lock(lock) do { (void)(lock); } while (0) +#define arch_write_lock(lock) do { (void)(lock); } while (0) +#define arch_read_trylock(lock) ({ (void)(lock); 1; }) +#define arch_write_trylock(lock) ({ (void)(lock); 1; }) +#define arch_read_unlock(lock) do { (void)(lock); } while (0) +#define arch_write_unlock(lock) do { (void)(lock); } while (0) + #define arch_spin_is_contended(lock) (((void)(lock), 0)) #define arch_read_can_lock(lock) (((void)(lock), 1)) diff --git a/include/linux/utsrelease.h b/include/linux/utsrelease.h new file mode 100644 index 0000000..a7e4640 --- /dev/null +++ b/include/linux/utsrelease.h @@ -0,0 +1 @@ +#define UTS_RELEASE "2.6.32.7" diff --git a/init/Kconfig b/init/Kconfig index d95ca7c..1c6c3bf 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -78,6 +78,7 @@ config INIT_ENV_ARG_LIMIT config LOCALVERSION string "Local version - append to kernel release" + default "-ipipe" help Append an extra string to the end of your kernel version. This will show up when you type uname, for example. diff --git a/init/main.c b/init/main.c index 4cb47a1..e714997 100644 --- a/init/main.c +++ b/init/main.c @@ -530,7 +530,7 @@ asmlinkage void __init start_kernel(void) cgroup_init_early(); - local_irq_disable(); + local_irq_disable_hw(); early_boot_irqs_off(); early_init_irq_lock_class(); @@ -593,6 +593,11 @@ asmlinkage void __init start_kernel(void) softirq_init(); timekeeping_init(); time_init(); + /* + * We need to wait for the interrupt and time subsystems to be + * initialized before enabling the pipeline. + */ + ipipe_init(); profile_init(); if (!irqs_disabled()) printk(KERN_CRIT "start_kernel(): bug: interrupts were " @@ -774,6 +779,7 @@ static void __init do_basic_setup(void) init_tmpfs(); driver_init(); init_irq_proc(); + ipipe_init_proc(); do_ctors(); do_initcalls(); } diff --git a/kernel/Makefile b/kernel/Makefile index 864ff75..864aa99 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -85,6 +85,7 @@ obj-$(CONFIG_TREE_PREEMPT_RCU) += rcutree.o obj-$(CONFIG_TREE_RCU_TRACE) += rcutree_trace.o obj-$(CONFIG_TINY_RCU) += rcutiny.o obj-$(CONFIG_RELAY) += relay.o +obj-$(CONFIG_IPIPE) += ipipe/ obj-$(CONFIG_SYSCTL) += utsname_sysctl.o obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o diff --git a/kernel/exit.c b/kernel/exit.c index 546774a..1c3531a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -967,6 +967,7 @@ NORET_TYPE void do_exit(long code) acct_process(); trace_sched_process_exit(tsk); + ipipe_exit_notify(tsk); exit_sem(tsk); exit_files(tsk); exit_fs(tsk); diff --git a/kernel/fork.c b/kernel/fork.c index f88bd98..545fd6e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -513,6 +513,7 @@ void mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); exit_mmap(mm); + ipipe_cleanup_notify(mm); set_mm_exe_file(mm, NULL); if (!list_empty(&mm->mmlist)) { spin_lock(&mmlist_lock); @@ -923,7 +924,7 @@ static void copy_flags(unsigned long clone_flags, struct task_struct *p) { unsigned long new_flags = p->flags; - new_flags &= ~PF_SUPERPRIV; + new_flags &= ~(PF_SUPERPRIV | PF_EVNOTIFY); new_flags |= PF_FORKNOEXEC; new_flags |= PF_STARTING; p->flags = new_flags; @@ -1300,6 +1301,9 @@ static struct task_struct *copy_process(unsigned long clone_flags, write_unlock_irq(&tasklist_lock); proc_fork_connector(p); cgroup_post_fork(p); +#ifdef CONFIG_IPIPE + memset(p->ptd, 0, sizeof(p->ptd)); +#endif /* CONFIG_IPIPE */ perf_event_fork(p); return p; @@ -1698,11 +1702,14 @@ SYSCALL_DEFINE1(unshare, unsigned long, unshare_flags) } if (new_mm) { + unsigned long flags; mm = current->mm; active_mm = current->active_mm; current->mm = new_mm; + ipipe_mm_switch_protect(flags); current->active_mm = new_mm; activate_mm(active_mm, new_mm); + ipipe_mm_switch_unprotect(flags); new_mm = mm; } diff --git a/kernel/ipipe/Kconfig b/kernel/ipipe/Kconfig new file mode 100644 index 0000000..8ff9d37 --- /dev/null +++ b/kernel/ipipe/Kconfig @@ -0,0 +1,23 @@ +config IPIPE + bool "Interrupt pipeline" + default y + ---help--- + Activate this option if you want the interrupt pipeline to be + compiled in. + +config IPIPE_DOMAINS + int "Max domains" + depends on IPIPE + default 4 + ---help--- + The maximum number of I-pipe domains to run concurrently. + +config IPIPE_DELAYED_ATOMICSW + bool + depends on IPIPE + default n + +config IPIPE_UNMASKED_CONTEXT_SWITCH + bool + depends on IPIPE + default n diff --git a/kernel/ipipe/Kconfig.debug b/kernel/ipipe/Kconfig.debug new file mode 100644 index 0000000..629c894 --- /dev/null +++ b/kernel/ipipe/Kconfig.debug @@ -0,0 +1,97 @@ +config IPIPE_DEBUG + bool "I-pipe debugging" + depends on IPIPE + +config IPIPE_DEBUG_CONTEXT + bool "Check for illicit cross-domain calls" + depends on IPIPE_DEBUG + default y + ---help--- + Enable this feature to arm checkpoints in the kernel that + verify the correct invocation context. On entry of critical + Linux services a warning is issued if the caller is not + running over the root domain. + +config IPIPE_DEBUG_INTERNAL + bool "Enable internal debug checks" + depends on IPIPE_DEBUG + default y + ---help--- + When this feature is enabled, I-pipe will perform internal + consistency checks of its subsystems, e.g. on per-cpu variable + access. + +config IPIPE_TRACE + bool "Latency tracing" + depends on IPIPE_DEBUG + select FRAME_POINTER + select KALLSYMS + select PROC_FS + ---help--- + Activate this option if you want to use per-function tracing of + the kernel. The tracer will collect data via instrumentation + features like the one below or with the help of explicite calls + of ipipe_trace_xxx(). See include/linux/ipipe_trace.h for the + in-kernel tracing API. The collected data and runtime control + is available via /proc/ipipe/trace/*. + +if IPIPE_TRACE + +config IPIPE_TRACE_ENABLE + bool "Enable tracing on boot" + default y + ---help--- + Disable this option if you want to arm the tracer after booting + manually ("echo 1 > /proc/ipipe/tracer/enable"). This can reduce + boot time on slow embedded devices due to the tracer overhead. + +config IPIPE_TRACE_MCOUNT + bool "Instrument function entries" + default y + select FUNCTION_TRACER + select TRACING + select CONTEXT_SWITCH_TRACER + select FTRACE_MCOUNT_RECORD + select DYNAMIC_FTRACE + ---help--- + When enabled, records every kernel function entry in the tracer + log. While this slows down the system noticeably, it provides + the highest level of information about the flow of events. + However, it can be switch off in order to record only explicit + I-pipe trace points. + +config IPIPE_TRACE_IRQSOFF + bool "Trace IRQs-off times" + default y + ---help--- + Activate this option if I-pipe shall trace the longest path + with hard-IRQs switched off. + +config IPIPE_TRACE_SHIFT + int "Depth of trace log (14 => 16Kpoints, 15 => 32Kpoints)" + range 10 18 + default 14 + ---help--- + The number of trace points to hold tracing data for each + trace path, as a power of 2. + +config IPIPE_TRACE_VMALLOC + bool "Use vmalloc'ed trace buffer" + default y if EMBEDDED + ---help--- + Instead of reserving static kernel data, the required buffer + is allocated via vmalloc during boot-up when this option is + enabled. This can help to start systems that are low on memory, + but it slightly degrades overall performance. Try this option + when a traced kernel hangs unexpectedly at boot time. + +config IPIPE_TRACE_PANIC + bool "Enable panic back traces" + default y + ---help--- + Provides services to freeze and dump a back trace on panic + situations. This is used on IPIPE_DEBUG_CONTEXT exceptions + as well as ordinary kernel oopses. You can control the number + of printed back trace points via /proc/ipipe/trace. + +endif diff --git a/kernel/ipipe/Makefile b/kernel/ipipe/Makefile new file mode 100644 index 0000000..6257dfa --- /dev/null +++ b/kernel/ipipe/Makefile @@ -0,0 +1,3 @@ + +obj-$(CONFIG_IPIPE) += core.o +obj-$(CONFIG_IPIPE_TRACE) += tracer.o diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c new file mode 100644 index 0000000..af1f7e2 --- /dev/null +++ b/kernel/ipipe/core.c @@ -0,0 +1,1970 @@ +/* -*- linux-c -*- + * linux/kernel/ipipe/core.c + * + * Copyright (C) 2002-2005 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Architecture-independent I-PIPE core support. + */ + +#include <linux/version.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/kernel.h> +#include <linux/sched.h> +#include <linux/sched.h> +#include <linux/kallsyms.h> +#include <linux/interrupt.h> +#include <linux/bitops.h> +#include <linux/tick.h> +#include <linux/prefetch.h> +#ifdef CONFIG_PROC_FS +#include <linux/proc_fs.h> +#include <linux/seq_file.h> +#endif /* CONFIG_PROC_FS */ +#include <linux/ipipe_trace.h> +#include <linux/ipipe_tickdev.h> +#include <linux/irq.h> + +static int __ipipe_ptd_key_count; + +static unsigned long __ipipe_ptd_key_map; + +static unsigned long __ipipe_domain_slot_map; + +struct ipipe_domain ipipe_root; + +#ifndef CONFIG_SMP +/* + * Create an alias to the unique root status, so that arch-dep code + * may get simple and easy access to this percpu variable. We also + * create an array of pointers to the percpu domain data; this tends + * to produce a better code when reaching non-root domains. We make + * sure that the early boot code would be able to dereference the + * pointer to the root domain data safely by statically initializing + * its value (local_irq*() routines depend on this). + */ +#if __GNUC__ >= 4 +extern unsigned long __ipipe_root_status +__attribute__((alias(__stringify(__raw_get_cpu_var(ipipe_percpu_darray))))); +EXPORT_SYMBOL(__ipipe_root_status); +#else /* __GNUC__ < 4 */ +/* + * Work around a GCC 3.x issue making alias symbols unusable as + * constant initializers. + */ +unsigned long *const __ipipe_root_status_addr = + &__raw_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT].status; +EXPORT_SYMBOL(__ipipe_root_status_addr); +#endif /* __GNUC__ < 4 */ + +DEFINE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]) = +{ [IPIPE_ROOT_SLOT] = (struct ipipe_percpu_domain_data *)&__raw_get_cpu_var(ipipe_percpu_darray) }; +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_daddr); +#endif /* !CONFIG_SMP */ + +DEFINE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]) = +{ [IPIPE_ROOT_SLOT] = { .status = IPIPE_STALL_MASK } }; /* Root domain stalled on each CPU at startup. */ + +DEFINE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain) = { &ipipe_root }; + +DEFINE_PER_CPU(unsigned long, ipipe_nmi_saved_root); /* Copy of root status during NMI */ + +static IPIPE_DEFINE_SPINLOCK(__ipipe_pipelock); + +LIST_HEAD(__ipipe_pipeline); + +unsigned long __ipipe_virtual_irq_map; + +#ifdef CONFIG_PRINTK +unsigned __ipipe_printk_virq; +#endif /* CONFIG_PRINTK */ + +int __ipipe_event_monitors[IPIPE_NR_EVENTS]; + +#ifdef CONFIG_GENERIC_CLOCKEVENTS + +DECLARE_PER_CPU(struct tick_device, tick_cpu_device); + +static DEFINE_PER_CPU(struct ipipe_tick_device, ipipe_tick_cpu_device); + +int ipipe_request_tickdev(const char *devname, + void (*emumode)(enum clock_event_mode mode, + struct clock_event_device *cdev), + int (*emutick)(unsigned long delta, + struct clock_event_device *cdev), + int cpu, unsigned long *tmfreq) +{ + struct ipipe_tick_device *itd; + struct tick_device *slave; + struct clock_event_device *evtdev; + unsigned long long freq; + unsigned long flags; + int status; + + flags = ipipe_critical_enter(NULL); + + itd = &per_cpu(ipipe_tick_cpu_device, cpu); + + if (itd->slave != NULL) { + status = -EBUSY; + goto out; + } + + slave = &per_cpu(tick_cpu_device, cpu); + + if (strcmp(slave->evtdev->name, devname)) { + /* + * No conflict so far with the current tick device, + * check whether the requested device is sane and has + * been blessed by the kernel. + */ + status = __ipipe_check_tickdev(devname) ? + CLOCK_EVT_MODE_UNUSED : CLOCK_EVT_MODE_SHUTDOWN; + goto out; + } + + /* + * Our caller asks for using the same clock event device for + * ticking than we do, let's create a tick emulation device to + * interpose on the set_next_event() method, so that we may + * both manage the device in oneshot mode. Only the tick + * emulation code will actually program the clockchip hardware + * for the next shot, though. + * + * CAUTION: we still have to grab the tick device even when it + * current runs in periodic mode, since the kernel may switch + * to oneshot dynamically (highres/no_hz tick mode). + */ + + evtdev = slave->evtdev; + status = evtdev->mode; + + if (status == CLOCK_EVT_MODE_SHUTDOWN) + goto out; + + itd->slave = slave; + itd->emul_set_mode = emumode; + itd->emul_set_tick = emutick; + itd->real_set_mode = evtdev->set_mode; + itd->real_set_tick = evtdev->set_next_event; + itd->real_max_delta_ns = evtdev->max_delta_ns; + itd->real_mult = evtdev->mult; + itd->real_shift = evtdev->shift; + freq = (1000000000ULL * evtdev->mult) >> evtdev->shift; + *tmfreq = (unsigned long)freq; + evtdev->set_mode = emumode; + evtdev->set_next_event = emutick; + evtdev->max_delta_ns = ULONG_MAX; + evtdev->mult = 1; + evtdev->shift = 0; +out: + ipipe_critical_exit(flags); + + return status; +} + +void ipipe_release_tickdev(int cpu) +{ + struct ipipe_tick_device *itd; + struct tick_device *slave; + struct clock_event_device *evtdev; + unsigned long flags; + + flags = ipipe_critical_enter(NULL); + + itd = &per_cpu(ipipe_tick_cpu_device, cpu); + + if (itd->slave != NULL) { + slave = &per_cpu(tick_cpu_device, cpu); + evtdev = slave->evtdev; + evtdev->set_mode = itd->real_set_mode; + evtdev->set_next_event = itd->real_set_tick; + evtdev->max_delta_ns = itd->real_max_delta_ns; + evtdev->mult = itd->real_mult; + evtdev->shift = itd->real_shift; + itd->slave = NULL; + } + + ipipe_critical_exit(flags); +} + +#endif /* CONFIG_GENERIC_CLOCKEVENTS */ + +/* + * ipipe_init() -- Initialization routine of the IPIPE layer. Called + * by the host kernel early during the boot procedure. + */ +void __init ipipe_init(void) +{ + struct ipipe_domain *ipd = &ipipe_root; + + __ipipe_check_platform(); /* Do platform dependent checks first. */ + + /* + * A lightweight registration code for the root domain. We are + * running on the boot CPU, hw interrupts are off, and + * secondary CPUs are still lost in space. + */ + + /* Reserve percpu data slot #0 for the root domain. */ + ipd->slot = 0; + set_bit(0, &__ipipe_domain_slot_map); + + ipd->name = "Linux"; + ipd->domid = IPIPE_ROOT_ID; + ipd->priority = IPIPE_ROOT_PRIO; + + __ipipe_init_stage(ipd); + + INIT_LIST_HEAD(&ipd->p_link); + list_add_tail(&ipd->p_link, &__ipipe_pipeline); + + __ipipe_init_platform(); + +#ifdef CONFIG_PRINTK + __ipipe_printk_virq = ipipe_alloc_virq(); /* Cannot fail here. */ + ipd->irqs[__ipipe_printk_virq].handler = &__ipipe_flush_printk; + ipd->irqs[__ipipe_printk_virq].cookie = NULL; + ipd->irqs[__ipipe_printk_virq].acknowledge = NULL; + ipd->irqs[__ipipe_printk_virq].control = IPIPE_HANDLE_MASK; +#endif /* CONFIG_PRINTK */ + + __ipipe_enable_pipeline(); + + printk(KERN_INFO "I-pipe %s: pipeline enabled.\n", + IPIPE_VERSION_STRING); +} + +void __ipipe_init_stage(struct ipipe_domain *ipd) +{ + struct ipipe_percpu_domain_data *p; + unsigned long status; + int cpu, n; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + status = p->status; + memset(p, 0, sizeof(*p)); + p->status = status; + } + + for (n = 0; n < IPIPE_NR_IRQS; n++) { + ipd->irqs[n].acknowledge = NULL; + ipd->irqs[n].handler = NULL; + ipd->irqs[n].control = IPIPE_PASS_MASK; /* Pass but don't handle */ + } + + for (n = 0; n < IPIPE_NR_EVENTS; n++) + ipd->evhand[n] = NULL; + + ipd->evself = 0LL; + mutex_init(&ipd->mutex); + + __ipipe_hook_critical_ipi(ipd); +} + +void __ipipe_cleanup_domain(struct ipipe_domain *ipd) +{ + ipipe_unstall_pipeline_from(ipd); + +#ifdef CONFIG_SMP + { + struct ipipe_percpu_domain_data *p; + int cpu; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + while (__ipipe_ipending_p(p)) + cpu_relax(); + } + } +#else + __raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = NULL; +#endif + + clear_bit(ipd->slot, &__ipipe_domain_slot_map); +} + +void __ipipe_unstall_root(void) +{ + struct ipipe_percpu_domain_data *p; + + local_irq_disable_hw(); + +#ifdef CONFIG_IPIPE_DEBUG_INTERNAL + /* This helps catching bad usage from assembly call sites. */ + BUG_ON(!__ipipe_root_domain_p); +#endif + + p = ipipe_root_cpudom_ptr(); + + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (unlikely(__ipipe_ipending_p(p))) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + local_irq_enable_hw(); +} + +void __ipipe_restore_root(unsigned long x) +{ +#ifdef CONFIG_IPIPE_DEBUG_INTERNAL + BUG_ON(!ipipe_root_domain_p); +#endif + + if (x) + __ipipe_stall_root(); + else + __ipipe_unstall_root(); +} + +void ipipe_stall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags; + /* + * We have to prevent against race on updating the status + * variable _and_ CPU migration at the same time, so disable + * hw IRQs here. + */ + local_irq_save_hw(flags); + + __set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (!__ipipe_pipeline_head_p(ipd)) + local_irq_restore_hw(flags); +} + +unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags, x; + + /* See ipipe_stall_pipeline_from() */ + local_irq_save_hw(flags); + + x = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (!__ipipe_pipeline_head_p(ipd)) + local_irq_restore_hw(flags); + + return x; +} + +unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags, x; + struct list_head *pos; + + local_irq_save_hw(flags); + + x = __test_and_clear_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (ipd == __ipipe_current_domain) + pos = &ipd->p_link; + else + pos = __ipipe_pipeline.next; + + __ipipe_walk_pipeline(pos); + + if (likely(__ipipe_pipeline_head_p(ipd))) + local_irq_enable_hw(); + else + local_irq_restore_hw(flags); + + return x; +} + +void ipipe_restore_pipeline_from(struct ipipe_domain *ipd, + unsigned long x) +{ + if (x) + ipipe_stall_pipeline_from(ipd); + else + ipipe_unstall_pipeline_from(ipd); +} + +void ipipe_unstall_pipeline_head(void) +{ + struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr(); + struct ipipe_domain *head_domain; + + local_irq_disable_hw(); + + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (unlikely(__ipipe_ipending_p(p))) { + head_domain = __ipipe_pipeline_head(); + if (likely(head_domain == __ipipe_current_domain)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else + __ipipe_walk_pipeline(&head_domain->p_link); + } + + local_irq_enable_hw(); +} + +void __ipipe_restore_pipeline_head(unsigned long x) +{ + struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr(); + struct ipipe_domain *head_domain; + + local_irq_disable_hw(); + + if (x) { +#ifdef CONFIG_DEBUG_KERNEL + static int warned; + if (!warned && test_and_set_bit(IPIPE_STALL_FLAG, &p->status)) { + /* + * Already stalled albeit ipipe_restore_pipeline_head() + * should have detected it? Send a warning once. + */ + warned = 1; + printk(KERN_WARNING + "I-pipe: ipipe_restore_pipeline_head() optimization failed.\n"); + dump_stack(); + } +#else /* !CONFIG_DEBUG_KERNEL */ + set_bit(IPIPE_STALL_FLAG, &p->status); +#endif /* CONFIG_DEBUG_KERNEL */ + } + else { + __clear_bit(IPIPE_STALL_FLAG, &p->status); + if (unlikely(__ipipe_ipending_p(p))) { + head_domain = __ipipe_pipeline_head(); + if (likely(head_domain == __ipipe_current_domain)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else + __ipipe_walk_pipeline(&head_domain->p_link); + } + local_irq_enable_hw(); + } +} + +void __ipipe_spin_lock_irq(ipipe_spinlock_t *lock) +{ + local_irq_disable_hw(); + arch_spin_lock(&lock->arch_lock); + __set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); +} + +void __ipipe_spin_unlock_irq(ipipe_spinlock_t *lock) +{ + arch_spin_unlock(&lock->arch_lock); + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_enable_hw(); +} + +unsigned long __ipipe_spin_lock_irqsave(ipipe_spinlock_t *lock) +{ + unsigned long flags; + int s; + + local_irq_save_hw(flags); + arch_spin_lock(&lock->arch_lock); + s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + + return raw_mangle_irq_bits(s, flags); +} + +int __ipipe_spin_trylock_irqsave(ipipe_spinlock_t *lock, + unsigned long *x) +{ + unsigned long flags; + int s; + + local_irq_save_hw(flags); + if (!arch_spin_trylock(&lock->arch_lock)) { + local_irq_restore_hw(flags); + return 0; + } + s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + *x = raw_mangle_irq_bits(s, flags); + + return 1; +} + +void __ipipe_spin_unlock_irqrestore(ipipe_spinlock_t *lock, + unsigned long x) +{ + arch_spin_unlock(&lock->arch_lock); + if (!raw_demangle_irq_bits(&x)) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_restore_hw(x); +} + +void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock) +{ + arch_spin_unlock(&lock->arch_lock); +} + +void __ipipe_spin_unlock_irqcomplete(unsigned long x) +{ + if (!raw_demangle_irq_bits(&x)) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_restore_hw(x); +} + +#ifdef __IPIPE_3LEVEL_IRQMAP + +/* Must be called hw IRQs off. */ +static inline void __ipipe_set_irq_held(struct ipipe_percpu_domain_data *p, + unsigned int irq) +{ + __set_bit(irq, p->irqheld_map); + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd); + int l0b, l1b; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + prefetchw(p); + + if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) { + __set_bit(irq, p->irqpend_lomap); + __set_bit(l1b, p->irqpend_mdmap); + __set_bit(l0b, &p->irqpend_himap); + } else + __set_bit(irq, p->irqheld_map); + + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b, l1b; + + if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + + p = ipipe_percpudom_ptr(ipd, cpu); + if (__test_and_clear_bit(irq, p->irqpend_lomap)) { + __set_bit(irq, p->irqheld_map); + if (p->irqpend_lomap[l1b] == 0) { + __clear_bit(l1b, p->irqpend_mdmap); + if (p->irqpend_mdmap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } + } +} + +/* Must be called hw IRQs off. */ +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b, l1b, cpu; + + if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + if (test_and_clear_bit(irq, p->irqheld_map)) { + /* We need atomic ops here: */ + set_bit(irq, p->irqpend_lomap); + set_bit(l1b, p->irqpend_mdmap); + set_bit(l0b, &p->irqpend_himap); + } + } +} + +static inline int __ipipe_next_irq(struct ipipe_percpu_domain_data *p, + int dovirt) +{ + unsigned long l0m, l1m, l2m, himask, mdmask; + int l0b, l1b, l2b, vl0b, vl1b; + unsigned int irq; + + if (dovirt) { + /* + * All virtual IRQs are mapped by a single long word. + * There is exactly BITS_PER_LONG virqs, and they are + * always last in the interrupt map, starting at + * IPIPE_VIRQ_BASE. Therefore, we only need to test a + * single bit within the high and middle maps to check + * whether a virtual IRQ is pending (the computations + * below are constant). + */ + vl0b = IPIPE_VIRQ_BASE / (BITS_PER_LONG * BITS_PER_LONG); + himask = (1L << vl0b); + vl1b = IPIPE_VIRQ_BASE / BITS_PER_LONG; + mdmask = (1L << (vl1b & (BITS_PER_LONG-1))); + } else + himask = mdmask = ~0L; + + l0m = p->irqpend_himap & himask; + if (unlikely(l0m == 0)) + return -1; + + l0b = __ipipe_ffnz(l0m); + l1m = p->irqpend_mdmap[l0b] & mdmask; + if (unlikely(l1m == 0)) + return -1; + + l1b = __ipipe_ffnz(l1m) + l0b * BITS_PER_LONG; + l2m = p->irqpend_lomap[l1b]; + if (unlikely(l2m == 0)) + return -1; + + l2b = __ipipe_ffnz(l2m); + irq = l1b * BITS_PER_LONG + l2b; + + __clear_bit(irq, p->irqpend_lomap); + if (p->irqpend_lomap[l1b] == 0) { + __clear_bit(l1b, p->irqpend_mdmap); + if (p->irqpend_mdmap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } + + return irq; +} + +#else /* __IPIPE_2LEVEL_IRQMAP */ + +/* Must be called hw IRQs off. */ +static inline void __ipipe_set_irq_held(struct ipipe_percpu_domain_data *p, + unsigned int irq) +{ + __set_bit(irq, p->irqheld_map); + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd); + int l0b = irq / BITS_PER_LONG; + + prefetchw(p); + + if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) { + __set_bit(irq, p->irqpend_lomap); + __set_bit(l0b, &p->irqpend_himap); + } else + __set_bit(irq, p->irqheld_map); + + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b = irq / BITS_PER_LONG; + + if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + p = ipipe_percpudom_ptr(ipd, cpu); + if (__test_and_clear_bit(irq, p->irqpend_lomap)) { + __set_bit(irq, p->irqheld_map); + if (p->irqpend_lomap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } +} + +/* Must be called hw IRQs off. */ +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b = irq / BITS_PER_LONG, cpu; + + if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + if (test_and_clear_bit(irq, p->irqheld_map)) { + /* We need atomic ops here: */ + set_bit(irq, p->irqpend_lomap); + set_bit(l0b, &p->irqpend_himap); + } + } +} + +static inline int __ipipe_next_irq(struct ipipe_percpu_domain_data *p, + int dovirt) +{ + unsigned long l0m, l1m, himask = ~0L; + int l0b, l1b; + + himask <<= dovirt ? IPIPE_VIRQ_BASE/BITS_PER_LONG : 0; + + l0m = p->irqpend_himap & himask; + if (unlikely(l0m == 0)) + return -1; + + l0b = __ipipe_ffnz(l0m); + l1m = p->irqpend_lomap[l0b]; + if (unlikely(l1m == 0)) + return -1; + + l1b = __ipipe_ffnz(l1m); + __clear_bit(l1b, &p->irqpend_lomap[l0b]); + if (p->irqpend_lomap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + + return l0b * BITS_PER_LONG + l1b; +} + +#endif /* __IPIPE_2LEVEL_IRQMAP */ + +/* + * __ipipe_walk_pipeline(): Plays interrupts pending in the log. Must + * be called with local hw interrupts disabled. + */ +void __ipipe_walk_pipeline(struct list_head *pos) +{ + struct ipipe_domain *this_domain = __ipipe_current_domain, *next_domain; + struct ipipe_percpu_domain_data *p, *np; + + p = ipipe_cpudom_ptr(this_domain); + + while (pos != &__ipipe_pipeline) { + + next_domain = list_entry(pos, struct ipipe_domain, p_link); + np = ipipe_cpudom_ptr(next_domain); + + if (test_bit(IPIPE_STALL_FLAG, &np->status)) + break; /* Stalled stage -- do not go further. */ + + if (__ipipe_ipending_p(np)) { + if (next_domain == this_domain) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else { + + p->evsync = 0; + __ipipe_current_domain = next_domain; + ipipe_suspend_domain(); /* Sync stage and propagate interrupts. */ + + if (__ipipe_current_domain == next_domain) + __ipipe_current_domain = this_domain; + /* + * Otherwise, something changed the current domain under our + * feet recycling the register set; do not override the new + * domain. + */ + + if (__ipipe_ipending_p(p) && + !test_bit(IPIPE_STALL_FLAG, &p->status)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + } + break; + } else if (next_domain == this_domain) + break; + + pos = next_domain->p_link.next; + } +} + +/* + * ipipe_suspend_domain() -- Suspend the current domain, switching to + * the next one which has pending work down the pipeline. + */ +void ipipe_suspend_domain(void) +{ + struct ipipe_domain *this_domain, *next_domain; + struct ipipe_percpu_domain_data *p; + struct list_head *ln; + unsigned long flags; + + local_irq_save_hw(flags); + + this_domain = next_domain = __ipipe_current_domain; + p = ipipe_cpudom_ptr(this_domain); + p->status &= ~(IPIPE_STALL_MASK|IPIPE_SYNC_MASK); + + if (__ipipe_ipending_p(p)) + goto sync_stage; + + for (;;) { + ln = next_domain->p_link.next; + + if (ln == &__ipipe_pipeline) + break; + + next_domain = list_entry(ln, struct ipipe_domain, p_link); + p = ipipe_cpudom_ptr(next_domain); + + if (p->status & IPIPE_STALL_MASK) + break; + + if (!__ipipe_ipending_p(p)) + continue; + + __ipipe_current_domain = next_domain; +sync_stage: + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + if (__ipipe_current_domain != next_domain) + /* + * Something has changed the current domain under our + * feet, recycling the register set; take note. + */ + this_domain = __ipipe_current_domain; + } + + __ipipe_current_domain = this_domain; + + local_irq_restore_hw(flags); +} + + +/* ipipe_alloc_virq() -- Allocate a pipelined virtual/soft interrupt. + * Virtual interrupts are handled in exactly the same way than their + * hw-generated counterparts wrt pipelining. + */ +unsigned ipipe_alloc_virq(void) +{ + unsigned long flags, irq = 0; + int ipos; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + if (__ipipe_virtual_irq_map != ~0) { + ipos = ffz(__ipipe_virtual_irq_map); + set_bit(ipos, &__ipipe_virtual_irq_map); + irq = ipos + IPIPE_VIRQ_BASE; + } + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return irq; +} + +/* + * ipipe_control_irq() -- Change modes of a pipelined interrupt for + * the current domain. + */ +int ipipe_virtualize_irq(struct ipipe_domain *ipd, + unsigned irq, + ipipe_irq_handler_t handler, + void *cookie, + ipipe_irq_ackfn_t acknowledge, + unsigned modemask) +{ + ipipe_irq_handler_t old_handler; + struct irq_desc *desc; + unsigned long flags; + int err; + + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + + if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) + return -EPERM; + + if (!test_bit(IPIPE_AHEAD_FLAG, &ipd->flags)) + /* Silently unwire interrupts for non-heading domains. */ + modemask &= ~IPIPE_WIRED_MASK; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + old_handler = ipd->irqs[irq].handler; + + if (handler != NULL) { + if (handler == IPIPE_SAME_HANDLER) { + handler = old_handler; + cookie = ipd->irqs[irq].cookie; + + if (handler == NULL) { + err = -EINVAL; + goto unlock_and_exit; + } + } else if ((modemask & IPIPE_EXCLUSIVE_MASK) != 0 && + old_handler != NULL) { + err = -EBUSY; + goto unlock_and_exit; + } + + /* Wired interrupts can only be delivered to domains + * always heading the pipeline, and using dynamic + * propagation. */ + + if ((modemask & IPIPE_WIRED_MASK) != 0) { + if ((modemask & (IPIPE_PASS_MASK | IPIPE_STICKY_MASK)) != 0) { + err = -EINVAL; + goto unlock_and_exit; + } + modemask |= (IPIPE_HANDLE_MASK); + } + + if ((modemask & IPIPE_STICKY_MASK) != 0) + modemask |= IPIPE_HANDLE_MASK; + } else + modemask &= + ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK | + IPIPE_EXCLUSIVE_MASK | IPIPE_WIRED_MASK); + + if (acknowledge == NULL && !ipipe_virtual_irq_p(irq)) + /* + * Acknowledge handler unspecified for a hw interrupt: + * use the Linux-defined handler instead. + */ + acknowledge = ipipe_root_domain->irqs[irq].acknowledge; + + ipd->irqs[irq].handler = handler; + ipd->irqs[irq].cookie = cookie; + ipd->irqs[irq].acknowledge = acknowledge; + ipd->irqs[irq].control = modemask; + + if (irq < NR_IRQS && !ipipe_virtual_irq_p(irq)) { + desc = irq_to_desc(irq); + if (handler != NULL) { + if (desc) + __ipipe_enable_irqdesc(ipd, irq); + + if ((modemask & IPIPE_ENABLE_MASK) != 0) { + if (ipd != __ipipe_current_domain) { + /* + * IRQ enable/disable state is domain-sensitive, so we + * may not change it for another domain. What is + * allowed however is forcing some domain to handle an + * interrupt source, by passing the proper 'ipd' + * descriptor which thus may be different from + * __ipipe_current_domain. + */ + err = -EPERM; + goto unlock_and_exit; + } + if (desc) + __ipipe_enable_irq(irq); + } + } else if (old_handler != NULL && desc) + __ipipe_disable_irqdesc(ipd, irq); + } + + err = 0; + + unlock_and_exit: + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return err; +} + +/* ipipe_control_irq() -- Change modes of a pipelined interrupt for + * the current domain. */ + +int ipipe_control_irq(unsigned irq, unsigned clrmask, unsigned setmask) +{ + struct ipipe_domain *ipd; + unsigned long flags; + + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + ipd = __ipipe_current_domain; + + if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) { + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + return -EPERM; + } + + if (ipd->irqs[irq].handler == NULL) + setmask &= ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK); + + if ((setmask & IPIPE_STICKY_MASK) != 0) + setmask |= IPIPE_HANDLE_MASK; + + if ((clrmask & (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK)) != 0) /* If one goes, both go. */ + clrmask |= (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK); + + ipd->irqs[irq].control &= ~clrmask; + ipd->irqs[irq].control |= setmask; + + if ((setmask & IPIPE_ENABLE_MASK) != 0) + __ipipe_enable_irq(irq); + else if ((clrmask & IPIPE_ENABLE_MASK) != 0) + __ipipe_disable_irq(irq); + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return 0; +} + +/* __ipipe_dispatch_event() -- Low-level event dispatcher. */ + +int __ipipe_dispatch_event (unsigned event, void *data) +{ + struct ipipe_domain *start_domain, *this_domain, *next_domain; + struct ipipe_percpu_domain_data *np; + ipipe_event_handler_t evhand; + struct list_head *pos, *npos; + unsigned long flags; + int propagate = 1; + + local_irq_save_hw(flags); + + start_domain = this_domain = __ipipe_current_domain; + + list_for_each_safe(pos, npos, &__ipipe_pipeline) { + /* + * Note: Domain migration may occur while running + * event or interrupt handlers, in which case the + * current register set is going to be recycled for a + * different domain than the initiating one. We do + * care for that, always tracking the current domain + * descriptor upon return from those handlers. + */ + next_domain = list_entry(pos, struct ipipe_domain, p_link); + np = ipipe_cpudom_ptr(next_domain); + + /* + * Keep a cached copy of the handler's address since + * ipipe_catch_event() may clear it under our feet. + */ + evhand = next_domain->evhand[event]; + + if (evhand != NULL) { + __ipipe_current_domain = next_domain; + np->evsync |= (1LL << event); + local_irq_restore_hw(flags); + propagate = !evhand(event, start_domain, data); + local_irq_save_hw(flags); + /* + * We may have a migration issue here, if the + * current task is migrated to another CPU on + * behalf of the invoked handler, usually when + * a syscall event is processed. However, + * ipipe_catch_event() will make sure that a + * CPU that clears a handler for any given + * event will not attempt to wait for itself + * to clear the evsync bit for that event, + * which practically plugs the hole, without + * resorting to a much more complex strategy. + */ + np->evsync &= ~(1LL << event); + if (__ipipe_current_domain != next_domain) + this_domain = __ipipe_current_domain; + } + + /* NEVER sync the root stage here. */ + if (next_domain != ipipe_root_domain && + __ipipe_ipending_p(np) && + !test_bit(IPIPE_STALL_FLAG, &np->status)) { + __ipipe_current_domain = next_domain; + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + if (__ipipe_current_domain != next_domain) + this_domain = __ipipe_current_domain; + } + + __ipipe_current_domain = this_domain; + + if (next_domain == this_domain || !propagate) + break; + } + + local_irq_restore_hw(flags); + + return !propagate; +} + +/* + * __ipipe_dispatch_wired -- Wired interrupt dispatcher. Wired + * interrupts are immediately and unconditionally delivered to the + * domain heading the pipeline upon receipt, and such domain must have + * been registered as an invariant head for the system (priority == + * IPIPE_HEAD_PRIORITY). The motivation for using wired interrupts is + * to get an extra-fast dispatching path for those IRQs, by relying on + * a straightforward logic based on assumptions that must always be + * true for invariant head domains. The following assumptions are + * made when dealing with such interrupts: + * + * 1- Wired interrupts are purely dynamic, i.e. the decision to + * propagate them down the pipeline must be done from the head domain + * ISR. + * 2- Wired interrupts cannot be shared or sticky. + * 3- The root domain cannot be an invariant pipeline head, in + * consequence of what the root domain cannot handle wired + * interrupts. + * 4- Wired interrupts must have a valid acknowledge handler for the + * head domain (if needed, see __ipipe_handle_irq). + * + * Called with hw interrupts off. + */ + +void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head); + + prefetchw(p); + + if (unlikely(test_bit(IPIPE_LOCK_FLAG, &head->irqs[irq].control))) { + /* + * If we can't process this IRQ right now, we must + * mark it as held, so that it will get played during + * normal log sync when the corresponding interrupt + * source is eventually unlocked. + */ + __ipipe_set_irq_held(p, irq); + return; + } + + if (test_bit(IPIPE_STALL_FLAG, &p->status)) { + __ipipe_set_irq_pending(head, irq); + return; + } + + __ipipe_dispatch_wired_nocheck(head, irq); +} + +void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq) /* hw interrupts off */ +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head); + struct ipipe_domain *old; + + prefetchw(p); + + old = __ipipe_current_domain; + __ipipe_current_domain = head; /* Switch to the head domain. */ + + p->irqall[irq]++; + __set_bit(IPIPE_STALL_FLAG, &p->status); + head->irqs[irq].handler(irq, head->irqs[irq].cookie); /* Call the ISR. */ + __ipipe_run_irqtail(); + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (__ipipe_current_domain == head) { + __ipipe_current_domain = old; + if (old == head) { + if (__ipipe_ipending_p(p)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + return; + } + } + + __ipipe_walk_pipeline(&head->p_link); +} + +/* + * __ipipe_sync_stage() -- Flush the pending IRQs for the current + * domain (and processor). This routine flushes the interrupt log + * (see "Optimistic interrupt protection" from D. Stodolsky et al. for + * more on the deferred interrupt scheme). Every interrupt that + * occurred while the pipeline was stalled gets played. WARNING: + * callers on SMP boxen should always check for CPU migration on + * return of this routine. + * + * This routine must be called with hw interrupts off. + */ +void __ipipe_sync_stage(int dovirt) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *ipd; + int cpu, irq; + + ipd = __ipipe_current_domain; + p = ipipe_cpudom_ptr(ipd); + + if (__test_and_set_bit(IPIPE_SYNC_FLAG, &p->status)) { + /* + * Some questionable code in the root domain may enter + * busy waits for IRQs over interrupt context, so we + * unfortunately have to allow piling up IRQs for + * them. Non-root domains are not allowed to do this. + */ + if (ipd != ipipe_root_domain) + return; + } + + cpu = ipipe_processor_id(); + + for (;;) { + irq = __ipipe_next_irq(p, dovirt); + if (irq < 0) + break; + /* + * Make sure the compiler does not reorder + * wrongly, so that all updates to maps are + * done before the handler gets called. + */ + barrier(); + + if (test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control)) + continue; + + __set_bit(IPIPE_STALL_FLAG, &p->status); + smp_wmb(); + + if (ipd == ipipe_root_domain) + trace_hardirqs_off(); + + __ipipe_run_isr(ipd, irq); + barrier(); + p = ipipe_cpudom_ptr(__ipipe_current_domain); +#ifdef CONFIG_SMP + { + int newcpu = ipipe_processor_id(); + + if (newcpu != cpu) { /* Handle CPU migration. */ + /* + * We expect any domain to clear the SYNC bit each + * time it switches in a new task, so that preemptions + * and/or CPU migrations (in the SMP case) over the + * ISR do not lock out the log syncer for some + * indefinite amount of time. In the Linux case, + * schedule() handles this (see kernel/sched.c). For + * this reason, we don't bother clearing it here for + * the source CPU in the migration handling case, + * since it must have scheduled another task in by + * now. + */ + __set_bit(IPIPE_SYNC_FLAG, &p->status); + cpu = newcpu; + } + } +#endif /* CONFIG_SMP */ +#ifdef CONFIG_TRACE_IRQFLAGS + if (__ipipe_root_domain_p && + test_bit(IPIPE_STALL_FLAG, &p->status)) + trace_hardirqs_on(); +#endif + __clear_bit(IPIPE_STALL_FLAG, &p->status); + } + + __clear_bit(IPIPE_SYNC_FLAG, &p->status); +} + +/* ipipe_register_domain() -- Link a new domain to the pipeline. */ + +int ipipe_register_domain(struct ipipe_domain *ipd, + struct ipipe_domain_attr *attr) +{ + struct ipipe_percpu_domain_data *p; + struct list_head *pos = NULL; + struct ipipe_domain *_ipd; + unsigned long flags; + + if (!ipipe_root_domain_p) { + printk(KERN_WARNING + "I-pipe: Only the root domain may register a new domain.\n"); + return -EPERM; + } + + flags = ipipe_critical_enter(NULL); + + if (attr->priority == IPIPE_HEAD_PRIORITY) { + if (test_bit(IPIPE_HEAD_SLOT, &__ipipe_domain_slot_map)) { + ipipe_critical_exit(flags); + return -EAGAIN; /* Cannot override current head. */ + } + ipd->slot = IPIPE_HEAD_SLOT; + } else + ipd->slot = ffz(__ipipe_domain_slot_map); + + if (ipd->slot < CONFIG_IPIPE_DOMAINS) { + set_bit(ipd->slot, &__ipipe_domain_slot_map); + list_for_each(pos, &__ipipe_pipeline) { + _ipd = list_entry(pos, struct ipipe_domain, p_link); + if (_ipd->domid == attr->domid) + break; + } + } + + ipipe_critical_exit(flags); + + if (pos != &__ipipe_pipeline) { + if (ipd->slot < CONFIG_IPIPE_DOMAINS) + clear_bit(ipd->slot, &__ipipe_domain_slot_map); + return -EBUSY; + } + +#ifndef CONFIG_SMP + /* + * Set up the perdomain pointers for direct access to the + * percpu domain data. This saves a costly multiply each time + * we need to refer to the contents of the percpu domain data + * array. + */ + __raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = &__raw_get_cpu_var(ipipe_percpu_darray)[ipd->slot]; +#endif + + ipd->name = attr->name; + ipd->domid = attr->domid; + ipd->pdd = attr->pdd; + ipd->flags = 0; + + if (attr->priority == IPIPE_HEAD_PRIORITY) { + ipd->priority = INT_MAX; + __set_bit(IPIPE_AHEAD_FLAG,&ipd->flags); + } + else + ipd->priority = attr->priority; + + __ipipe_init_stage(ipd); + + INIT_LIST_HEAD(&ipd->p_link); + +#ifdef CONFIG_PROC_FS + __ipipe_add_domain_proc(ipd); +#endif /* CONFIG_PROC_FS */ + + flags = ipipe_critical_enter(NULL); + + list_for_each(pos, &__ipipe_pipeline) { + _ipd = list_entry(pos, struct ipipe_domain, p_link); + if (ipd->priority > _ipd->priority) + break; + } + + list_add_tail(&ipd->p_link, pos); + + ipipe_critical_exit(flags); + + printk(KERN_INFO "I-pipe: Domain %s registered.\n", ipd->name); + + if (attr->entry == NULL) + return 0; + + /* + * Finally, allow the new domain to perform its initialization + * duties. + */ + local_irq_save_hw_smp(flags); + __ipipe_current_domain = ipd; + local_irq_restore_hw_smp(flags); + attr->entry(); + local_irq_save_hw(flags); + __ipipe_current_domain = ipipe_root_domain; + p = ipipe_root_cpudom_ptr(); + + if (__ipipe_ipending_p(p) && + !test_bit(IPIPE_STALL_FLAG, &p->status)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + local_irq_restore_hw(flags); + + return 0; +} + +/* ipipe_unregister_domain() -- Remove a domain from the pipeline. */ + +int ipipe_unregister_domain(struct ipipe_domain *ipd) +{ + unsigned long flags; + + if (!ipipe_root_domain_p) { + printk(KERN_WARNING + "I-pipe: Only the root domain may unregister a domain.\n"); + return -EPERM; + } + + if (ipd == ipipe_root_domain) { + printk(KERN_WARNING + "I-pipe: Cannot unregister the root domain.\n"); + return -EPERM; + } +#ifdef CONFIG_SMP + { + struct ipipe_percpu_domain_data *p; + unsigned int irq; + int cpu; + + /* + * In the SMP case, wait for the logged events to drain on + * other processors before eventually removing the domain + * from the pipeline. + */ + + ipipe_unstall_pipeline_from(ipd); + + flags = ipipe_critical_enter(NULL); + + for (irq = 0; irq < IPIPE_NR_IRQS; irq++) { + clear_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control); + clear_bit(IPIPE_STICKY_FLAG, &ipd->irqs[irq].control); + set_bit(IPIPE_PASS_FLAG, &ipd->irqs[irq].control); + } + + ipipe_critical_exit(flags); + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + while (__ipipe_ipending_p(p)) + cpu_relax(); + } + } +#endif /* CONFIG_SMP */ + + mutex_lock(&ipd->mutex); + +#ifdef CONFIG_PROC_FS + __ipipe_remove_domain_proc(ipd); +#endif /* CONFIG_PROC_FS */ + + /* + * Simply remove the domain from the pipeline and we are almost done. + */ + + flags = ipipe_critical_enter(NULL); + list_del_init(&ipd->p_link); + ipipe_critical_exit(flags); + + __ipipe_cleanup_domain(ipd); + + mutex_unlock(&ipd->mutex); + + printk(KERN_INFO "I-pipe: Domain %s unregistered.\n", ipd->name); + + return 0; +} + +/* + * ipipe_propagate_irq() -- Force a given IRQ propagation on behalf of + * a running interrupt handler to the next domain down the pipeline. + * ipipe_schedule_irq() -- Does almost the same as above, but attempts + * to pend the interrupt for the current domain first. + * Must be called hw IRQs off. + */ +void __ipipe_pend_irq(unsigned irq, struct list_head *head) +{ + struct ipipe_domain *ipd; + struct list_head *ln; + +#ifdef CONFIG_IPIPE_DEBUG + BUG_ON(irq >= IPIPE_NR_IRQS || + (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))); +#endif + for (ln = head; ln != &__ipipe_pipeline; ln = ipd->p_link.next) { + ipd = list_entry(ln, struct ipipe_domain, p_link); + if (test_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control)) { + __ipipe_set_irq_pending(ipd, irq); + return; + } + } +} + +/* ipipe_free_virq() -- Release a virtual/soft interrupt. */ + +int ipipe_free_virq(unsigned virq) +{ + if (!ipipe_virtual_irq_p(virq)) + return -EINVAL; + + clear_bit(virq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map); + + return 0; +} + +void ipipe_init_attr(struct ipipe_domain_attr *attr) +{ + attr->name = "anon"; + attr->domid = 1; + attr->entry = NULL; + attr->priority = IPIPE_ROOT_PRIO; + attr->pdd = NULL; +} + +/* + * ipipe_catch_event() -- Interpose or remove an event handler for a + * given domain. + */ +ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd, + unsigned event, + ipipe_event_handler_t handler) +{ + ipipe_event_handler_t old_handler; + unsigned long flags; + int self = 0, cpu; + + if (event & IPIPE_EVENT_SELF) { + event &= ~IPIPE_EVENT_SELF; + self = 1; + } + + if (event >= IPIPE_NR_EVENTS) + return NULL; + + flags = ipipe_critical_enter(NULL); + + if (!(old_handler = xchg(&ipd->evhand[event],handler))) { + if (handler) { + if (self) + ipd->evself |= (1LL << event); + else + __ipipe_event_monitors[event]++; + } + } + else if (!handler) { + if (ipd->evself & (1LL << event)) + ipd->evself &= ~(1LL << event); + else + __ipipe_event_monitors[event]--; + } else if ((ipd->evself & (1LL << event)) && !self) { + __ipipe_event_monitors[event]++; + ipd->evself &= ~(1LL << event); + } else if (!(ipd->evself & (1LL << event)) && self) { + __ipipe_event_monitors[event]--; + ipd->evself |= (1LL << event); + } + + ipipe_critical_exit(flags); + + if (!handler && ipipe_root_domain_p) { + /* + * If we cleared a handler on behalf of the root + * domain, we have to wait for any current invocation + * to drain, since our caller might subsequently unmap + * the target domain. To this aim, this code + * synchronizes with __ipipe_dispatch_event(), + * guaranteeing that either the dispatcher sees a null + * handler in which case it discards the invocation + * (which also prevents from entering a livelock), or + * finds a valid handler and calls it. Symmetrically, + * ipipe_catch_event() ensures that the called code + * won't be unmapped under our feet until the event + * synchronization flag is cleared for the given event + * on all CPUs. + */ + preempt_disable(); + cpu = smp_processor_id(); + /* + * Hack: this solves the potential migration issue + * raised in __ipipe_dispatch_event(). This is a + * work-around which makes the assumption that other + * CPUs will subsequently, either process at least one + * interrupt for the target domain, or call + * __ipipe_dispatch_event() without going through a + * migration while running the handler at least once; + * practically, this is safe on any normally running + * system. + */ + ipipe_percpudom(ipd, evsync, cpu) &= ~(1LL << event); + preempt_enable(); + + for_each_online_cpu(cpu) { + while (ipipe_percpudom(ipd, evsync, cpu) & (1LL << event)) + schedule_timeout_interruptible(HZ / 50); + } + } + + return old_handler; +} + +cpumask_t ipipe_set_irq_affinity (unsigned irq, cpumask_t cpumask) +{ +#ifdef CONFIG_SMP + if (irq >= IPIPE_NR_XIRQS) + /* Allow changing affinity of external IRQs only. */ + return CPU_MASK_NONE; + + if (num_online_cpus() > 1) + return __ipipe_set_irq_affinity(irq,cpumask); +#endif /* CONFIG_SMP */ + + return CPU_MASK_NONE; +} + +int ipipe_send_ipi (unsigned ipi, cpumask_t cpumask) + +{ +#ifdef CONFIG_SMP + return __ipipe_send_ipi(ipi,cpumask); +#else /* !CONFIG_SMP */ + return -EINVAL; +#endif /* CONFIG_SMP */ +} + +int ipipe_alloc_ptdkey (void) +{ + unsigned long flags; + int key = -1; + + spin_lock_irqsave(&__ipipe_pipelock,flags); + + if (__ipipe_ptd_key_count < IPIPE_ROOT_NPTDKEYS) { + key = ffz(__ipipe_ptd_key_map); + set_bit(key,&__ipipe_ptd_key_map); + __ipipe_ptd_key_count++; + } + + spin_unlock_irqrestore(&__ipipe_pipelock,flags); + + return key; +} + +int ipipe_free_ptdkey (int key) +{ + unsigned long flags; + + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return -EINVAL; + + spin_lock_irqsave(&__ipipe_pipelock,flags); + + if (test_and_clear_bit(key,&__ipipe_ptd_key_map)) + __ipipe_ptd_key_count--; + + spin_unlock_irqrestore(&__ipipe_pipelock,flags); + + return 0; +} + +int ipipe_set_ptd (int key, void *value) + +{ + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return -EINVAL; + + current->ptd[key] = value; + + return 0; +} + +void *ipipe_get_ptd (int key) + +{ + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return NULL; + + return current->ptd[key]; +} + +#ifdef CONFIG_PROC_FS + +struct proc_dir_entry *ipipe_proc_root; + +static int __ipipe_version_info_proc(char *page, + char **start, + off_t off, int count, int *eof, void *data) +{ + int len = sprintf(page, "%s\n", IPIPE_VERSION_STRING); + + len -= off; + + if (len <= off + count) + *eof = 1; + + *start = page + off; + + if(len > count) + len = count; + + if(len < 0) + len = 0; + + return len; +} + +static int __ipipe_common_info_show(struct seq_file *p, void *data) +{ + struct ipipe_domain *ipd = (struct ipipe_domain *)p->private; + char handling, stickiness, lockbit, exclusive, virtuality; + + unsigned long ctlbits; + unsigned irq; + + seq_printf(p, " +----- Handling ([A]ccepted, [G]rabbed, [W]ired, [D]iscarded)\n"); + seq_printf(p, " |+---- Sticky\n"); + seq_printf(p, " ||+--- Locked\n"); + seq_printf(p, " |||+-- Exclusive\n"); + seq_printf(p, " ||||+- Virtual\n"); + seq_printf(p, "[IRQ] |||||\n"); + + mutex_lock(&ipd->mutex); + + for (irq = 0; irq < IPIPE_NR_IRQS; irq++) { + /* Remember to protect against + * ipipe_virtual_irq/ipipe_control_irq if more fields + * get involved. */ + ctlbits = ipd->irqs[irq].control; + + if (irq >= IPIPE_NR_XIRQS && !ipipe_virtual_irq_p(irq)) + /* + * There might be a hole between the last external + * IRQ and the first virtual one; skip it. + */ + continue; + + if (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map)) + /* Non-allocated virtual IRQ; skip it. */ + continue; + + /* + * Statuses are as follows: + * o "accepted" means handled _and_ passed down the pipeline. + * o "grabbed" means handled, but the interrupt might be + * terminated _or_ passed down the pipeline depending on + * what the domain handler asks for to the I-pipe. + * o "wired" is basically the same as "grabbed", except that + * the interrupt is unconditionally delivered to an invariant + * pipeline head domain. + * o "passed" means unhandled by the domain but passed + * down the pipeline. + * o "discarded" means unhandled and _not_ passed down the + * pipeline. The interrupt merely disappears from the + * current domain down to the end of the pipeline. + */ + if (ctlbits & IPIPE_HANDLE_MASK) { + if (ctlbits & IPIPE_PASS_MASK) + handling = 'A'; + else if (ctlbits & IPIPE_WIRED_MASK) + handling = 'W'; + else + handling = 'G'; + } else if (ctlbits & IPIPE_PASS_MASK) + /* Do not output if no major action is taken. */ + continue; + else + handling = 'D'; + + if (ctlbits & IPIPE_STICKY_MASK) + stickiness = 'S'; + else + stickiness = '.'; + + if (ctlbits & IPIPE_LOCK_MASK) + lockbit = 'L'; + else + lockbit = '.'; + + if (ctlbits & IPIPE_EXCLUSIVE_MASK) + exclusive = 'X'; + else + exclusive = '.'; + + if (ipipe_virtual_irq_p(irq)) + virtuality = 'V'; + else + virtuality = '.'; + + seq_printf(p, " %3u: %c%c%c%c%c\n", + irq, handling, stickiness, lockbit, exclusive, virtuality); + } + + seq_printf(p, "[Domain info]\n"); + + seq_printf(p, "id=0x%.8x\n", ipd->domid); + + if (test_bit(IPIPE_AHEAD_FLAG,&ipd->flags)) + seq_printf(p, "priority=topmost\n"); + else + seq_printf(p, "priority=%d\n", ipd->priority); + + mutex_unlock(&ipd->mutex); + + return 0; +} + +static int __ipipe_common_info_open(struct inode *inode, struct file *file) +{ + return single_open(file, __ipipe_common_info_show, PROC_I(inode)->pde->data); +} + +static struct file_operations __ipipe_info_proc_ops = { + .owner = THIS_MODULE, + .open = __ipipe_common_info_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +void __ipipe_add_domain_proc(struct ipipe_domain *ipd) +{ + struct proc_dir_entry *e = create_proc_entry(ipd->name, 0444, ipipe_proc_root); + if (e) { + e->proc_fops = &__ipipe_info_proc_ops; + e->data = (void*) ipd; + } +} + +void __ipipe_remove_domain_proc(struct ipipe_domain *ipd) +{ + remove_proc_entry(ipd->name,ipipe_proc_root); +} + +void __init ipipe_init_proc(void) +{ + ipipe_proc_root = create_proc_entry("ipipe",S_IFDIR, 0); + create_proc_read_entry("version",0444,ipipe_proc_root,&__ipipe_version_info_proc,NULL); + __ipipe_add_domain_proc(ipipe_root_domain); + + __ipipe_init_tracer(); +} + +#endif /* CONFIG_PROC_FS */ + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + +DEFINE_PER_CPU(int, ipipe_percpu_context_check) = { 1 }; +DEFINE_PER_CPU(int, ipipe_saved_context_check_state); + +void ipipe_check_context(struct ipipe_domain *border_domain) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *this_domain; + unsigned long flags; + int cpu; + + local_irq_save_hw_smp(flags); + + this_domain = __ipipe_current_domain; + p = ipipe_head_cpudom_ptr(); + if (likely(this_domain->priority <= border_domain->priority && + !test_bit(IPIPE_STALL_FLAG, &p->status))) { + local_irq_restore_hw_smp(flags); + return; + } + + cpu = ipipe_processor_id(); + if (!per_cpu(ipipe_percpu_context_check, cpu)) { + local_irq_restore_hw_smp(flags); + return; + } + + local_irq_restore_hw_smp(flags); + + ipipe_context_check_off(); + ipipe_trace_panic_freeze(); + ipipe_set_printk_sync(__ipipe_current_domain); + + if (this_domain->priority > border_domain->priority) + printk(KERN_ERR "I-pipe: Detected illicit call from domain " + "'%s'\n" + KERN_ERR " into a service reserved for domain " + "'%s' and below.\n", + this_domain->name, border_domain->name); + else + printk(KERN_ERR "I-pipe: Detected stalled topmost domain, " + "probably caused by a bug.\n" + " A critical section may have been " + "left unterminated.\n"); + dump_stack(); + ipipe_trace_panic_dump(); +} + +EXPORT_SYMBOL(ipipe_check_context); + +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ + +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) + +int notrace __ipipe_check_percpu_access(void) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *this_domain; + unsigned long flags; + int ret = 0; + + local_irq_save_hw_notrace(flags); + + this_domain = __raw_get_cpu_var(ipipe_percpu_domain); + + /* + * Only the root domain may implement preemptive CPU migration + * of tasks, so anything above in the pipeline should be fine. + */ + if (this_domain->priority > IPIPE_ROOT_PRIO) + goto out; + + if (raw_irqs_disabled_flags(flags)) + goto out; + + /* + * Last chance: hw interrupts were enabled on entry while + * running over the root domain, but the root stage might be + * currently stalled, in which case preemption would be + * disabled, and no migration could occur. + */ + if (this_domain == ipipe_root_domain) { + p = ipipe_root_cpudom_ptr(); + if (test_bit(IPIPE_STALL_FLAG, &p->status)) + goto out; + } + /* + * Our caller may end up accessing the wrong per-cpu variable + * instance due to CPU migration; tell it to complain about + * this. + */ + ret = 1; +out: + local_irq_restore_hw_notrace(flags); + + return ret; +} + +#endif /* CONFIG_IPIPE_DEBUG_INTERNAL && CONFIG_SMP */ + +EXPORT_SYMBOL(ipipe_virtualize_irq); +EXPORT_SYMBOL(ipipe_control_irq); +EXPORT_SYMBOL(ipipe_suspend_domain); +EXPORT_SYMBOL(ipipe_alloc_virq); +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_domain); +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_darray); +EXPORT_SYMBOL(ipipe_root); +EXPORT_SYMBOL(ipipe_stall_pipeline_from); +EXPORT_SYMBOL(ipipe_test_and_stall_pipeline_from); +EXPORT_SYMBOL(ipipe_test_and_unstall_pipeline_from); +EXPORT_SYMBOL(ipipe_restore_pipeline_from); +EXPORT_SYMBOL(ipipe_unstall_pipeline_head); +EXPORT_SYMBOL(__ipipe_restore_pipeline_head); +EXPORT_SYMBOL(__ipipe_unstall_root); +EXPORT_SYMBOL(__ipipe_restore_root); +EXPORT_SYMBOL(__ipipe_spin_lock_irq); +EXPORT_SYMBOL(__ipipe_spin_unlock_irq); +EXPORT_SYMBOL(__ipipe_spin_lock_irqsave); +EXPORT_SYMBOL(__ipipe_spin_trylock_irqsave); +EXPORT_SYMBOL(__ipipe_spin_unlock_irqrestore); +EXPORT_SYMBOL(__ipipe_pipeline); +EXPORT_SYMBOL(__ipipe_lock_irq); +EXPORT_SYMBOL(__ipipe_unlock_irq); +EXPORT_SYMBOL(ipipe_register_domain); +EXPORT_SYMBOL(ipipe_unregister_domain); +EXPORT_SYMBOL(ipipe_free_virq); +EXPORT_SYMBOL(ipipe_init_attr); +EXPORT_SYMBOL(ipipe_catch_event); +EXPORT_SYMBOL(ipipe_alloc_ptdkey); +EXPORT_SYMBOL(ipipe_free_ptdkey); +EXPORT_SYMBOL(ipipe_set_ptd); +EXPORT_SYMBOL(ipipe_get_ptd); +EXPORT_SYMBOL(ipipe_set_irq_affinity); +EXPORT_SYMBOL(ipipe_send_ipi); +EXPORT_SYMBOL(__ipipe_pend_irq); +EXPORT_SYMBOL(__ipipe_set_irq_pending); +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) +EXPORT_SYMBOL(__ipipe_check_percpu_access); +#endif +#ifdef CONFIG_GENERIC_CLOCKEVENTS +EXPORT_SYMBOL(ipipe_request_tickdev); +EXPORT_SYMBOL(ipipe_release_tickdev); +#endif + +EXPORT_SYMBOL(ipipe_critical_enter); +EXPORT_SYMBOL(ipipe_critical_exit); +EXPORT_SYMBOL(ipipe_trigger_irq); +EXPORT_SYMBOL(ipipe_get_sysinfo); diff --git a/kernel/ipipe/tracer.c b/kernel/ipipe/tracer.c new file mode 100644 index 0000000..001a83e --- /dev/null +++ b/kernel/ipipe/tracer.c @@ -0,0 +1,1441 @@ +/* -*- linux-c -*- + * kernel/ipipe/tracer.c + * + * Copyright (C) 2005 Luotao Fu. + * 2005-2008 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/version.h> +#include <linux/kallsyms.h> +#include <linux/seq_file.h> +#include <linux/proc_fs.h> +#include <linux/ctype.h> +#include <linux/vmalloc.h> +#include <linux/pid.h> +#include <linux/vermagic.h> +#include <linux/sched.h> +#include <linux/ipipe.h> +#include <linux/ftrace.h> +#include <asm/uaccess.h> + +#define IPIPE_TRACE_PATHS 4 /* <!> Do not lower below 3 */ +#define IPIPE_DEFAULT_ACTIVE 0 +#define IPIPE_DEFAULT_MAX 1 +#define IPIPE_DEFAULT_FROZEN 2 + +#define IPIPE_TRACE_POINTS (1 << CONFIG_IPIPE_TRACE_SHIFT) +#define WRAP_POINT_NO(point) ((point) & (IPIPE_TRACE_POINTS-1)) + +#define IPIPE_DEFAULT_PRE_TRACE 10 +#define IPIPE_DEFAULT_POST_TRACE 10 +#define IPIPE_DEFAULT_BACK_TRACE 100 + +#define IPIPE_DELAY_NOTE 1000 /* in nanoseconds */ +#define IPIPE_DELAY_WARN 10000 /* in nanoseconds */ + +#define IPIPE_TFLG_NMI_LOCK 0x0001 +#define IPIPE_TFLG_NMI_HIT 0x0002 +#define IPIPE_TFLG_NMI_FREEZE_REQ 0x0004 + +#define IPIPE_TFLG_HWIRQ_OFF 0x0100 +#define IPIPE_TFLG_FREEZING 0x0200 +#define IPIPE_TFLG_CURRDOM_SHIFT 10 /* bits 10..11: current domain */ +#define IPIPE_TFLG_CURRDOM_MASK 0x0C00 +#define IPIPE_TFLG_DOMSTATE_SHIFT 12 /* bits 12..15: domain stalled? */ +#define IPIPE_TFLG_DOMSTATE_BITS 3 + +#define IPIPE_TFLG_DOMAIN_STALLED(point, n) \ + (point->flags & (1 << (n + IPIPE_TFLG_DOMSTATE_SHIFT))) +#define IPIPE_TFLG_CURRENT_DOMAIN(point) \ + ((point->flags & IPIPE_TFLG_CURRDOM_MASK) >> IPIPE_TFLG_CURRDOM_SHIFT) + +struct ipipe_trace_point { + short type; + short flags; + unsigned long eip; + unsigned long parent_eip; + unsigned long v; + unsigned long long timestamp; +}; + +struct ipipe_trace_path { + volatile int flags; + int dump_lock; /* separated from flags due to cross-cpu access */ + int trace_pos; /* next point to fill */ + int begin, end; /* finalised path begin and end */ + int post_trace; /* non-zero when in post-trace phase */ + unsigned long long length; /* max path length in cycles */ + unsigned long nmi_saved_eip; /* for deferred requests from NMIs */ + unsigned long nmi_saved_parent_eip; + unsigned long nmi_saved_v; + struct ipipe_trace_point point[IPIPE_TRACE_POINTS]; +} ____cacheline_aligned_in_smp; + +enum ipipe_trace_type +{ + IPIPE_TRACE_FUNC = 0, + IPIPE_TRACE_BEGIN, + IPIPE_TRACE_END, + IPIPE_TRACE_FREEZE, + IPIPE_TRACE_SPECIAL, + IPIPE_TRACE_PID, + IPIPE_TRACE_EVENT, +}; + +#define IPIPE_TYPE_MASK 0x0007 +#define IPIPE_TYPE_BITS 3 + +#ifdef CONFIG_IPIPE_TRACE_VMALLOC +static DEFINE_PER_CPU(struct ipipe_trace_path *, trace_path); +#else /* !CONFIG_IPIPE_TRACE_VMALLOC */ +static DEFINE_PER_CPU(struct ipipe_trace_path, trace_path[IPIPE_TRACE_PATHS]) = + { [0 ... IPIPE_TRACE_PATHS-1] = { .begin = -1, .end = -1 } }; +#endif /* CONFIG_IPIPE_TRACE_VMALLOC */ + +int ipipe_trace_enable = 0; + +static DEFINE_PER_CPU(int, active_path) = { IPIPE_DEFAULT_ACTIVE }; +static DEFINE_PER_CPU(int, max_path) = { IPIPE_DEFAULT_MAX }; +static DEFINE_PER_CPU(int, frozen_path) = { IPIPE_DEFAULT_FROZEN }; +static IPIPE_DEFINE_SPINLOCK(global_path_lock); +static int pre_trace = IPIPE_DEFAULT_PRE_TRACE; +static int post_trace = IPIPE_DEFAULT_POST_TRACE; +static int back_trace = IPIPE_DEFAULT_BACK_TRACE; +static int verbose_trace = 1; +static unsigned long trace_overhead; + +static unsigned long trigger_begin; +static unsigned long trigger_end; + +static DEFINE_MUTEX(out_mutex); +static struct ipipe_trace_path *print_path; +#ifdef CONFIG_IPIPE_TRACE_PANIC +static struct ipipe_trace_path *panic_path; +#endif /* CONFIG_IPIPE_TRACE_PANIC */ +static int print_pre_trace; +static int print_post_trace; + + +static long __ipipe_signed_tsc2us(long long tsc); +static void +__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point); +static void __ipipe_print_symname(struct seq_file *m, unsigned long eip); + + +static notrace void +__ipipe_store_domain_states(struct ipipe_trace_point *point) +{ + struct ipipe_domain *ipd; + struct list_head *pos; + int i = 0; + + list_for_each_prev(pos, &__ipipe_pipeline) { + ipd = list_entry(pos, struct ipipe_domain, p_link); + + if (test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status))) + point->flags |= 1 << (i + IPIPE_TFLG_DOMSTATE_SHIFT); + + if (ipd == __ipipe_current_domain) + point->flags |= i << IPIPE_TFLG_CURRDOM_SHIFT; + + if (++i > IPIPE_TFLG_DOMSTATE_BITS) + break; + } +} + +static notrace int __ipipe_get_free_trace_path(int old, int cpu) +{ + int new_active = old; + struct ipipe_trace_path *tp; + + do { + if (++new_active == IPIPE_TRACE_PATHS) + new_active = 0; + tp = &per_cpu(trace_path, cpu)[new_active]; + } while (new_active == per_cpu(max_path, cpu) || + new_active == per_cpu(frozen_path, cpu) || + tp->dump_lock); + + return new_active; +} + +static notrace void +__ipipe_migrate_pre_trace(struct ipipe_trace_path *new_tp, + struct ipipe_trace_path *old_tp, int old_pos) +{ + int i; + + new_tp->trace_pos = pre_trace+1; + + for (i = new_tp->trace_pos; i > 0; i--) + memcpy(&new_tp->point[WRAP_POINT_NO(new_tp->trace_pos-i)], + &old_tp->point[WRAP_POINT_NO(old_pos-i)], + sizeof(struct ipipe_trace_point)); + + /* mark the end (i.e. the point before point[0]) invalid */ + new_tp->point[IPIPE_TRACE_POINTS-1].eip = 0; +} + +static notrace struct ipipe_trace_path * +__ipipe_trace_end(int cpu, struct ipipe_trace_path *tp, int pos) +{ + struct ipipe_trace_path *old_tp = tp; + long active = per_cpu(active_path, cpu); + unsigned long long length; + + /* do we have a new worst case? */ + length = tp->point[tp->end].timestamp - + tp->point[tp->begin].timestamp; + if (length > per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)].length) { + /* we need protection here against other cpus trying + to start a proc dump */ + spin_lock(&global_path_lock); + + /* active path holds new worst case */ + tp->length = length; + per_cpu(max_path, cpu) = active; + + /* find next unused trace path */ + active = __ipipe_get_free_trace_path(active, cpu); + + spin_unlock(&global_path_lock); + + tp = &per_cpu(trace_path, cpu)[active]; + + /* migrate last entries for pre-tracing */ + __ipipe_migrate_pre_trace(tp, old_tp, pos); + } + + return tp; +} + +static notrace struct ipipe_trace_path * +__ipipe_trace_freeze(int cpu, struct ipipe_trace_path *tp, int pos) +{ + struct ipipe_trace_path *old_tp = tp; + long active = per_cpu(active_path, cpu); + int n; + + /* frozen paths have no core (begin=end) */ + tp->begin = tp->end; + + /* we need protection here against other cpus trying + * to set their frozen path or to start a proc dump */ + spin_lock(&global_path_lock); + + per_cpu(frozen_path, cpu) = active; + + /* find next unused trace path */ + active = __ipipe_get_free_trace_path(active, cpu); + + /* check if this is the first frozen path */ + for_each_possible_cpu(n) { + if (n != cpu && + per_cpu(trace_path, n)[per_cpu(frozen_path, n)].end >= 0) + tp->end = -1; + } + + spin_unlock(&global_path_lock); + + tp = &per_cpu(trace_path, cpu)[active]; + + /* migrate last entries for pre-tracing */ + __ipipe_migrate_pre_trace(tp, old_tp, pos); + + return tp; +} + +void notrace +__ipipe_trace(enum ipipe_trace_type type, unsigned long eip, + unsigned long parent_eip, unsigned long v) +{ + struct ipipe_trace_path *tp, *old_tp; + int pos, next_pos, begin; + struct ipipe_trace_point *point; + unsigned long flags; + int cpu; + + local_irq_save_hw_notrace(flags); + + cpu = ipipe_processor_id(); + restart: + tp = old_tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + /* here starts a race window with NMIs - catched below */ + + /* check for NMI recursion */ + if (unlikely(tp->flags & IPIPE_TFLG_NMI_LOCK)) { + tp->flags |= IPIPE_TFLG_NMI_HIT; + + /* first freeze request from NMI context? */ + if ((type == IPIPE_TRACE_FREEZE) && + !(tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ)) { + /* save arguments and mark deferred freezing */ + tp->flags |= IPIPE_TFLG_NMI_FREEZE_REQ; + tp->nmi_saved_eip = eip; + tp->nmi_saved_parent_eip = parent_eip; + tp->nmi_saved_v = v; + } + return; /* no need for restoring flags inside IRQ */ + } + + /* clear NMI events and set lock (atomically per cpu) */ + tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT | + IPIPE_TFLG_NMI_FREEZE_REQ)) + | IPIPE_TFLG_NMI_LOCK; + + /* check active_path again - some nasty NMI may have switched + * it meanwhile */ + if (unlikely(tp != + &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)])) { + /* release lock on wrong path and restart */ + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* there is no chance that the NMI got deferred + * => no need to check for pending freeze requests */ + goto restart; + } + + /* get the point buffer */ + pos = tp->trace_pos; + point = &tp->point[pos]; + + /* store all trace point data */ + point->type = type; + point->flags = raw_irqs_disabled_flags(flags) ? IPIPE_TFLG_HWIRQ_OFF : 0; + point->eip = eip; + point->parent_eip = parent_eip; + point->v = v; + ipipe_read_tsc(point->timestamp); + + __ipipe_store_domain_states(point); + + /* forward to next point buffer */ + next_pos = WRAP_POINT_NO(pos+1); + tp->trace_pos = next_pos; + + /* only mark beginning if we haven't started yet */ + begin = tp->begin; + if (unlikely(type == IPIPE_TRACE_BEGIN) && (begin < 0)) + tp->begin = pos; + + /* end of critical path, start post-trace if not already started */ + if (unlikely(type == IPIPE_TRACE_END) && + (begin >= 0) && !tp->post_trace) + tp->post_trace = post_trace + 1; + + /* freeze only if the slot is free and we are not already freezing */ + if ((unlikely(type == IPIPE_TRACE_FREEZE) || + (unlikely(eip >= trigger_begin && eip <= trigger_end) && + type == IPIPE_TRACE_FUNC)) && + per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)].begin < 0 && + !(tp->flags & IPIPE_TFLG_FREEZING)) { + tp->post_trace = post_trace + 1; + tp->flags |= IPIPE_TFLG_FREEZING; + } + + /* enforce end of trace in case of overflow */ + if (unlikely(WRAP_POINT_NO(next_pos + 1) == begin)) { + tp->end = pos; + goto enforce_end; + } + + /* stop tracing this path if we are in post-trace and + * a) that phase is over now or + * b) a new TRACE_BEGIN came in but we are not freezing this path */ + if (unlikely((tp->post_trace > 0) && ((--tp->post_trace == 0) || + ((type == IPIPE_TRACE_BEGIN) && + !(tp->flags & IPIPE_TFLG_FREEZING))))) { + /* store the path's end (i.e. excluding post-trace) */ + tp->end = WRAP_POINT_NO(pos - post_trace + tp->post_trace); + + enforce_end: + if (tp->flags & IPIPE_TFLG_FREEZING) + tp = __ipipe_trace_freeze(cpu, tp, pos); + else + tp = __ipipe_trace_end(cpu, tp, pos); + + /* reset the active path, maybe already start a new one */ + tp->begin = (type == IPIPE_TRACE_BEGIN) ? + WRAP_POINT_NO(tp->trace_pos - 1) : -1; + tp->end = -1; + tp->post_trace = 0; + tp->flags = 0; + + /* update active_path not earlier to avoid races with NMIs */ + per_cpu(active_path, cpu) = tp - per_cpu(trace_path, cpu); + } + + /* we still have old_tp and point, + * let's reset NMI lock and check for catches */ + old_tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + if (unlikely(old_tp->flags & IPIPE_TFLG_NMI_HIT)) { + /* well, this late tagging may not immediately be visible for + * other cpus already dumping this path - a minor issue */ + point->flags |= IPIPE_TFLG_NMI_HIT; + + /* handle deferred freezing from NMI context */ + if (old_tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ) + __ipipe_trace(IPIPE_TRACE_FREEZE, old_tp->nmi_saved_eip, + old_tp->nmi_saved_parent_eip, + old_tp->nmi_saved_v); + } + + local_irq_restore_hw_notrace(flags); +} + +static unsigned long __ipipe_global_path_lock(void) +{ + unsigned long flags; + int cpu; + struct ipipe_trace_path *tp; + + spin_lock_irqsave(&global_path_lock, flags); + + cpu = ipipe_processor_id(); + restart: + tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + /* here is small race window with NMIs - catched below */ + + /* clear NMI events and set lock (atomically per cpu) */ + tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT | + IPIPE_TFLG_NMI_FREEZE_REQ)) + | IPIPE_TFLG_NMI_LOCK; + + /* check active_path again - some nasty NMI may have switched + * it meanwhile */ + if (tp != &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]) { + /* release lock on wrong path and restart */ + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* there is no chance that the NMI got deferred + * => no need to check for pending freeze requests */ + goto restart; + } + + return flags; +} + +static void __ipipe_global_path_unlock(unsigned long flags) +{ + int cpu; + struct ipipe_trace_path *tp; + + /* release spinlock first - it's not involved in the NMI issue */ + __ipipe_spin_unlock_irqbegin(&global_path_lock); + + cpu = ipipe_processor_id(); + tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* handle deferred freezing from NMI context */ + if (tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ) + __ipipe_trace(IPIPE_TRACE_FREEZE, tp->nmi_saved_eip, + tp->nmi_saved_parent_eip, tp->nmi_saved_v); + + /* See __ipipe_spin_lock_irqsave() and friends. */ + __ipipe_spin_unlock_irqcomplete(flags); +} + +void notrace ipipe_trace_begin(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_BEGIN, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_begin); + +void notrace ipipe_trace_end(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_END, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_end); + +void notrace ipipe_trace_freeze(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_FREEZE, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_freeze); + +void notrace ipipe_trace_special(unsigned char id, unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_SPECIAL | (id << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_special); + +void notrace ipipe_trace_pid(pid_t pid, short prio) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_PID | (prio << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, pid); +} +EXPORT_SYMBOL(ipipe_trace_pid); + +void notrace ipipe_trace_event(unsigned char id, unsigned long delay_tsc) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_EVENT | (id << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, delay_tsc); +} +EXPORT_SYMBOL(ipipe_trace_event); + +int ipipe_trace_max_reset(void) +{ + int cpu; + unsigned long flags; + struct ipipe_trace_path *path; + int ret = 0; + + flags = __ipipe_global_path_lock(); + + for_each_possible_cpu(cpu) { + path = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)]; + + if (path->dump_lock) { + ret = -EBUSY; + break; + } + + path->begin = -1; + path->end = -1; + path->trace_pos = 0; + path->length = 0; + } + + __ipipe_global_path_unlock(flags); + + return ret; +} +EXPORT_SYMBOL(ipipe_trace_max_reset); + +int ipipe_trace_frozen_reset(void) +{ + int cpu; + unsigned long flags; + struct ipipe_trace_path *path; + int ret = 0; + + flags = __ipipe_global_path_lock(); + + for_each_online_cpu(cpu) { + path = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)]; + + if (path->dump_lock) { + ret = -EBUSY; + break; + } + + path->begin = -1; + path->end = -1; + path->trace_pos = 0; + path->length = 0; + } + + __ipipe_global_path_unlock(flags); + + return ret; +} +EXPORT_SYMBOL(ipipe_trace_frozen_reset); + +static void +__ipipe_get_task_info(char *task_info, struct ipipe_trace_point *point, + int trylock) +{ + struct task_struct *task = NULL; + char buf[8]; + int i; + int locked = 1; + + if (trylock) { + if (!read_trylock(&tasklist_lock)) + locked = 0; + } else + read_lock(&tasklist_lock); + + if (locked) + task = find_task_by_pid_ns((pid_t)point->v, &init_pid_ns); + + if (task) + strncpy(task_info, task->comm, 11); + else + strcpy(task_info, "-<?>-"); + + if (locked) + read_unlock(&tasklist_lock); + + for (i = strlen(task_info); i < 11; i++) + task_info[i] = ' '; + + sprintf(buf, " %d ", point->type >> IPIPE_TYPE_BITS); + strcpy(task_info + (11 - strlen(buf)), buf); +} + +static void +__ipipe_get_event_date(char *buf,struct ipipe_trace_path *path, + struct ipipe_trace_point *point) +{ + long time; + int type; + + time = __ipipe_signed_tsc2us(point->timestamp - + path->point[path->begin].timestamp + point->v); + type = point->type >> IPIPE_TYPE_BITS; + + if (type == 0) + /* + * Event type #0 is predefined, stands for the next + * timer tick. + */ + sprintf(buf, "tick@%-6ld", time); + else + sprintf(buf, "%3d@%-7ld", type, time); +} + +#ifdef CONFIG_IPIPE_TRACE_PANIC +void ipipe_trace_panic_freeze(void) +{ + unsigned long flags; + int cpu; + + if (!ipipe_trace_enable) + return; + + ipipe_trace_enable = 0; + local_irq_save_hw_notrace(flags); + + cpu = ipipe_processor_id(); + + panic_path = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + local_irq_restore_hw(flags); +} +EXPORT_SYMBOL(ipipe_trace_panic_freeze); + +void ipipe_trace_panic_dump(void) +{ + int cnt = back_trace; + int start, pos; + char buf[16]; + + if (!panic_path) + return; + + ipipe_context_check_off(); + + printk("I-pipe tracer log (%d points):\n", cnt); + + start = pos = WRAP_POINT_NO(panic_path->trace_pos-1); + + while (cnt-- > 0) { + struct ipipe_trace_point *point = &panic_path->point[pos]; + long time; + char info[16]; + int i; + + printk(" %c", + (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' '); + + for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--) + printk("%c", + (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ? + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '#' : '+') : + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '*' : ' ')); + + if (!point->eip) + printk("-<invalid>-\n"); + else { + __ipipe_trace_point_type(buf, point); + printk("%s", buf); + + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + printk(" "); + break; + + case IPIPE_TRACE_PID: + __ipipe_get_task_info(info, + point, 1); + printk("%s", info); + break; + + case IPIPE_TRACE_EVENT: + __ipipe_get_event_date(info, + panic_path, point); + printk("%s", info); + break; + + default: + printk("0x%08lx ", point->v); + } + + time = __ipipe_signed_tsc2us(point->timestamp - + panic_path->point[start].timestamp); + printk(" %5ld ", time); + + __ipipe_print_symname(NULL, point->eip); + printk(" ("); + __ipipe_print_symname(NULL, point->parent_eip); + printk(")\n"); + } + pos = WRAP_POINT_NO(pos - 1); + } + + panic_path = NULL; +} +EXPORT_SYMBOL(ipipe_trace_panic_dump); +#endif /* CONFIG_IPIPE_TRACE_PANIC */ + + +/* --- /proc output --- */ + +static notrace int __ipipe_in_critical_trpath(long point_no) +{ + return ((WRAP_POINT_NO(point_no-print_path->begin) < + WRAP_POINT_NO(print_path->end-print_path->begin)) || + ((print_path->end == print_path->begin) && + (WRAP_POINT_NO(point_no-print_path->end) > + print_post_trace))); +} + +static long __ipipe_signed_tsc2us(long long tsc) +{ + unsigned long long abs_tsc; + long us; + + /* ipipe_tsc2us works on unsigned => handle sign separately */ + abs_tsc = (tsc >= 0) ? tsc : -tsc; + us = ipipe_tsc2us(abs_tsc); + if (tsc < 0) + return -us; + else + return us; +} + +static void +__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point) +{ + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + strcpy(buf, "func "); + break; + + case IPIPE_TRACE_BEGIN: + strcpy(buf, "begin "); + break; + + case IPIPE_TRACE_END: + strcpy(buf, "end "); + break; + + case IPIPE_TRACE_FREEZE: + strcpy(buf, "freeze "); + break; + + case IPIPE_TRACE_SPECIAL: + sprintf(buf, "(0x%02x) ", + point->type >> IPIPE_TYPE_BITS); + break; + + case IPIPE_TRACE_PID: + sprintf(buf, "[%5d] ", (pid_t)point->v); + break; + + case IPIPE_TRACE_EVENT: + sprintf(buf, "event "); + break; + } +} + +static void +__ipipe_print_pathmark(struct seq_file *m, struct ipipe_trace_point *point) +{ + char mark = ' '; + int point_no = point - print_path->point; + int i; + + if (print_path->end == point_no) + mark = '<'; + else if (print_path->begin == point_no) + mark = '>'; + else if (__ipipe_in_critical_trpath(point_no)) + mark = ':'; + seq_printf(m, "%c%c", mark, + (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' '); + + if (!verbose_trace) + return; + + for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--) + seq_printf(m, "%c", + (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ? + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '#' : '+') : + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? '*' : ' ')); +} + +static void +__ipipe_print_delay(struct seq_file *m, struct ipipe_trace_point *point) +{ + unsigned long delay = 0; + int next; + char *mark = " "; + + next = WRAP_POINT_NO(point+1 - print_path->point); + + if (next != print_path->trace_pos) + delay = ipipe_tsc2ns(print_path->point[next].timestamp - + point->timestamp); + + if (__ipipe_in_critical_trpath(point - print_path->point)) { + if (delay > IPIPE_DELAY_WARN) + mark = "! "; + else if (delay > IPIPE_DELAY_NOTE) + mark = "+ "; + } + seq_puts(m, mark); + + if (verbose_trace) + seq_printf(m, "%3lu.%03lu%c ", delay/1000, delay%1000, + (point->flags & IPIPE_TFLG_NMI_HIT) ? 'N' : ' '); + else + seq_puts(m, " "); +} + +static void __ipipe_print_symname(struct seq_file *m, unsigned long eip) +{ + char namebuf[KSYM_NAME_LEN+1]; + unsigned long size, offset; + const char *sym_name; + char *modname; + + sym_name = kallsyms_lookup(eip, &size, &offset, &modname, namebuf); + +#ifdef CONFIG_IPIPE_TRACE_PANIC + if (!m) { + /* panic dump */ + if (sym_name) { + printk("%s+0x%lx", sym_name, offset); + if (modname) + printk(" [%s]", modname); + } + } else +#endif /* CONFIG_IPIPE_TRACE_PANIC */ + { + if (sym_name) { + if (verbose_trace) { + seq_printf(m, "%s+0x%lx", sym_name, offset); + if (modname) + seq_printf(m, " [%s]", modname); + } else + seq_puts(m, sym_name); + } else + seq_printf(m, "<%08lx>", eip); + } +} + +static void __ipipe_print_headline(struct seq_file *m) +{ + seq_printf(m, "Calibrated minimum trace-point overhead: %lu.%03lu " + "us\n\n", trace_overhead/1000, trace_overhead%1000); + + if (verbose_trace) { + const char *name[4] = { [0 ... 3] = "<unused>" }; + struct list_head *pos; + int i = 0; + + list_for_each_prev(pos, &__ipipe_pipeline) { + struct ipipe_domain *ipd = + list_entry(pos, struct ipipe_domain, p_link); + + name[i] = ipd->name; + if (++i > 3) + break; + } + + seq_printf(m, + " +----- Hard IRQs ('|': locked)\n" + " |+---- %s\n" + " ||+--- %s\n" + " |||+-- %s\n" + " ||||+- %s%s\n" + " ||||| +---------- " + "Delay flag ('+': > %d us, '!': > %d us)\n" + " ||||| | +- " + "NMI noise ('N')\n" + " ||||| | |\n" + " Type User Val. Time Delay Function " + "(Parent)\n", + name[3], name[2], name[1], name[0], + name[0] ? " ('*': domain stalled, '+': current, " + "'#': current+stalled)" : "", + IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000); + } else + seq_printf(m, + " +--------------- Hard IRQs ('|': locked)\n" + " | +- Delay flag " + "('+': > %d us, '!': > %d us)\n" + " | |\n" + " Type Time Function (Parent)\n", + IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000); +} + +static void *__ipipe_max_prtrace_start(struct seq_file *m, loff_t *pos) +{ + loff_t n = *pos; + + mutex_lock(&out_mutex); + + if (!n) { + struct ipipe_trace_path *tp; + unsigned long length_usecs; + int points, cpu; + unsigned long flags; + + /* protect against max_path/frozen_path updates while we + * haven't locked our target path, also avoid recursively + * taking global_path_lock from NMI context */ + flags = __ipipe_global_path_lock(); + + /* find the longest of all per-cpu paths */ + print_path = NULL; + for_each_online_cpu(cpu) { + tp = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)]; + if ((print_path == NULL) || + (tp->length > print_path->length)) { + print_path = tp; + break; + } + } + print_path->dump_lock = 1; + + __ipipe_global_path_unlock(flags); + + /* does this path actually contain data? */ + if (print_path->end == print_path->begin) + return NULL; + + /* number of points inside the critical path */ + points = WRAP_POINT_NO(print_path->end-print_path->begin+1); + + /* pre- and post-tracing length, post-trace length was frozen + in __ipipe_trace, pre-trace may have to be reduced due to + buffer overrun */ + print_pre_trace = pre_trace; + print_post_trace = WRAP_POINT_NO(print_path->trace_pos - + print_path->end - 1); + if (points+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1) + print_pre_trace = IPIPE_TRACE_POINTS - 1 - points - + print_post_trace; + + length_usecs = ipipe_tsc2us(print_path->length); + seq_printf(m, "I-pipe worst-case tracing service on %s/ipipe-%s\n" + "------------------------------------------------------------\n", + UTS_RELEASE, IPIPE_ARCH_STRING); + seq_printf(m, "CPU: %d, Begin: %lld cycles, Trace Points: " + "%d (-%d/+%d), Length: %lu us\n", + cpu, print_path->point[print_path->begin].timestamp, + points, print_pre_trace, print_post_trace, length_usecs); + __ipipe_print_headline(m); + } + + /* check if we are inside the trace range */ + if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 + + print_pre_trace + print_post_trace)) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin - + print_pre_trace + n)]; +} + +static void *__ipipe_prtrace_next(struct seq_file *m, void *p, loff_t *pos) +{ + loff_t n = ++*pos; + + /* check if we are inside the trace range with the next entry */ + if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 + + print_pre_trace + print_post_trace)) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin - + print_pre_trace + *pos)]; +} + +static void __ipipe_prtrace_stop(struct seq_file *m, void *p) +{ + if (print_path) + print_path->dump_lock = 0; + mutex_unlock(&out_mutex); +} + +static int __ipipe_prtrace_show(struct seq_file *m, void *p) +{ + long time; + struct ipipe_trace_point *point = p; + char buf[16]; + + if (!point->eip) { + seq_puts(m, "-<invalid>-\n"); + return 0; + } + + __ipipe_print_pathmark(m, point); + __ipipe_trace_point_type(buf, point); + seq_puts(m, buf); + if (verbose_trace) + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + seq_puts(m, " "); + break; + + case IPIPE_TRACE_PID: + __ipipe_get_task_info(buf, point, 0); + seq_puts(m, buf); + break; + + case IPIPE_TRACE_EVENT: + __ipipe_get_event_date(buf, print_path, point); + seq_puts(m, buf); + break; + + default: + seq_printf(m, "0x%08lx ", point->v); + } + + time = __ipipe_signed_tsc2us(point->timestamp - + print_path->point[print_path->begin].timestamp); + seq_printf(m, "%5ld", time); + + __ipipe_print_delay(m, point); + __ipipe_print_symname(m, point->eip); + seq_puts(m, " ("); + __ipipe_print_symname(m, point->parent_eip); + seq_puts(m, ")\n"); + + return 0; +} + +static struct seq_operations __ipipe_max_ptrace_ops = { + .start = __ipipe_max_prtrace_start, + .next = __ipipe_prtrace_next, + .stop = __ipipe_prtrace_stop, + .show = __ipipe_prtrace_show +}; + +static int __ipipe_max_prtrace_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &__ipipe_max_ptrace_ops); +} + +static ssize_t +__ipipe_max_reset(struct file *file, const char __user *pbuffer, + size_t count, loff_t *data) +{ + mutex_lock(&out_mutex); + ipipe_trace_max_reset(); + mutex_unlock(&out_mutex); + + return count; +} + +struct file_operations __ipipe_max_prtrace_fops = { + .open = __ipipe_max_prtrace_open, + .read = seq_read, + .write = __ipipe_max_reset, + .llseek = seq_lseek, + .release = seq_release, +}; + +static void *__ipipe_frozen_prtrace_start(struct seq_file *m, loff_t *pos) +{ + loff_t n = *pos; + + mutex_lock(&out_mutex); + + if (!n) { + struct ipipe_trace_path *tp; + int cpu; + unsigned long flags; + + /* protect against max_path/frozen_path updates while we + * haven't locked our target path, also avoid recursively + * taking global_path_lock from NMI context */ + flags = __ipipe_global_path_lock(); + + /* find the first of all per-cpu frozen paths */ + print_path = NULL; + for_each_online_cpu(cpu) { + tp = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)]; + if (tp->end >= 0) { + print_path = tp; + break; + } + } + if (print_path) + print_path->dump_lock = 1; + + __ipipe_global_path_unlock(flags); + + if (!print_path) + return NULL; + + /* back- and post-tracing length, post-trace length was frozen + in __ipipe_trace, back-trace may have to be reduced due to + buffer overrun */ + print_pre_trace = back_trace-1; /* substract freeze point */ + print_post_trace = WRAP_POINT_NO(print_path->trace_pos - + print_path->end - 1); + if (1+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1) + print_pre_trace = IPIPE_TRACE_POINTS - 2 - + print_post_trace; + + seq_printf(m, "I-pipe frozen back-tracing service on %s/ipipe-%s\n" + "------------------------------------------------------" + "------\n", + UTS_RELEASE, IPIPE_ARCH_STRING); + seq_printf(m, "CPU: %d, Freeze: %lld cycles, Trace Points: %d (+%d)\n", + cpu, print_path->point[print_path->begin].timestamp, + print_pre_trace+1, print_post_trace); + __ipipe_print_headline(m); + } + + /* check if we are inside the trace range */ + if (n >= print_pre_trace + 1 + print_post_trace) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin- + print_pre_trace+n)]; +} + +static struct seq_operations __ipipe_frozen_ptrace_ops = { + .start = __ipipe_frozen_prtrace_start, + .next = __ipipe_prtrace_next, + .stop = __ipipe_prtrace_stop, + .show = __ipipe_prtrace_show +}; + +static int __ipipe_frozen_prtrace_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &__ipipe_frozen_ptrace_ops); +} + +static ssize_t +__ipipe_frozen_ctrl(struct file *file, const char __user *pbuffer, + size_t count, loff_t *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, pbuffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + ipipe_trace_frozen_reset(); + if (val > 0) + ipipe_trace_freeze(-1); + mutex_unlock(&out_mutex); + + return count; +} + +struct file_operations __ipipe_frozen_prtrace_fops = { + .open = __ipipe_frozen_prtrace_open, + .read = seq_read, + .write = __ipipe_frozen_ctrl, + .llseek = seq_lseek, + .release = seq_release, +}; + +static int __ipipe_rd_proc_val(char *page, char **start, off_t off, + int count, int *eof, void *data) +{ + int len; + + len = sprintf(page, "%u\n", *(int *)data); + len -= off; + if (len <= off + count) + *eof = 1; + *start = page + off; + if (len > count) + len = count; + if (len < 0) + len = 0; + + return len; +} + +static int __ipipe_wr_proc_val(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, buffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + *(int *)data = val; + mutex_unlock(&out_mutex); + + return count; +} + +static int __ipipe_rd_trigger(char *page, char **start, off_t off, int count, + int *eof, void *data) +{ + int len; + + if (!trigger_begin) + return 0; + + len = sprint_symbol(page, trigger_begin); + page[len++] = '\n'; + + len -= off; + if (len <= off + count) + *eof = 1; + *start = page + off; + if (len > count) + len = count; + if (len < 0) + len = 0; + + return len; +} + +static int __ipipe_wr_trigger(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char buf[KSYM_SYMBOL_LEN]; + unsigned long begin, end; + + if (count > sizeof(buf) - 1) + count = sizeof(buf) - 1; + if (copy_from_user(buf, buffer, count)) + return -EFAULT; + buf[count] = 0; + if (buf[count-1] == '\n') + buf[count-1] = 0; + + begin = kallsyms_lookup_name(buf); + if (!begin || !kallsyms_lookup_size_offset(begin, &end, NULL)) + return -ENOENT; + end += begin - 1; + + mutex_lock(&out_mutex); + /* invalidate the current range before setting a new one */ + trigger_end = 0; + wmb(); + ipipe_trace_frozen_reset(); + + /* set new range */ + trigger_begin = begin; + wmb(); + trigger_end = end; + mutex_unlock(&out_mutex); + + return count; +} + +#ifdef CONFIG_IPIPE_TRACE_MCOUNT +static void notrace +ipipe_trace_function(unsigned long ip, unsigned long parent_ip) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_FUNC, ip, parent_ip, 0); +} + +static struct ftrace_ops ipipe_trace_ops = { + .func = ipipe_trace_function +}; + +static int __ipipe_wr_enable(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, buffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + + if (ipipe_trace_enable) { + if (!val) + unregister_ftrace_function(&ipipe_trace_ops); + } else if (val) + register_ftrace_function(&ipipe_trace_ops); + + ipipe_trace_enable = val; + + mutex_unlock(&out_mutex); + + return count; +} +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ + +extern struct proc_dir_entry *ipipe_proc_root; + +static struct proc_dir_entry * __init +__ipipe_create_trace_proc_val(struct proc_dir_entry *trace_dir, + const char *name, int *value_ptr) +{ + struct proc_dir_entry *entry; + + entry = create_proc_entry(name, 0644, trace_dir); + if (entry) { + entry->data = value_ptr; + entry->read_proc = __ipipe_rd_proc_val; + entry->write_proc = __ipipe_wr_proc_val; + } + return entry; +} + +void __init __ipipe_init_tracer(void) +{ + struct proc_dir_entry *trace_dir; + struct proc_dir_entry *entry; + unsigned long long start, end, min = ULLONG_MAX; + int i; +#ifdef CONFIG_IPIPE_TRACE_VMALLOC + int cpu, path; + + for_each_possible_cpu(cpu) { + struct ipipe_trace_path *tp_buf; + + tp_buf = vmalloc_node(sizeof(struct ipipe_trace_path) * + IPIPE_TRACE_PATHS, cpu_to_node(cpu)); + if (!tp_buf) { + printk(KERN_ERR "I-pipe: " + "insufficient memory for trace buffer.\n"); + return; + } + memset(tp_buf, 0, + sizeof(struct ipipe_trace_path) * IPIPE_TRACE_PATHS); + for (path = 0; path < IPIPE_TRACE_PATHS; path++) { + tp_buf[path].begin = -1; + tp_buf[path].end = -1; + } + per_cpu(trace_path, cpu) = tp_buf; + } +#endif /* CONFIG_IPIPE_TRACE_VMALLOC */ + + /* Calculate minimum overhead of __ipipe_trace() */ + local_irq_disable_hw(); + for (i = 0; i < 100; i++) { + ipipe_read_tsc(start); + __ipipe_trace(IPIPE_TRACE_FUNC, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, 0); + ipipe_read_tsc(end); + + end -= start; + if (end < min) + min = end; + } + local_irq_enable_hw(); + trace_overhead = ipipe_tsc2ns(min); + +#ifdef CONFIG_IPIPE_TRACE_ENABLE + ipipe_trace_enable = 1; +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + register_ftrace_function(&ipipe_trace_ops); +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ +#endif /* CONFIG_IPIPE_TRACE_ENABLE */ + + trace_dir = create_proc_entry("trace", S_IFDIR, ipipe_proc_root); + + entry = create_proc_entry("max", 0644, trace_dir); + if (entry) + entry->proc_fops = &__ipipe_max_prtrace_fops; + + entry = create_proc_entry("frozen", 0644, trace_dir); + if (entry) + entry->proc_fops = &__ipipe_frozen_prtrace_fops; + + entry = create_proc_entry("trigger", 0644, trace_dir); + if (entry) { + entry->read_proc = __ipipe_rd_trigger; + entry->write_proc = __ipipe_wr_trigger; + } + + __ipipe_create_trace_proc_val(trace_dir, "pre_trace_points", + &pre_trace); + __ipipe_create_trace_proc_val(trace_dir, "post_trace_points", + &post_trace); + __ipipe_create_trace_proc_val(trace_dir, "back_trace_points", + &back_trace); + __ipipe_create_trace_proc_val(trace_dir, "verbose", + &verbose_trace); + entry = __ipipe_create_trace_proc_val(trace_dir, "enable", + &ipipe_trace_enable); +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + if (entry) + entry->write_proc = __ipipe_wr_enable; +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ +} diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c index ecc3fa2..e3e4263 100644 --- a/kernel/irq/chip.c +++ b/kernel/irq/chip.c @@ -15,6 +15,7 @@ #include <linux/module.h> #include <linux/interrupt.h> #include <linux/kernel_stat.h> +#include <linux/ipipe.h> #include "internals.h" @@ -425,7 +426,9 @@ handle_level_irq(unsigned int irq, struct irq_desc *desc) irqreturn_t action_ret; raw_spin_lock(&desc->lock); +#ifndef CONFIG_IPIPE mask_ack_irq(desc, irq); +#endif if (unlikely(desc->status & IRQ_INPROGRESS)) goto out_unlock; @@ -505,8 +508,13 @@ handle_fasteoi_irq(unsigned int irq, struct irq_desc *desc) raw_spin_lock(&desc->lock); desc->status &= ~IRQ_INPROGRESS; +#ifdef CONFIG_IPIPE + desc->chip->unmask(irq); +out: +#else out: desc->chip->eoi(irq); +#endif raw_spin_unlock(&desc->lock); } @@ -548,8 +556,10 @@ handle_edge_irq(unsigned int irq, struct irq_desc *desc) kstat_incr_irqs_this_cpu(irq, desc); /* Start handling the irq */ +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* Mark the IRQ currently in progress.*/ desc->status |= IRQ_INPROGRESS; @@ -603,8 +613,10 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc) kstat_incr_irqs_this_cpu(irq, desc); +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* CONFIG_IPIPE */ action_ret = handle_IRQ_event(irq, desc->action); if (!noirqdebug) @@ -614,6 +626,134 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc) desc->chip->eoi(irq); } +#ifdef CONFIG_IPIPE + +void __ipipe_ack_simple_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_end_simple_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_ack_level_irq(unsigned irq, struct irq_desc *desc) +{ + mask_ack_irq(desc, irq); +} + +void __ipipe_end_level_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->unmask) + desc->chip->unmask(irq); +} + +void __ipipe_ack_fasteoi_irq(unsigned irq, struct irq_desc *desc) +{ + desc->chip->eoi(irq); +} + +void __ipipe_end_fasteoi_irq(unsigned irq, struct irq_desc *desc) +{ + /* + * Non-requestable IRQs should not be masked in EOI handler. + */ + if (!(desc->status & IRQ_NOREQUEST)) + desc->chip->unmask(irq); +} + +void __ipipe_ack_edge_irq(unsigned irq, struct irq_desc *desc) +{ + desc->chip->ack(irq); +} + +void __ipipe_ack_percpu_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->ack) + desc->chip->ack(irq); +} + +void __ipipe_end_percpu_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->eoi) + desc->chip->eoi(irq); +} + +void __ipipe_end_edge_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_ack_bad_irq(unsigned irq, struct irq_desc *desc) +{ + static int done; + + handle_bad_irq(irq, desc); + + if (!done) { + printk(KERN_WARNING "%s: unknown flow handler for IRQ %d\n", + __FUNCTION__, irq); + done = 1; + } +} + +void __ipipe_noack_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_noend_irq(unsigned irq, struct irq_desc *desc) +{ +} + +irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, int is_chained) +{ + if (unlikely(handle == NULL)) { + desc->ipipe_ack = &__ipipe_ack_bad_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } else { + if (is_chained) { + desc->ipipe_ack = handle; + desc->ipipe_end = &__ipipe_noend_irq; + handle = __ipipe_noack_irq; + } else if (handle == &handle_simple_irq) { + desc->ipipe_ack = &__ipipe_ack_simple_irq; + desc->ipipe_end = &__ipipe_end_simple_irq; + } else if (handle == &handle_level_irq) { + desc->ipipe_ack = &__ipipe_ack_level_irq; + desc->ipipe_end = &__ipipe_end_level_irq; + } else if (handle == &handle_edge_irq) { + desc->ipipe_ack = &__ipipe_ack_edge_irq; + desc->ipipe_end = &__ipipe_end_edge_irq; + } else if (handle == &handle_fasteoi_irq) { + desc->ipipe_ack = &__ipipe_ack_fasteoi_irq; + desc->ipipe_end = &__ipipe_end_fasteoi_irq; + } else if (handle == &handle_percpu_irq) { + desc->ipipe_ack = &__ipipe_ack_percpu_irq; + desc->ipipe_end = &__ipipe_end_percpu_irq; + } else if (desc->chip == &no_irq_chip) { + desc->ipipe_ack = &__ipipe_noack_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } else { + desc->ipipe_ack = &__ipipe_ack_bad_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } + } + + /* Suppress intermediate trampoline routine. */ + ipipe_root_domain->irqs[desc->irq].acknowledge = desc->ipipe_ack; + + return handle; +} + +#else /* !CONFIG_IPIPE */ + +irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, int is_chained) +{ + return handle; +} + +#endif /* !CONFIG_IPIPE */ + void __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, const char *name) @@ -645,6 +785,8 @@ __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, chip_bus_lock(irq, desc); raw_spin_lock_irqsave(&desc->lock, flags); + handle = __fixup_irq_handler(desc, handle, is_chained); + /* Uninstall? */ if (handle == handle_bad_irq) { if (desc->chip != &no_irq_chip) diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c index 814940e..4c2e41d 100644 --- a/kernel/irq/handle.c +++ b/kernel/irq/handle.c @@ -462,8 +462,10 @@ unsigned int __do_IRQ(unsigned int irq) /* * No locking required for CPU-local interrupts: */ +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif if (likely(!(desc->status & IRQ_DISABLED))) { action_ret = handle_IRQ_event(irq, desc->action); if (!noirqdebug) @@ -474,8 +476,10 @@ unsigned int __do_IRQ(unsigned int irq) } raw_spin_lock(&desc->lock); +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* * REPLAY is when Linux resends an IRQ that was dropped earlier * WAITING is used by probe to mark irqs that are being tested diff --git a/kernel/lockdep.c b/kernel/lockdep.c index c62ec14..6d41b73 100644 --- a/kernel/lockdep.c +++ b/kernel/lockdep.c @@ -2327,7 +2327,7 @@ void trace_hardirqs_on_caller(unsigned long ip) /* we'll do an OFF -> ON transition: */ curr->hardirqs_enabled = 1; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (DEBUG_LOCKS_WARN_ON(current->hardirq_context)) return; @@ -2370,7 +2370,7 @@ void trace_hardirqs_off_caller(unsigned long ip) if (unlikely(!debug_locks || current->lockdep_recursion)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->hardirqs_enabled) { @@ -2402,7 +2402,7 @@ void trace_softirqs_on(unsigned long ip) if (unlikely(!debug_locks)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->softirqs_enabled) { @@ -2436,7 +2436,7 @@ void trace_softirqs_off(unsigned long ip) if (unlikely(!debug_locks)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->softirqs_enabled) { diff --git a/kernel/panic.c b/kernel/panic.c index c787333..ba50967 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -23,6 +23,7 @@ #include <linux/init.h> #include <linux/nmi.h> #include <linux/dmi.h> +#include <linux/ipipe_trace.h> int panic_on_oops; static unsigned long tainted_mask; @@ -307,6 +308,8 @@ void oops_enter(void) { tracing_off(); /* can't trust the integrity of the kernel anymore: */ + ipipe_trace_panic_freeze(); + ipipe_disable_context_check(ipipe_processor_id()); debug_locks_off(); do_oops_enter_exit(); } diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c index bbfe472..be44776 100644 --- a/kernel/power/hibernate.c +++ b/kernel/power/hibernate.c @@ -268,6 +268,7 @@ static int create_image(int platform_mode) goto Enable_cpus; local_irq_disable(); + local_irq_disable_hw_cond(); error = sysdev_suspend(PMSG_FREEZE); if (error) { @@ -297,6 +298,7 @@ static int create_image(int platform_mode) */ Enable_irqs: + local_irq_enable_hw_cond(); local_irq_enable(); Enable_cpus: @@ -389,6 +391,7 @@ static int resume_target_kernel(bool platform_mode) goto Enable_cpus; local_irq_disable(); + local_irq_disable_hw_cond(); error = sysdev_suspend(PMSG_QUIESCE); if (error) @@ -420,6 +423,7 @@ static int resume_target_kernel(bool platform_mode) sysdev_resume(); Enable_irqs: + local_irq_enable_hw_cond(); local_irq_enable(); Enable_cpus: @@ -501,6 +505,7 @@ int hibernation_platform_enter(void) goto Platform_finish; local_irq_disable(); + local_irq_disable_hw_cond(); sysdev_suspend(PMSG_HIBERNATE); hibernation_ops->enter(); /* We should never get here */ diff --git a/kernel/printk.c b/kernel/printk.c index 1751c45..a1e027a 100644 --- a/kernel/printk.c +++ b/kernel/printk.c @@ -566,6 +566,41 @@ static int have_callable_console(void) return 0; } +#ifdef CONFIG_IPIPE + +static IPIPE_DEFINE_SPINLOCK(__ipipe_printk_lock); + +static int __ipipe_printk_fill; + +static char __ipipe_printk_buf[__LOG_BUF_LEN]; + +void __ipipe_flush_printk (unsigned virq, void *cookie) +{ + char *p = __ipipe_printk_buf; + int len, lmax, out = 0; + unsigned long flags; + + goto start; + + do { + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); + start: + lmax = __ipipe_printk_fill; + while (out < lmax) { + len = strlen(p) + 1; + printk("%s",p); + p += len; + out += len; + } + spin_lock_irqsave(&__ipipe_printk_lock, flags); + } + while (__ipipe_printk_fill != lmax); + + __ipipe_printk_fill = 0; + + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); +} + /** * printk - print a kernel message * @fmt: format string @@ -590,6 +625,65 @@ static int have_callable_console(void) asmlinkage int printk(const char *fmt, ...) { + int r, fbytes, oldcount; + unsigned long flags; + int sprintk = 1; + int cs = -1; + va_list args; + + va_start(args, fmt); + + local_irq_save_hw(flags); + + if (test_bit(IPIPE_SPRINTK_FLAG, &__ipipe_current_domain->flags) || + oops_in_progress) + cs = ipipe_disable_context_check(ipipe_processor_id()); + else if (__ipipe_current_domain == ipipe_root_domain) { + struct ipipe_domain *dom; + + list_for_each_entry(dom, &__ipipe_pipeline, p_link) { + if (dom == ipipe_root_domain) + break; + if (test_bit(IPIPE_STALL_FLAG, + &ipipe_cpudom_var(dom, status))) + sprintk = 0; + } + } else + sprintk = 0; + + local_irq_restore_hw(flags); + + if (sprintk) { + r = vprintk(fmt, args); + if (cs != -1) + ipipe_restore_context_check(ipipe_processor_id(), cs); + goto out; + } + + spin_lock_irqsave(&__ipipe_printk_lock, flags); + + oldcount = __ipipe_printk_fill; + fbytes = __LOG_BUF_LEN - oldcount; + + if (fbytes > 1) { + r = vscnprintf(__ipipe_printk_buf + __ipipe_printk_fill, + fbytes, fmt, args) + 1; /* account for the null byte */ + __ipipe_printk_fill += r; + } else + r = 0; + + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); + + if (oldcount == 0) + ipipe_trigger_irq(__ipipe_printk_virq); +out: + va_end(args); + + return r; +} +#else /* !CONFIG_IPIPE */ +asmlinkage int printk(const char *fmt, ...) +{ va_list args; int r; @@ -599,6 +693,7 @@ asmlinkage int printk(const char *fmt, ...) return r; } +#endif /* CONFIG_IPIPE */ /* cpu currently holding logbuf_lock */ static volatile unsigned int printk_cpu = UINT_MAX; diff --git a/kernel/sched.c b/kernel/sched.c index 3a8fb30..cdb5bfd 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -2368,6 +2368,7 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) { int cpu, orig_cpu, this_cpu, success = 0; + unsigned int old_state; unsigned long flags; struct rq *rq, *orig_rq; @@ -2379,7 +2380,9 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, smp_wmb(); rq = orig_rq = task_rq_lock(p, &flags); update_rq_clock(rq); - if (!(p->state & state)) + old_state = p->state; + if (!(old_state & state) || + (old_state & (TASK_NOWAKEUP|TASK_ATOMICSWITCH))) goto out; if (p->se.on_rq) @@ -2869,13 +2872,15 @@ asmlinkage void schedule_tail(struct task_struct *prev) #endif if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); + + ipipe_init_notify(current); } /* * context_switch - switch to the new MM and the new * thread's register state. */ -static inline void +static inline int context_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { @@ -2917,12 +2922,23 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_to(prev, next, prev); barrier(); + +#ifdef CONFIG_IPIPE_DELAYED_ATOMICSW + current->state &= ~TASK_ATOMICSWITCH; +#else + prev->state &= ~TASK_ATOMICSWITCH; +#endif + if (task_hijacked(prev)) + return 1; + /* * this_rq must be evaluated again because prev may have moved * CPUs since it called schedule(), thus the 'rq' on its stack * frame will be invalid. */ finish_task_switch(this_rq(), prev); + + return 0; } /* @@ -5332,6 +5348,7 @@ notrace unsigned long get_parent_ip(unsigned long addr) void __kprobes add_preempt_count(int val) { + ipipe_check_context(ipipe_root_domain); #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? @@ -5354,6 +5371,7 @@ EXPORT_SYMBOL(add_preempt_count); void __kprobes sub_preempt_count(int val) { + ipipe_check_context(ipipe_root_domain); #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? @@ -5402,6 +5420,7 @@ static noinline void __schedule_bug(struct task_struct *prev) */ static inline void schedule_debug(struct task_struct *prev) { + ipipe_check_context(ipipe_root_domain); /* * Test if we are atomic. Since do_exit() needs to call into * schedule() atomically, we ignore that path for now. @@ -5478,7 +5497,7 @@ pick_next_task(struct rq *rq) /* * schedule() is the main scheduler function. */ -asmlinkage void __sched schedule(void) +asmlinkage int __sched schedule(void) { struct task_struct *prev, *next; unsigned long *switch_count; @@ -5492,6 +5511,9 @@ need_resched: rcu_sched_qs(cpu); prev = rq->curr; switch_count = &prev->nivcsw; + if (unlikely(prev->state & TASK_ATOMICSWITCH)) + /* Pop one disable level -- one still remains. */ + preempt_enable(); release_kernel_lock(prev); need_resched_nonpreemptible: @@ -5529,15 +5551,18 @@ need_resched_nonpreemptible: rq->curr = next; ++*switch_count; - context_switch(rq, prev, next); /* unlocks the rq */ + if (context_switch(rq, prev, next)) /* unlocks the rq */ + return 1; /* task hijacked by higher domain */ /* * the context switch might have flipped the stack from under * us, hence refresh the local variables. */ cpu = smp_processor_id(); rq = cpu_rq(cpu); - } else + } else { + prev->state &= ~TASK_ATOMICSWITCH; raw_spin_unlock_irq(&rq->lock); + } post_schedule(rq); @@ -5550,6 +5575,8 @@ need_resched_nonpreemptible: preempt_enable_no_resched(); if (need_resched()) goto need_resched; + + return 0; } EXPORT_SYMBOL(schedule); @@ -5633,7 +5660,8 @@ asmlinkage void __sched preempt_schedule(void) do { add_preempt_count(PREEMPT_ACTIVE); - schedule(); + if (schedule()) + return; sub_preempt_count(PREEMPT_ACTIVE); /* @@ -6396,6 +6424,7 @@ recheck: oldprio = p->prio; __setscheduler(rq, p, policy, param->sched_priority); + ipipe_setsched_notify(p); if (running) p->sched_class->set_curr_task(rq); @@ -7048,6 +7077,7 @@ void __cpuinit init_idle(struct task_struct *idle, int cpu) #else task_thread_info(idle)->preempt_count = 0; #endif + ipipe_check_context(ipipe_root_domain); /* * The idle tasks have their own, simple scheduling class: */ @@ -11039,3 +11069,64 @@ void synchronize_sched_expedited(void) EXPORT_SYMBOL_GPL(synchronize_sched_expedited); #endif /* #else #ifndef CONFIG_SMP */ + +#ifdef CONFIG_IPIPE + +int ipipe_setscheduler_root(struct task_struct *p, int policy, int prio) +{ + const struct sched_class *prev_class = p->sched_class; + int oldprio, on_rq, running; + unsigned long flags; + struct rq *rq; + + raw_spin_lock_irqsave(&p->pi_lock, flags); + rq = __task_rq_lock(p); + update_rq_clock(rq); + on_rq = p->se.on_rq; + running = task_current(rq, p); + if (on_rq) + deactivate_task(rq, p, 0); + if (running) + p->sched_class->put_prev_task(rq, p); + + p->sched_reset_on_fork = 0; + + oldprio = p->prio; + __setscheduler(rq, p, policy, prio); + ipipe_setsched_notify(p); + + if (running) + p->sched_class->set_curr_task(rq); + if (on_rq) { + activate_task(rq, p, 0); + + check_class_changed(rq, p, prev_class, oldprio, running); + } + __task_rq_unlock(rq); + raw_spin_unlock_irqrestore(&p->pi_lock, flags); + + rt_mutex_adjust_pi(p); + + return 0; +} +EXPORT_SYMBOL_GPL(ipipe_setscheduler_root); + +int ipipe_reenter_root(struct task_struct *prev, int policy, int prio) +{ + struct rq *rq = this_rq(); + + finish_task_switch(rq, prev); + + post_schedule(rq); + + (void)reacquire_kernel_lock(current); + preempt_enable_no_resched(); + + if (current->policy != policy || current->rt_priority != prio) + return ipipe_setscheduler_root(current, policy, prio); + + return 0; +} +EXPORT_SYMBOL_GPL(ipipe_reenter_root); + +#endif /* CONFIG_IPIPE */ diff --git a/kernel/signal.c b/kernel/signal.c index 934ae5e..7181e5f 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -541,6 +541,7 @@ void signal_wake_up(struct task_struct *t, int resume) unsigned int mask; set_tsk_thread_flag(t, TIF_SIGPENDING); + ipipe_sigwake_notify(t); /* TIF_SIGPENDING must be set first. */ /* * For SIGKILL, we want to wake it up in the stopped/traced/killable diff --git a/kernel/spinlock.c b/kernel/spinlock.c index be6517f..862aed4 100644 --- a/kernel/spinlock.c +++ b/kernel/spinlock.c @@ -26,7 +26,9 @@ * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) /* * The __lock_function inlines are taken from * include/linux/spinlock_api_smp.h diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c index b6b898d..75c2031 100644 --- a/kernel/time/tick-common.c +++ b/kernel/time/tick-common.c @@ -69,7 +69,7 @@ static void tick_periodic(int cpu) write_sequnlock(&xtime_lock); } - update_process_times(user_mode(get_irq_regs())); + update_root_process_times(get_irq_regs()); profile_tick(CPU_PROFILING); } @@ -177,6 +177,10 @@ static void tick_setup_device(struct tick_device *td, td->evtdev = newdev; + /* I-pipe: derive global tick IRQ from CPU 0 */ + if (cpu == 0) + ipipe_update_tick_evtdev(newdev); + /* * When the device is not per cpu, pin the interrupt to the * current cpu: diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index f992762..a9314ea 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -551,7 +551,7 @@ static void tick_nohz_handler(struct clock_event_device *dev) ts->idle_jiffies++; } - update_process_times(user_mode(regs)); + update_root_process_times(regs); profile_tick(CPU_PROFILING); while (tick_nohz_reprogram(ts, now)) { @@ -711,7 +711,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer) touch_softlockup_watchdog(); ts->idle_jiffies++; } - update_process_times(user_mode(regs)); + update_root_process_times(regs); profile_tick(CPU_PROFILING); } diff --git a/kernel/timer.c b/kernel/timer.c index c61a794..fb57c0c 100644 --- a/kernel/timer.c +++ b/kernel/timer.c @@ -1203,6 +1203,25 @@ void update_process_times(int user_tick) run_posix_cpu_timers(p); } +#ifdef CONFIG_IPIPE + +void update_root_process_times(struct pt_regs *regs) +{ + int cpu, user_tick = user_mode(regs); + + if (__ipipe_root_tick_p(regs)) { + update_process_times(user_tick); + return; + } + + run_local_timers(); + cpu = smp_processor_id(); + rcu_check_callbacks(cpu, user_tick); + run_posix_cpu_timers(current); +} + +#endif + /* * This function runs timers and the timer-tq in bottom half context. */ diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 60e2ce0..5b7a125 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -468,6 +468,7 @@ config DYNAMIC_FTRACE bool "enable/disable ftrace tracepoints dynamically" depends on FUNCTION_TRACER depends on HAVE_DYNAMIC_FTRACE + depends on !IPIPE_TRACE_MCOUNT default y help This option will modify all the calls to ftrace dynamically diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 1e6640f..5a3aa6b 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -28,6 +28,7 @@ #include <linux/ctype.h> #include <linux/list.h> #include <linux/hash.h> +#include <linux/ipipe.h> #include <trace/events/sched.h> @@ -1151,6 +1152,9 @@ static int __ftrace_modify_code(void *data) static void ftrace_run_update_code(int command) { +#ifdef CONFIG_IPIPE + unsigned long flags; +#endif /* CONFIG_IPIPE */ int ret; ret = ftrace_arch_code_modify_prepare(); @@ -1158,7 +1162,13 @@ static void ftrace_run_update_code(int command) if (ret) return; +#ifdef CONFIG_IPIPE + flags = ipipe_critical_enter(NULL); + __ftrace_modify_code(&command); + ipipe_critical_exit(flags); +#else /* !CONFIG_IPIPE */ stop_machine(__ftrace_modify_code, &command, NULL); +#endif /* !CONFIG_IPIPE */ ret = ftrace_arch_code_modify_post_process(); FTRACE_WARN_ON(ret); @@ -2663,9 +2673,9 @@ static int ftrace_process_locs(struct module *mod, } /* disable interrupts to prevent kstop machine */ - local_irq_save(flags); + local_irq_save_hw_notrace(flags); ftrace_update_code(mod); - local_irq_restore(flags); + local_irq_restore_hw_notrace(flags); mutex_unlock(&ftrace_lock); return 0; @@ -2744,9 +2754,9 @@ void __init ftrace_init(void) /* Keep the ftrace pointer to the stub */ addr = (unsigned long)ftrace_stub; - local_irq_save(flags); + local_irq_save_hw_notrace(flags); ftrace_dyn_arch_init(&addr); - local_irq_restore(flags); + local_irq_restore_hw_notrace(flags); /* ftrace_dyn_arch_init places the return code in addr */ if (addr) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 25c3ed5..6b9a5d0 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -136,6 +136,8 @@ config DEBUG_SECTION_MISMATCH - Enable verbose reporting from modpost to help solving the section mismatches reported. +source "kernel/ipipe/Kconfig.debug" + config DEBUG_KERNEL bool "Kernel debugging" help diff --git a/lib/bust_spinlocks.c b/lib/bust_spinlocks.c index 9681d54..2dba50c 100644 --- a/lib/bust_spinlocks.c +++ b/lib/bust_spinlocks.c @@ -13,6 +13,7 @@ #include <linux/wait.h> #include <linux/vt_kern.h> #include <linux/console.h> +#include <linux/ipipe_trace.h> void __attribute__((weak)) bust_spinlocks(int yes) @@ -24,6 +25,7 @@ void __attribute__((weak)) bust_spinlocks(int yes) unblank_screen(); #endif console_unblank(); + ipipe_trace_panic_dump(); if (--oops_in_progress == 0) wake_up_klogd(); } diff --git a/lib/ioremap.c b/lib/ioremap.c index 14c6078..a275469 100644 --- a/lib/ioremap.c +++ b/lib/ioremap.c @@ -85,8 +85,8 @@ int ioremap_page_range(unsigned long addr, if (err) break; } while (pgd++, addr = next, addr != end); - - flush_cache_vmap(start, end); + __ipipe_pin_range_globally(start, end); + flush_cache_vmap(start, end); return err; } diff --git a/lib/smp_processor_id.c b/lib/smp_processor_id.c index 4689cb0..3d12764 100644 --- a/lib/smp_processor_id.c +++ b/lib/smp_processor_id.c @@ -12,10 +12,13 @@ notrace unsigned int debug_smp_processor_id(void) unsigned long preempt_count = preempt_count(); int this_cpu = raw_smp_processor_id(); + if (!ipipe_root_domain_p) + goto out; + if (likely(preempt_count)) goto out; - if (irqs_disabled()) + if (irqs_disabled() || irqs_disabled_hw()) goto out; /* diff --git a/mm/memory.c b/mm/memory.c index 09e4b1b..bef0089 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -56,6 +56,7 @@ #include <linux/kallsyms.h> #include <linux/swapops.h> #include <linux/elf.h> +#include <linux/vmalloc.h> #include <asm/io.h> #include <asm/pgalloc.h> @@ -566,6 +567,32 @@ out: return pfn_to_page(pfn); } +static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) +{ + /* + * If the source page was a PFN mapping, we don't have + * a "struct page" for it. We do a best-effort copy by + * just copying from the original user address. If that + * fails, we just zero-fill it. Live with it. + */ + if (unlikely(!src)) { + void *kaddr = kmap_atomic(dst, KM_USER0); + void __user *uaddr = (void __user *)(va & PAGE_MASK); + + /* + * This really shouldn't fail, because the page is there + * in the page tables. But it might just be unreadable, + * in which case we just give up and fill the result with + * zeroes. + */ + if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) + memset(kaddr, 0, PAGE_SIZE); + kunmap_atomic(kaddr, KM_USER0); + flush_dcache_page(dst); + } else + copy_user_highpage(dst, src, va, vma); +} + /* * copy one vm_area from one task to the other. Assumes the page tables * already present in the new task to be cleared in the whole range @@ -574,8 +601,8 @@ out: static inline unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, - pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, - unsigned long addr, int *rss) + pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, + unsigned long addr, int *rss, struct page *uncow_page) { unsigned long vm_flags = vma->vm_flags; pte_t pte = *src_pte; @@ -616,6 +643,21 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * in the parent and the child */ if (is_cow_mapping(vm_flags)) { +#ifdef CONFIG_IPIPE + if (uncow_page) { + struct page *old_page = vm_normal_page(vma, addr, pte); + cow_user_page(uncow_page, old_page, addr, vma); + pte = mk_pte(uncow_page, vma->vm_page_prot); + + if (vm_flags & VM_SHARED) + pte = pte_mkclean(pte); + pte = pte_mkold(pte); + + page_add_new_anon_rmap(uncow_page, vma, addr); + rss[!!PageAnon(uncow_page)]++; + goto out_set_pte; + } +#endif /* CONFIG_IPIPE */ ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -650,12 +692,26 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, int progress = 0; int rss[2]; swp_entry_t entry = (swp_entry_t){0}; - + struct page *uncow_page = NULL; +#ifdef CONFIG_IPIPE + int do_cow_break = 0; +again: + if (do_cow_break) { + uncow_page = alloc_page_vma(GFP_HIGHUSER, vma, addr); + if (uncow_page == NULL) + return -ENOMEM; + do_cow_break = 0; + } +#else again: rss[1] = rss[0] = 0; +#endif dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); - if (!dst_pte) + if (!dst_pte) { + if (uncow_page) + page_cache_release(uncow_page); return -ENOMEM; + } src_pte = pte_offset_map_nested(src_pmd, addr); src_ptl = pte_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -678,8 +734,25 @@ again: progress++; continue; } +#ifdef CONFIG_IPIPE + if (likely(uncow_page == NULL) && likely(pte_present(*src_pte))) { + if (is_cow_mapping(vma->vm_flags) && + test_bit(MMF_VM_PINNED, &src_mm->flags) && + ((vma->vm_flags|src_mm->def_flags) & VM_LOCKED)) { + arch_leave_lazy_mmu_mode(); + spin_unlock(src_ptl); + pte_unmap_nested(src_pte); + add_mm_rss(dst_mm, rss[0], rss[1]); + pte_unmap_unlock(dst_pte, dst_ptl); + cond_resched(); + do_cow_break = 1; + goto again; + } + } +#endif entry.val = copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, - vma, addr, rss); + vma, addr, rss, uncow_page); + uncow_page = NULL; if (entry.val) break; progress += 8; @@ -1956,32 +2029,6 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) return pte; } -static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) -{ - /* - * If the source page was a PFN mapping, we don't have - * a "struct page" for it. We do a best-effort copy by - * just copying from the original user address. If that - * fails, we just zero-fill it. Live with it. - */ - if (unlikely(!src)) { - void *kaddr = kmap_atomic(dst, KM_USER0); - void __user *uaddr = (void __user *)(va & PAGE_MASK); - - /* - * This really shouldn't fail, because the page is there - * in the page tables. But it might just be unreadable, - * in which case we just give up and fill the result with - * zeroes. - */ - if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) - memset(kaddr, 0, PAGE_SIZE); - kunmap_atomic(kaddr, KM_USER0); - flush_dcache_page(dst); - } else - copy_user_highpage(dst, src, va, vma); -} - /* * This routine handles present pages, when users try to write * to a shared page. It is done by copying the page to a new address @@ -3402,3 +3449,111 @@ void might_fault(void) } EXPORT_SYMBOL(might_fault); #endif + +#ifdef CONFIG_IPIPE + +static inline int ipipe_pin_pte_range(struct mm_struct *mm, pmd_t *pmd, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + spinlock_t *ptl; + pte_t *pte; + + do { + pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + continue; + + if (!pte_present(*pte) || pte_write(*pte)) { + pte_unmap_unlock(pte, ptl); + continue; + } + + if (do_wp_page(mm, vma, addr, pte, pmd, ptl, *pte) == VM_FAULT_OOM) + return -ENOMEM; + } while (addr += PAGE_SIZE, addr != end); + return 0; +} + +static inline int ipipe_pin_pmd_range(struct mm_struct *mm, pud_t *pud, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + unsigned long next; + pmd_t *pmd; + + pmd = pmd_offset(pud, addr); + do { + next = pmd_addr_end(addr, end); + if (pmd_none_or_clear_bad(pmd)) + continue; + if (ipipe_pin_pte_range(mm, pmd, vma, addr, next)) + return -ENOMEM; + } while (pmd++, addr = next, addr != end); + return 0; +} + +static inline int ipipe_pin_pud_range(struct mm_struct *mm, pgd_t *pgd, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + unsigned long next; + pud_t *pud; + + pud = pud_offset(pgd, addr); + do { + next = pud_addr_end(addr, end); + if (pud_none_or_clear_bad(pud)) + continue; + if (ipipe_pin_pmd_range(mm, pud, vma, addr, next)) + return -ENOMEM; + } while (pud++, addr = next, addr != end); + return 0; +} + +int ipipe_disable_ondemand_mappings(struct task_struct *tsk) +{ + unsigned long addr, next, end; + struct vm_area_struct *vma; + struct mm_struct *mm; + int result = 0; + pgd_t *pgd; + + mm = get_task_mm(tsk); + if (!mm) + return -EPERM; + + down_write(&mm->mmap_sem); + if (test_bit(MMF_VM_PINNED, &mm->flags)) + goto done_mm; + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (!is_cow_mapping(vma->vm_flags) + || !(vma->vm_flags & VM_WRITE)) + continue; + + addr = vma->vm_start; + end = vma->vm_end; + + pgd = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none_or_clear_bad(pgd)) + continue; + if (ipipe_pin_pud_range(mm, pgd, vma, addr, next)) { + result = -ENOMEM; + goto done_mm; + } + } while (pgd++, addr = next, addr != end); + } + set_bit(MMF_VM_PINNED, &mm->flags); + + done_mm: + up_write(&mm->mmap_sem); + mmput(mm); + return result; +} + +EXPORT_SYMBOL(ipipe_disable_ondemand_mappings); + +#endif diff --git a/mm/mmu_context.c b/mm/mmu_context.c index ded9081..cb2ac0e 100644 --- a/mm/mmu_context.c +++ b/mm/mmu_context.c @@ -23,15 +23,18 @@ void use_mm(struct mm_struct *mm) { struct mm_struct *active_mm; struct task_struct *tsk = current; + unsigned long flags; task_lock(tsk); active_mm = tsk->active_mm; + ipipe_mm_switch_protect(flags); if (active_mm != mm) { atomic_inc(&mm->mm_count); tsk->active_mm = mm; } tsk->mm = mm; - switch_mm(active_mm, mm, tsk); + __switch_mm(active_mm, mm, tsk); + ipipe_mm_switch_unprotect(flags); task_unlock(tsk); if (active_mm != mm) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ae00746..ca1597c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -172,6 +172,8 @@ static int vmap_page_range_noflush(unsigned long start, unsigned long end, return err; } while (pgd++, addr = next, addr != end); + __ipipe_pin_range_globally(start, end); + return nr; } ^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 8:11 ` Fredrik Asplund @ 2010-07-01 8:59 ` Fredrik Asplund 2010-07-01 9:54 ` Gilles Chanteperdrix 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-01 8:59 UTC (permalink / raw) To: gilles.chanteperdrix@xenomai.org, cavalkaf@domain.hid; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 1242 bytes --] >>I am actually trying to do the exact same thing. > >Looking at forums and email archives it seems a lot of >people have been trying to do this, but that no-one has posted their results. :/ > >>Anyway, I've managed to change the files manually - the changes seem >>very minor (the changes in include/asm/cacheflush.h just move a >>function elsewhere, no code changes at all). > >I have attached a patch file showing the changes I have done (and an updated Adeos patch file). This was >fairly trivial, but I guess the problem could be if Gumstix have added some changes which will require >additional patching for Adeos. Anyway, could you share a patch file of exactly what you have changed? > >>So, any clues? I can send my changes and logs tomorrow morning, if they are of any help. > >Sure, please do so. I don´t have much time to put on this, so I have actually only have >had to go through the source files since yesterday. Today (or during the weekend if I don´t >get the time) I will create a new defconfig file, compile, test and debug. If you have >updated BitBake file, Defconfig file, Logs, etc., please share them. Sorry, I attached the wrong updated Xenomai Patch file, this is the one I use. / Fredrik [-- Attachment #2: adeos-ipipe-2.6.33-arm-1.16-01_updated.patch --] [-- Type: application/octet-stream, Size: 437333 bytes --] diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 184a6bd..1a8674b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -254,6 +254,8 @@ config ARCH_VERSATILE config ARCH_AT91 bool "Atmel AT91" select GENERIC_GPIO + select GENERIC_TIME if IPIPE + select GENERIC_CLOCKEVENTS if IPIPE select ARCH_REQUIRE_GPIOLIB select HAVE_CLK help @@ -514,7 +516,7 @@ config ARCH_KS8695 bool "Micrel/Kendin KS8695" select CPU_ARM922T select GENERIC_GPIO - select ARCH_REQUIRE_GPIOLIB + select ARCH_REQUIRE_GPIOLIB help Support for Micrel/Kendin KS8695 "Centaur" (ARM922T) based System-on-Chip devices. @@ -1070,6 +1072,8 @@ config LOCAL_TIMERS accounting to be spread across the timer interval, preventing a "thundering herd" at every timer tick. +source kernel/ipipe/Kconfig + source kernel/Kconfig.preempt config HZ diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S index 4fddc50..cbb2e4c 100644 --- a/arch/arm/boot/compressed/head.S +++ b/arch/arm/boot/compressed/head.S @@ -1074,6 +1074,15 @@ memdump: mov r12, r0 mov pc, r10 #endif +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + .text + .align 0 + .type mcount %function + .global mcount +mcount: + mov pc, lr @ just return +#endif + .ltorg reloc_end: diff --git a/arch/arm/common/it8152.c b/arch/arm/common/it8152.c index 2793447..5ec1999 100644 --- a/arch/arm/common/it8152.c +++ b/arch/arm/common/it8152.c @@ -26,6 +26,7 @@ #include <linux/ioport.h> #include <linux/irq.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <asm/mach/pci.h> #include <asm/hardware/it8152.h> @@ -120,21 +121,21 @@ void it8152_irq_demux(unsigned int irq, struct irq_desc *desc) bits_pd &= ((1 << IT8152_PD_IRQ_COUNT) - 1); while (bits_pd) { i = __ffs(bits_pd); - generic_handle_irq(IT8152_PD_IRQ(i)); + ipipe_handle_irq_cond(IT8152_PD_IRQ(i)); bits_pd &= ~(1 << i); } bits_lp &= ((1 << IT8152_LP_IRQ_COUNT) - 1); while (bits_lp) { i = __ffs(bits_lp); - generic_handle_irq(IT8152_LP_IRQ(i)); + ipipe_handle_irq_cond(IT8152_LP_IRQ(i)); bits_lp &= ~(1 << i); } bits_ld &= ((1 << IT8152_LD_IRQ_COUNT) - 1); while (bits_ld) { i = __ffs(bits_ld); - generic_handle_irq(IT8152_LD_IRQ(i)); + ipipe_handle_irq_cond(IT8152_LD_IRQ(i)); bits_ld &= ~(1 << i); } } diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 00f46d9..84ea0d3 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -81,6 +81,18 @@ .macro enable_irq_notrace cpsie i .endm + + .macro disable_irq_cond +#ifdef CONFIG_IPIPE + cpsid i +#endif /* CONFIG_IPIPE */ + .endm + + .macro enable_irq_cond +#ifdef CONFIG_IPIPE + cpsie i +#endif /* CONFIG_IPIPE */ + .endm #else .macro disable_irq_notrace msr cpsr_c, #PSR_I_BIT | SVC_MODE @@ -89,6 +101,18 @@ .macro enable_irq_notrace msr cpsr_c, #SVC_MODE .endm + + .macro disable_irq_cond +#ifdef CONFIG_IPIPE + msr cpsr_c, #PSR_I_BIT | SVC_MODE +#endif /* CONFIG_IPIPE */ + .endm + + .macro enable_irq_cond +#ifdef CONFIG_IPIPE + msr cpsr_c, #SVC_MODE +#endif /* CONFIG_IPIPE */ + .endm #endif .macro asm_trace_hardirqs_off diff --git a/arch/arm/include/asm/atomic.h b/arch/arm/include/asm/atomic.h index d0daeab..d4f9618 100644 --- a/arch/arm/include/asm/atomic.h +++ b/arch/arm/include/asm/atomic.h @@ -158,10 +158,10 @@ static inline int atomic_add_return(int i, atomic_t *v) unsigned long flags; int val; - raw_local_irq_save(flags); + local_irq_save_hw(flags); val = v->counter; v->counter = val += i; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return val; } @@ -172,10 +172,10 @@ static inline int atomic_sub_return(int i, atomic_t *v) unsigned long flags; int val; - raw_local_irq_save(flags); + local_irq_save_hw(flags); val = v->counter; v->counter = val -= i; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return val; } @@ -186,11 +186,11 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new) int ret; unsigned long flags; - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = v->counter; if (likely(ret == old)) v->counter = new; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return ret; } @@ -199,9 +199,9 @@ static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) { unsigned long flags; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *addr &= ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } #endif /* __LINUX_ARM_ARCH__ */ diff --git a/arch/arm/include/asm/bitops.h b/arch/arm/include/asm/bitops.h index 338ff19..acc35cd 100644 --- a/arch/arm/include/asm/bitops.h +++ b/arch/arm/include/asm/bitops.h @@ -41,9 +41,9 @@ static inline void ____atomic_set_bit(unsigned int bit, volatile unsigned long * p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p |= mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline void ____atomic_clear_bit(unsigned int bit, volatile unsigned long *p) @@ -53,9 +53,9 @@ static inline void ____atomic_clear_bit(unsigned int bit, volatile unsigned long p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p &= ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline void ____atomic_change_bit(unsigned int bit, volatile unsigned long *p) @@ -65,9 +65,9 @@ static inline void ____atomic_change_bit(unsigned int bit, volatile unsigned lon p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p ^= mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline int @@ -79,10 +79,10 @@ ____atomic_test_and_set_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res | mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } @@ -96,10 +96,10 @@ ____atomic_test_and_clear_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res & ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } @@ -113,10 +113,10 @@ ____atomic_test_and_change_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res ^ mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } diff --git a/arch/arm/include/asm/cpu-multi32.h b/arch/arm/include/asm/cpu-multi32.h index e2b5b0b..18ca8a0 100644 --- a/arch/arm/include/asm/cpu-multi32.h +++ b/arch/arm/include/asm/cpu-multi32.h @@ -52,7 +52,8 @@ extern struct processor { /* * Set the page table */ - void (*switch_mm)(unsigned long pgd_phys, struct mm_struct *mm); + void (*switch_mm)(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); /* * Set a possibly extended PTE. Non-extended PTEs should * ignore 'ext'. @@ -66,4 +67,4 @@ extern struct processor { #define cpu_do_idle() processor._do_idle() #define cpu_dcache_clean_area(addr,sz) processor.dcache_clean_area(addr,sz) #define cpu_set_pte_ext(ptep,pte,ext) processor.set_pte_ext(ptep,pte,ext) -#define cpu_do_switch_mm(pgd,mm) processor.switch_mm(pgd,mm) +#define cpu_do_switch_mm(pgd,mm,flush) processor.switch_mm(pgd,mm,flush) diff --git a/arch/arm/include/asm/cpu-single.h b/arch/arm/include/asm/cpu-single.h index f073a6d..ec556e3 100644 --- a/arch/arm/include/asm/cpu-single.h +++ b/arch/arm/include/asm/cpu-single.h @@ -39,6 +39,7 @@ extern void cpu_proc_init(void); extern void cpu_proc_fin(void); extern int cpu_do_idle(void); extern void cpu_dcache_clean_area(void *, int); -extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm); +extern void cpu_do_switch_mm(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); extern void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext); extern void cpu_reset(unsigned long addr) __attribute__((noreturn)); diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h new file mode 100644 index 0000000..cfdf14a --- /dev/null +++ b/arch/arm/include/asm/fcse.h @@ -0,0 +1,178 @@ +/* + * Filename: arch/arm/include/asm/fcse.h + * Description: ARM Process ID (PID) includes for Fast Address Space Switching + * (FASS) in ARM Linux. + * + * Copyright: (C) 2001, 2002 Adam Wiggins <awiggins@cse.unsw.edu.au> + * (C) 2007 Sebastian Smolorz <ssm@emlix.com> + * (C) 2008 Richard Cochran + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of teh GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#ifndef __ASM_ARM_FCSE_H +#define __ASM_ARM_FCSE_H + +#ifdef CONFIG_ARM_FCSE + +#include <linux/mm_types.h> /* For struct mm_struct */ +#include <linux/sched.h> +#include <linux/hardirq.h> + +#include <asm/bitops.h> +#include <asm/cachetype.h> + +#define FCSE_PID_SHIFT 25 + +/* Size of PID relocation area */ +#define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT) + +/* Mask to get rid of PID from relocated address */ +#define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1) + +extern unsigned long fcse_pids_cache_dirty[]; + +#ifdef CONFIG_ARM_FCSE_DEBUG +#define FCSE_BUG_ON(expr) BUG_ON(expr) +#else /* !CONFIG_ARM_FCSE_DEBUG */ +#define FCSE_BUG_ON(expr) do { } while(0) +#endif /* !CONFIG_ARM_FCSE_DEBUG */ + +#if defined(CONFIG_ARM_FCSE_DYNPID) && !defined(CONFIG_PREEMPT_NONE) +#define fcse_check_context(mm) \ + FCSE_BUG_ON(!in_atomic() \ + && (mm)->context.fcse.active \ + && atomic_read(&(mm)->mm_users) \ + && !(mm)->core_state \ + && !rwsem_is_locked(&(mm)->mmap_sem) \ + && !irqs_disabled_hw()); +#else /* !CONFIG_ARM_FCSE_DYNPID */ +#define fcse_check_context(mm) do { (void)(mm); } while(0) +#endif /* !CONFIG_ARM_FCSE_DYNPID */ + +int fcse_pid_alloc(struct mm_struct *mm); +void fcse_pid_free(struct mm_struct *mm); +unsigned fcse_flush_all_start(struct mm_struct *mm); +int fcse_flush_all_done(struct mm_struct *mm, unsigned seq, unsigned dirty); +unsigned long +fcse_check_mmap_inner(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl); + +/* Sets the CPU's PID Register */ +static inline void fcse_pid_set(unsigned long pid) +{ + __asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0" + : /* */: "r" (pid) : "cc", "memory"); +} + +static inline unsigned long +fcse_va_to_mva(struct mm_struct *mm, unsigned long va) +{ + if (cache_is_vivt() && va < FCSE_PID_TASK_SIZE) { + fcse_check_context(mm); + return mm->context.fcse.pid | va; + } + return va; +} + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +struct fcse_user { + struct mm_struct *mm; + unsigned count; +}; +extern struct fcse_user fcse_pids_user[]; +int fcse_switch_mm_inner(struct mm_struct *prev, struct mm_struct *next); +void fcse_pid_reference(unsigned pid); + +static inline int fcse_switch_mm(struct mm_struct *prev, struct mm_struct *next) +{ + if (!cache_is_vivt()) + return 0; + + return fcse_switch_mm_inner(prev, next); +} + +static inline int fcse_mm_in_cache(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + int res; + fcse_check_context(mm); + res = test_bit(fcse_pid, fcse_pids_cache_dirty) + && fcse_pids_user[fcse_pid].mm == mm; + return res; +} + +static inline unsigned long +fcse_check_mmap_addr(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ + const unsigned long stack_base = ALIGN(mm->start_stack, PAGE_SIZE) + - current->signal->rlim[RLIMIT_STACK].rlim_cur; + + if (addr + len <= stack_base) + return addr; + + return fcse_check_mmap_inner(mm, start_addr, addr, len, fl); +} + +#else /* CONFIG_ARM_FCSE_GUARANTEED */ +static inline int +fcse_switch_mm(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned fcse_pid; + + if (!cache_is_vivt()) + return 0; + + fcse_pid = next->context.fcse.pid >> FCSE_PID_SHIFT; + set_bit(fcse_pid, fcse_pids_cache_dirty); + fcse_pid_set(next->context.fcse.pid); + return 0; +} + +static inline int fcse_mm_in_cache(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + return test_bit(fcse_pid, fcse_pids_cache_dirty); +} + +static inline unsigned long +fcse_check_mmap_addr(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ + if (addr + len <= FCSE_TASK_SIZE) + return addr; + + return fcse_check_mmap_inner(mm, start_addr, addr, len, fl); +} +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ + +static inline void fcse_mark_dirty(struct mm_struct *mm) +{ + if (cache_is_vivt()) { + set_bit(mm->context.fcse.pid >> FCSE_PID_SHIFT, + fcse_pids_cache_dirty); + FCSE_BUG_ON(!fcse_mm_in_cache(mm)); + } +} + +#else /* ! CONFIG_ARM_FCSE */ +#define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); }) +#define fcse_mark_dirty(mm) do { (void)(mm); } while(0) +#define fcse_switch_mm(prev, next) (1) +#define fcse_flush_all_start(mm) (0) +#define fcse_flush_all_done(mm, seq, dirty) (1) +#define fcse_notify_flush_all() do { } while (0) +#define fcse_mm_in_cache(mm) \ + (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) +#endif /* ! CONFIG_ARM_FCSE */ + +#ifdef CONFIG_ARM_FCSE_MESSAGES +void fcse_notify_segv(struct mm_struct *mm, + unsigned long addr, struct pt_regs *regs); +#else /* !FCSE_MESSAGES */ +#define fcse_notify_segv(mm, addr, regs) do { } while(0) +#endif /* !FCSE_MESSAGES */ + +#endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/ipipe.h b/arch/arm/include/asm/ipipe.h new file mode 100644 index 0000000..8ae0cb3 --- /dev/null +++ b/arch/arm/include/asm/ipipe.h @@ -0,0 +1,274 @@ +/* -*- linux-c -*- + * arch/arm/include/asm/ipipe.h + * + * Copyright (C) 2002-2005 Philippe Gerum. + * Copyright (C) 2005 Stelian Pop. + * Copyright (C) 2006-2008 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __ARM_IPIPE_H +#define __ARM_IPIPE_H + +#ifdef CONFIG_IPIPE + +#include <linux/ipipe_percpu.h> +#include <mach/irqs.h> /* For __IPIPE_FEATURE_PIC_MUTE */ + +#define IPIPE_ARCH_STRING "1.16-01" +#define IPIPE_MAJOR_NUMBER 1 +#define IPIPE_MINOR_NUMBER 16 +#define IPIPE_PATCH_NUMBER 1 + +#ifdef CONFIG_SMP +#error "I-pipe/arm: SMP not yet implemented" +#define ipipe_processor_id() (current_thread_info()->cpu) +#else /* !CONFIG_SMP */ +#define ipipe_processor_id() 0 +#endif /* CONFIG_SMP */ + +#define smp_processor_id_hw() ipipe_processor_id() + +#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + +#define prepare_arch_switch(next) \ + do { \ + local_irq_enable_hw(); \ + ipipe_schedule_notify(current, next); \ + } while(0) + +#define task_hijacked(p) \ + ({ \ + int x = !ipipe_root_domain_p; \ + clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \ + x; \ + }) + +#define ipipe_mm_switch_protect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#define ipipe_mm_switch_unprotect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + +#define prepare_arch_switch(next) \ + do { \ + ipipe_schedule_notify(current ,next); \ + local_irq_disable_hw(); \ + } while(0) + +#define task_hijacked(p) \ + ({ \ + int x = !ipipe_root_domain_p; \ + __clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \ + if (!x) local_irq_enable_hw(); x; \ + }) + +#define ipipe_mm_switch_protect(flags) \ + local_irq_save_hw_cond(flags) + +#define ipipe_mm_switch_unprotect(flags) \ + local_irq_restore_hw_cond(flags) + +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + +extern unsigned long arm_return_addr(int level); + +#define BROKEN_BUILTIN_RETURN_ADDRESS +#define __BUILTIN_RETURN_ADDRESS0 arm_return_addr(0) +#define __BUILTIN_RETURN_ADDRESS1 arm_return_addr(1) + + +struct ipipe_domain; + +#define IPIPE_TSC_TYPE_NONE 0 +#define IPIPE_TSC_TYPE_FREERUNNING 1 +#define IPIPE_TSC_TYPE_DECREMENTER 2 + +struct __ipipe_tscinfo { + unsigned type; + union { + struct { + unsigned *counter; /* Hw counter physical address */ + unsigned mask; /* Significant bits in the hw counter. */ + unsigned long long *tsc; /* 64 bits tsc value. */ + } fr; + struct { + unsigned *counter; /* Hw counter physical address */ + unsigned mask; /* Significant bits in the hw counter. */ + unsigned *last_cnt; /* Counter value when updating + tsc value. */ + unsigned long long *tsc; /* 64 bits tsc value. */ + } dec; + } u; +}; + +struct ipipe_sysinfo { + + int ncpus; /* Number of CPUs on board */ + u64 cpufreq; /* CPU frequency (in Hz) */ + + /* Arch-dependent block */ + + struct { + unsigned tmirq; /* Timer tick IRQ */ + u64 tmfreq; /* Timer frequency */ + struct __ipipe_tscinfo tsc; /* exported data for u.s. tsc */ + } archdep; +}; + +DECLARE_PER_CPU(struct mm_struct *,ipipe_active_mm); +/* arch specific stuff */ +extern void *__ipipe_tsc_area; +extern int __ipipe_mach_timerint; +extern int __ipipe_mach_timerstolen; +extern unsigned int __ipipe_mach_ticks_per_jiffy; +extern void __ipipe_mach_acktimer(void); +extern unsigned long long __ipipe_mach_get_tsc(void); +extern void __ipipe_mach_set_dec(unsigned long); +extern void __ipipe_mach_release_timer(void); +extern unsigned long __ipipe_mach_get_dec(void); +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info); +int __ipipe_check_tickdev(const char *devname); + +#define ipipe_read_tsc(t) do { t = __ipipe_mach_get_tsc(); } while (0) +#define __ipipe_read_timebase() __ipipe_mach_get_tsc() + +#define ipipe_cpu_freq() (HZ * __ipipe_mach_ticks_per_jiffy) +#define ipipe_tsc2ns(t) \ +({ \ + unsigned long long delta = (t)*1000; \ + do_div(delta, ipipe_cpu_freq() / 1000000 + 1); \ + (unsigned long)delta; \ +}) +#define ipipe_tsc2us(t) \ +({ \ + unsigned long long delta = (t); \ + do_div(delta, ipipe_cpu_freq() / 1000000 + 1); \ + (unsigned long)delta; \ +}) + +#define ipipe_handle_irq_cond(irq) \ + __ipipe_handle_irq(irq, (struct pt_regs *)1) + +/* Private interface -- Internal use only */ + +#define __ipipe_check_platform() do { } while(0) + +#define __ipipe_init_platform() do { } while(0) + +#define __ipipe_enable_irq(irq) irq_desc[irq].chip->enable(irq) + +#define __ipipe_disable_irq(irq) irq_desc[irq].chip->disable(irq) + +#define __ipipe_hook_critical_ipi(ipd) do { } while(0) + +void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq); + +#ifndef __IPIPE_FEATURE_PIC_MUTE +#define __ipipe_disable_irqdesc(ipd, irq) do { } while (0) +#else /* __IPIPE_FEATURE_PIC_MUTE */ + +typedef unsigned long +__ipipe_irqbits_t[(NR_IRQS + BITS_PER_LONG - 1) / BITS_PER_LONG]; +extern __ipipe_irqbits_t __ipipe_irqbits; + +void __ipipe_disable_irqdesc(struct ipipe_domain *ipd, unsigned irq); + +void ipipe_mute_pic(void); + +void ipipe_unmute_pic(void); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +void __ipipe_enable_pipeline(void); + +void __ipipe_do_critical_sync(unsigned irq, + void *cookie); + +DECLARE_PER_CPU(struct pt_regs, __ipipe_tick_regs); + +int __ipipe_handle_irq(int irq, + struct pt_regs *regs); + +#define ipipe_update_tick_evtdev(evtdev) do { } while (0) + +#define __ipipe_tick_irq __ipipe_mach_timerint + +static inline unsigned long __ipipe_ffnz(unsigned long ul) +{ + return ffs(ul) - 1; +} + +/* When running handlers, enable hw interrupts for all domains but the + * one heading the pipeline, so that IRQs can never be significantly + * deferred for the latter. */ +#define __ipipe_run_isr(ipd, irq) \ +do { \ + if (!__ipipe_pipeline_head_p(ipd)) \ + local_irq_enable_hw(); \ + if (ipd == ipipe_root_domain) { \ + if (likely(!ipipe_virtual_irq_p(irq))) \ + ((void (*)(unsigned, struct pt_regs *)) \ + ipd->irqs[irq].handler) (irq, \ + &__raw_get_cpu_var(__ipipe_tick_regs)); \ + else { \ + irq_enter(); \ + ipd->irqs[irq].handler(irq,ipd->irqs[irq].cookie); \ + irq_exit(); \ + } \ + } else { \ + __clear_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \ + ipd->irqs[irq].handler(irq,ipd->irqs[irq].cookie); \ + __set_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \ + } \ + local_irq_disable_hw(); \ +} while(0) + +#define __ipipe_syscall_watched_p(p, sc) \ + (((p)->flags & PF_EVNOTIFY) || (unsigned long)sc >= __ARM_NR_BASE + 64) + +#define __ipipe_root_tick_p(regs) (!raw_irqs_disabled_flags(regs->ARM_cpsr)) + +#else /* !CONFIG_IPIPE */ + +#define task_hijacked(p) 0 + +#define ipipe_update_tick_evtdev(evtdev) do { } while (0) + +#define smp_processor_id_hw() smp_processor_id() + +#define ipipe_handle_irq_cond(irq) \ + generic_handle_irq(irq) + +#define ipipe_mm_switch_protect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#define ipipe_mm_switch_unprotect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__ARM_IPIPE_H */ diff --git a/arch/arm/include/asm/ipipe_base.h b/arch/arm/include/asm/ipipe_base.h new file mode 100644 index 0000000..20e97a4 --- /dev/null +++ b/arch/arm/include/asm/ipipe_base.h @@ -0,0 +1,108 @@ +/* -*- linux-c -*- + * arch/arm/include/asm/ipipe_base.h + * + * Copyright (C) 2007 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __ARM_IPIPE_BASE_H +#define __ARM_IPIPE_BASE_H + +#include <linux/threads.h> +#include <asm/ptrace.h> +#include <asm/irq.h> + +#define IPIPE_NR_XIRQS NR_IRQS +#define IPIPE_IRQ_ISHIFT 5 /* 25 for 32bits arch. */ + +/* ARM traps */ +#define IPIPE_TRAP_ACCESS 0 /* Data or instruction access exception */ +#define IPIPE_TRAP_SECTION 1 /* Section fault */ +#define IPIPE_TRAP_DABT 2 /* Generic data abort */ +#define IPIPE_TRAP_UNKNOWN 3 /* Unknown exception */ +#define IPIPE_TRAP_BREAK 4 /* Instruction breakpoint */ +#define IPIPE_TRAP_FPU 5 /* Floating point exception */ +#define IPIPE_TRAP_VFP 6 /* VFP floating point exception */ +#define IPIPE_TRAP_UNDEFINSTR 7 /* Undefined instruction */ +#define IPIPE_TRAP_ALIGNMENT 8 /* Unaligned access exception */ +#define IPIPE_NR_FAULTS 9 + +/* Pseudo-vectors used for kernel events */ +#define IPIPE_FIRST_EVENT IPIPE_NR_FAULTS +#define IPIPE_EVENT_SYSCALL (IPIPE_FIRST_EVENT) +#define IPIPE_EVENT_SCHEDULE (IPIPE_FIRST_EVENT + 1) +#define IPIPE_EVENT_SIGWAKE (IPIPE_FIRST_EVENT + 2) +#define IPIPE_EVENT_SETSCHED (IPIPE_FIRST_EVENT + 3) +#define IPIPE_EVENT_INIT (IPIPE_FIRST_EVENT + 4) +#define IPIPE_EVENT_EXIT (IPIPE_FIRST_EVENT + 5) +#define IPIPE_EVENT_CLEANUP (IPIPE_FIRST_EVENT + 6) +#define IPIPE_LAST_EVENT IPIPE_EVENT_CLEANUP +#define IPIPE_NR_EVENTS (IPIPE_LAST_EVENT + 1) + +#ifndef __ASSEMBLY__ + +#include <asm/irqflags.h> + +#ifdef CONFIG_SMP +#error "SMP not implemented." +#define __ipipe_root_status ipipe_root_cpudom_var(status) + +#else /* !CONFIG_SMP */ + +#ifdef CONFIG_VFP +#define __IPIPE_FEATURE_VFP_SAFE 1 +#endif + +#if __GNUC__ >= 4 +/* Alias to ipipe_root_cpudom_var(status) */ +extern unsigned long __ipipe_root_status; +#else +extern unsigned long *const __ipipe_root_status_addr; +#define __ipipe_root_status (*__ipipe_root_status_addr) +#endif + +static inline void __ipipe_stall_root(void) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_root_status |= 1; + local_irq_restore_hw(flags); +} + +static inline unsigned __ipipe_test_root(void) +{ + return __ipipe_root_status & 1; +} + +static inline unsigned __ipipe_test_and_stall_root(void) +{ + unsigned long flags, res; + + local_irq_save_hw(flags); + res = __ipipe_root_status; + __ipipe_root_status = res | 1; + local_irq_restore_hw(flags); + + return res & 1; +} + +#endif /* CONFIG_SMP */ + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ARM_IPIPE_BASE_H */ diff --git a/arch/arm/include/asm/irqflags.h b/arch/arm/include/asm/irqflags.h index 6d09974..0aa4056 100644 --- a/arch/arm/include/asm/irqflags.h +++ b/arch/arm/include/asm/irqflags.h @@ -10,30 +10,30 @@ */ #if __LINUX_ARM_ARCH__ >= 6 -#define raw_local_irq_save(x) \ +#define local_irq_save_hw_notrace(x) \ ({ \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_save\n" \ + "mrs %0, cpsr @ local_irq_save_hw\n" \ "cpsid i" \ : "=r" (x) : : "memory", "cc"); \ }) -#define raw_local_irq_enable() __asm__("cpsie i @ __sti" : : : "memory", "cc") -#define raw_local_irq_disable() __asm__("cpsid i @ __cli" : : : "memory", "cc") -#define local_fiq_enable() __asm__("cpsie f @ __stf" : : : "memory", "cc") -#define local_fiq_disable() __asm__("cpsid f @ __clf" : : : "memory", "cc") +#define local_irq_enable_hw_notrace() __asm__("cpsie i @ __sti" : : : "memory", "cc") +#define local_irq_disable_hw_notrace() __asm__("cpsid i @ __cli" : : : "memory", "cc") +#define local_fiq_enable_hw_notrace() __asm__("cpsie f @ __stf" : : : "memory", "cc") +#define local_fiq_disable_hw_notrace() __asm__("cpsid f @ __clf" : : : "memory", "cc") #else /* * Save the current interrupt enable state & disable IRQs */ -#define raw_local_irq_save(x) \ +#define local_irq_save_hw_notrace(x) \ ({ \ unsigned long temp; \ (void) (&temp == &x); \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_save\n" \ + "mrs %0, cpsr @ local_irq_save_hw\n" \ " orr %1, %0, #128\n" \ " msr cpsr_c, %1" \ : "=r" (x), "=r" (temp) \ @@ -44,11 +44,11 @@ /* * Enable IRQs */ -#define raw_local_irq_enable() \ +#define local_irq_enable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_enable\n" \ + "mrs %0, cpsr @ local_irq_enable_hw\n" \ " bic %0, %0, #128\n" \ " msr cpsr_c, %0" \ : "=r" (temp) \ @@ -59,11 +59,11 @@ /* * Disable IRQs */ -#define raw_local_irq_disable() \ +#define local_irq_disable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_disable\n" \ + "mrs %0, cpsr @ local_irq_disable_hw\n" \ " orr %0, %0, #128\n" \ " msr cpsr_c, %0" \ : "=r" (temp) \ @@ -74,7 +74,7 @@ /* * Enable FIQs */ -#define local_fiq_enable() \ +#define local_fiq_enable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ @@ -89,7 +89,7 @@ /* * Disable FIQs */ -#define local_fiq_disable() \ +#define local_fiq_disable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ @@ -106,19 +106,19 @@ /* * Save the current interrupt enable state. */ -#define raw_local_save_flags(x) \ +#define local_save_flags_hw(x) \ ({ \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_save_flags" \ + "mrs %0, cpsr @ local_save_flags_hw" \ : "=r" (x) : : "memory", "cc"); \ }) /* * restore saved IRQ & FIQ state */ -#define raw_local_irq_restore(x) \ +#define local_irq_restore_hw_notrace(x) \ __asm__ __volatile__( \ - "msr cpsr_c, %0 @ local_irq_restore\n" \ + "msr cpsr_c, %0 @ local_irq_restore_hw\n" \ : \ : "r" (x) \ : "memory", "cc") @@ -128,5 +128,99 @@ (int)((flags) & PSR_I_BIT); \ }) +#define irqs_disabled_hw() \ +({ \ + unsigned long flags; \ + local_save_flags_hw(flags); \ + raw_irqs_disabled_flags(flags); \ +}) + +static inline unsigned long raw_mangle_irq_bits(int virt, unsigned long real) +{ + /* Merge virtual and real interrupt mask bits into a single + 32bit word. */ + return (real & ~(1L << 8)) | ((virt != 0) << 8); +} + +static inline int raw_demangle_irq_bits(unsigned long *x) +{ + int virt = (*x & (1 << 8)) != 0; + *x &= ~(1L << 8); + return virt; +} + +#ifdef CONFIG_IPIPE + +void __ipipe_unstall_root(void); +void __ipipe_restore_root(unsigned long flags); + +/* PSR_I_BIT is bit no. 7 and is set if interrupts are _disabled_ */ +#define raw_local_irq_save(flags) ((flags) = __ipipe_test_and_stall_root() << 7) +#define raw_local_irq_enable() __ipipe_unstall_root() +#define raw_local_irq_disable() __ipipe_stall_root() +#define local_fiq_enable() __ipipe_unstall_root() +#define local_fiq_disable() __ipipe_stall_root() +#define raw_local_save_flags(flags) ((flags) = __ipipe_test_root() << 7) +#define raw_local_irq_restore(flags) __ipipe_restore_root(flags & (1 << 7)) + +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF + +#include <linux/ipipe_trace.h> + +#define local_irq_disable_hw() do { \ + if (!irqs_disabled_hw()) { \ + local_irq_disable_hw_notrace(); \ + ipipe_trace_begin(0x80000000); \ + } \ +} while (0) +#define local_irq_enable_hw() do { \ + if (irqs_disabled_hw()) { \ + ipipe_trace_end(0x80000000); \ + local_irq_enable_hw_notrace(); \ + } \ +} while (0) +#define local_irq_save_hw(x) do { \ + local_save_flags_hw(x); \ + if (!raw_irqs_disabled_flags(x)) { \ + local_irq_disable_hw_notrace(); \ + ipipe_trace_begin(0x80000001); \ + } \ +} while (0) +#define local_irq_restore_hw(x) do { \ + if (!raw_irqs_disabled_flags(x)) \ + ipipe_trace_end(0x80000001); \ + local_irq_restore_hw_notrace(x); \ +} while (0) + +#else /* !CONFIG_IPIPE_TRACE_IRQSOFF */ + +#define local_irq_save_hw(flags) local_irq_save_hw_notrace(flags) +#define local_irq_enable_hw() local_irq_enable_hw_notrace() +#define local_irq_disable_hw() local_irq_disable_hw_notrace() +#define local_fiq_enable_hw() local_fiq_enable_hw_notrace() +#define local_fiq_disable_hw() local_fiq_disable_hw_notrace() +#define local_irq_restore_hw(flags) local_irq_restore_hw_notrace(flags) + +#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */ + +#else /* !CONFIG_IPIPE */ + +#define raw_local_irq_save(flags) local_irq_save_hw_notrace(flags) +#define raw_local_irq_enable() local_irq_enable_hw_notrace() +#define raw_local_irq_disable() local_irq_disable_hw_notrace() +#define local_fiq_enable() local_fiq_enable_hw_notrace() +#define local_fiq_disable() local_fiq_disable_hw_notrace() +#define raw_local_save_flags(flags) local_save_flags_hw(flags) +#define raw_local_irq_restore(flags) local_irq_restore_hw_notrace(flags) + +#define local_irq_save_hw(flags) local_irq_save_hw_notrace(flags) +#define local_irq_enable_hw() local_irq_enable_hw_notrace() +#define local_irq_disable_hw() local_irq_disable_hw_notrace() +#define local_fiq_enable_hw() local_fiq_enable_hw_notrace() +#define local_fiq_disable_hw() local_fiq_disable_hw_notrace() +#define local_irq_restore_hw(flags) local_irq_restore_hw_notrace(flags) + +#endif /* CONFIG_IPIPE */ + #endif #endif diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h index 5421d82..578d189 100644 --- a/arch/arm/include/asm/memory.h +++ b/arch/arm/include/asm/memory.h @@ -33,7 +33,12 @@ */ #define PAGE_OFFSET UL(CONFIG_PAGE_OFFSET) #define TASK_SIZE (UL(CONFIG_PAGE_OFFSET) - UL(0x01000000)) +#ifndef CONFIG_ARM_FCSE #define TASK_UNMAPPED_BASE (UL(CONFIG_PAGE_OFFSET) / 3) +#else /* CONFIG_ARM_FCSE */ +#define TASK_UNMAPPED_BASE UL(0x00800000) +#define FCSE_TASK_SIZE UL(0x02000000) +#endif /* CONFIG_ARM_FCSE */ /* * The maximum size of a 26-bit user space task. diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index b561584..4087d0b 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -7,6 +7,17 @@ typedef struct { #ifdef CONFIG_CPU_HAS_ASID unsigned int id; #endif +#ifdef CONFIG_ARM_FCSE + struct { + unsigned long pid; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned shared_dirty_pages; + unsigned active : 1; + unsigned large : 1; + unsigned high_pages; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + } fcse; +#endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; } mm_context_t; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index de6cefb..edb8922 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -19,6 +19,7 @@ #include <asm/cachetype.h> #include <asm/proc-fns.h> #include <asm-generic/mm_hooks.h> +#include <asm/fcse.h> void __check_kvm_seq(struct mm_struct *mm); @@ -68,11 +69,50 @@ static inline void check_context(struct mm_struct *mm) #endif } -#define init_new_context(tsk,mm) 0 +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE + int fcse_pid; + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (!mm->context.fcse.large) { + fcse_pid = fcse_pid_alloc(mm); + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; + } else { + /* We are normally forking a process vith a virtual address + space larger than 32 MB, so its pid should be 0. */ + FCSE_BUG_ON(mm->context.fcse.pid); + fcse_pid_reference(0); + } + /* If we are forking, set_pte_at will restore the correct high pages + count, and shared writable pages are write-protected again. */ + mm->context.fcse.shared_dirty_pages = 0; + mm->context.fcse.high_pages = 0; + mm->context.fcse.active = 0; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + fcse_pid = fcse_pid_alloc(mm); + if (fcse_pid < 0) + return fcse_pid; + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#endif /* CONFIG_ARM_FCSE */ + + return 0; +} #endif -#define destroy_context(mm) do { } while(0) +static inline void destroy_context(struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + FCSE_BUG_ON(mm->context.fcse.shared_dirty_pages); + FCSE_BUG_ON(mm->context.fcse.high_pages); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + fcse_pid_free(mm); +#endif /* CONFIG_ARM_FCSE */ +} /* * This is called when "tsk" is about to enter lazy TLB mode. @@ -95,11 +135,11 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) * actually changed. */ static inline void -switch_mm(struct mm_struct *prev, struct mm_struct *next, - struct task_struct *tsk) +__switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *tsk) { #ifdef CONFIG_MMU - unsigned int cpu = smp_processor_id(); + unsigned int cpu = smp_processor_id_hw(); #ifdef CONFIG_SMP /* check for possible thread migration */ @@ -109,14 +149,64 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, #endif if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) { check_context(next); - cpu_switch_mm(next->pgd, next); - if (cache_is_vivt()) +#if defined(CONFIG_IPIPE) + if (ipipe_root_domain_p) { + /* mark mm state as undefined. */ + per_cpu(ipipe_active_mm, cpu) = NULL; + barrier(); + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(prev, next)); + barrier(); + per_cpu(ipipe_active_mm, cpu) = next; + while (test_and_clear_thread_flag(TIF_MMSWITCH_INT)) { + /* mark mm state as undefined. */ + per_cpu(ipipe_active_mm, cpu) = NULL; + barrier(); + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(NULL, next)); + barrier(); + per_cpu(ipipe_active_mm, cpu) = next; + } + } else +#endif /* CONFIG_IPIPE */ + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(prev, next)); +#if defined(CONFIG_IPIPE) && defined(CONFIG_ARM_FCSE) + if (tsk) + set_tsk_thread_flag(tsk, TIF_SWITCHED); +#endif /* CONFIG_IPIPE && CONFIG_ARM_FCSE */ + if (cache_is_vivt() && prev) cpumask_clear_cpu(cpu, mm_cpumask(prev)); - } + } else + fcse_mark_dirty(next); #endif } +static inline void +switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *tsk) +{ +#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + unsigned long flags; + local_irq_save_hw(flags); +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + __switch_mm(prev, next, tsk); +#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + local_irq_restore_hw(flags); +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ +} + #define deactivate_mm(tsk,mm) do { } while (0) + +#ifndef CONFIG_ARM_FCSE_BEST_EFFORT #define activate_mm(prev,next) switch_mm(prev, next, NULL) +#else +#define activate_mm(prev,next) \ + ({ \ + switch_mm(prev, next, NULL); \ + next->context.fcse.active = 1; \ + FCSE_BUG_ON(current->mm == next && !fcse_mm_in_cache(next)); \ + }) +#endif #endif diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 1139768..3d71ffb 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -112,6 +112,8 @@ #define LIBRARY_TEXT_START 0x0c000000 #ifndef __ASSEMBLY__ +#include <asm/fcse.h> + extern void __pte_error(const char *file, int line, unsigned long val); extern void __pmd_error(const char *file, int line, unsigned long val); extern void __pgd_error(const char *file, int line, unsigned long val); @@ -248,6 +250,40 @@ extern pgprot_t pgprot_kernel; #define __S111 __PAGE_SHARED_EXEC #ifndef __ASSEMBLY__ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#define fcse_account_page_removal(mm, addr, val) do { \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && ((_val) & L_PTE_SHARED)) \ + --_mm->context.fcse.shared_dirty_pages; \ + if (pte_present(_val) && _addr < TASK_SIZE) { \ + if (_addr >= FCSE_TASK_SIZE) \ + --_mm->context.fcse.high_pages; \ + } \ +} while (0) + +#define fcse_account_page_addition(mm, addr, val) ({ \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && (_val & L_PTE_SHARED)) { \ + if ((_val & (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + != (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + _val &= ~L_PTE_SHARED; \ + else \ + ++_mm->context.fcse.shared_dirty_pages; \ + } \ + if (pte_present(_val) \ + && _addr < TASK_SIZE && _addr >= FCSE_TASK_SIZE) \ + ++_mm->context.fcse.high_pages; \ + _val; \ +}) +#else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ +#define fcse_account_page_removal(mm, addr, val) do { } while (0) +#define fcse_account_page_addition(mm, addr, val) (val) +#endif /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ + /* * ZERO_PAGE is a global shared page that is always zero: used * for zero-mapped memory areas etc.. @@ -259,7 +295,10 @@ extern struct page *empty_zero_page; #define pfn_pte(pfn,prot) (__pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot))) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte_ext(ptep, __pte(0), 0) +#define pte_clear(mm,addr,ptep) do { \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + set_pte_ext(ptep, __pte(0), 0); \ +} while (0) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) #define pte_offset_kernel(dir,addr) (pmd_page_vaddr(*(dir)) + __pte_index(addr)) @@ -278,9 +317,13 @@ extern struct page *empty_zero_page; #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext) -#define set_pte_at(mm,addr,ptep,pteval) do { \ - set_pte_ext(ptep, pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ - } while (0) +#define set_pte_at(mm,addr,ptep,pteval) do { \ + unsigned long _pteval = (pteval); \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + pte_val(_pteval) = \ + fcse_account_page_addition(mm, addr, pte_val(_pteval)); \ + set_pte_ext(ptep, _pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ +} while (0) /* * The following only work if pte_present() is true. @@ -372,10 +415,14 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd) /* to find an entry in a page-table-directory */ #define pgd_index(addr) ((addr) >> PGDIR_SHIFT) -#define pgd_offset(mm, addr) ((mm)->pgd+pgd_index(addr)) +#define pgd_offset(mm, addr) \ + ({ \ + struct mm_struct *_mm = (mm); \ + (_mm->pgd + pgd_index(fcse_va_to_mva(_mm, (addr)))); \ + }) /* to find an entry in a kernel page-table-directory */ -#define pgd_offset_k(addr) pgd_offset(&init_mm, addr) +#define pgd_offset_k(addr) (init_mm.pgd + pgd_index(addr)) /* Find an entry in the second-level page table.. */ #define pmd_offset(dir, addr) ((pmd_t *)(dir)) diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h index 8fdae9b..0c280ca 100644 --- a/arch/arm/include/asm/proc-fns.h +++ b/arch/arm/include/asm/proc-fns.h @@ -261,12 +261,13 @@ #ifdef CONFIG_MMU -#define cpu_switch_mm(pgd,mm) cpu_do_switch_mm(virt_to_phys(pgd),mm) +#define cpu_switch_mm(pgd,mm,flush) \ + cpu_do_switch_mm(virt_to_phys(pgd), (mm), (flush)) #define cpu_get_pgd() \ ({ \ unsigned long pg; \ - __asm__("mrc p15, 0, %0, c2, c0, 0" \ + __asm__ __volatile__ ("mrc p15, 0, %0, c2, c0, 0" \ : "=r" (pg) : : "cc"); \ pg &= ~0x3fff; \ (pgd_t *)phys_to_virt(pg); \ diff --git a/arch/arm/include/asm/processor.h b/arch/arm/include/asm/processor.h index 6a89567..bc33df1 100644 --- a/arch/arm/include/asm/processor.h +++ b/arch/arm/include/asm/processor.h @@ -23,9 +23,14 @@ #include <asm/types.h> #ifdef __KERNEL__ +#ifndef CONFIG_ARM_FCSE #define STACK_TOP ((current->personality & ADDR_LIMIT_32BIT) ? \ TASK_SIZE : TASK_SIZE_26) #define STACK_TOP_MAX TASK_SIZE +#else /* CONFIG_ARM_FCSE */ +#define STACK_TOP FCSE_TASK_SIZE +#define STACK_TOP_MAX FCSE_TASK_SIZE +#endif /* CONFIG_ARM_FCSE */ #endif union debug_insn { diff --git a/arch/arm/include/asm/resource.h b/arch/arm/include/asm/resource.h index 734b581..5ba5042 100644 --- a/arch/arm/include/asm/resource.h +++ b/arch/arm/include/asm/resource.h @@ -1,6 +1,16 @@ #ifndef _ARM_RESOURCE_H #define _ARM_RESOURCE_H +/* + * When FCSE is enabled, reduce the default stack size to 1MB, and maximum + * to 16MB, the address space is only 32MB. + */ +#ifdef CONFIG_ARM_FCSE +#define _STK_LIM (1024*1024) + +#define _STK_LIM_MAX (16*1024*1024) +#endif /* CONFIG_ARM_FCSE */ + #include <asm-generic/resource.h> #endif diff --git a/arch/arm/include/asm/system.h b/arch/arm/include/asm/system.h index 058e7e9..6ba023d 100644 --- a/arch/arm/include/asm/system.h +++ b/arch/arm/include/asm/system.h @@ -218,10 +218,19 @@ static inline void set_copro_access(unsigned int val) */ extern struct task_struct *__switch_to(struct task_struct *, struct thread_info *, struct thread_info *); +#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH #define switch_to(prev,next,last) \ do { \ - last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ + local_irq_disable_hw_cond(); \ + last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ + local_irq_enable_hw_cond(); \ } while (0) +#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ +#define switch_to(prev,next,last) \ +do { \ + last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ +} while (0) +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ #if defined(CONFIG_CPU_SA1100) || defined(CONFIG_CPU_SA110) /* @@ -282,17 +291,17 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int size #error SMP is not supported on this platform #endif case 1: - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = *(volatile unsigned char *)ptr; *(volatile unsigned char *)ptr = x; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); break; case 4: - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = *(volatile unsigned long *)ptr; *(volatile unsigned long *)ptr = x; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); break; #else case 1: diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 2dfb7d7..e20256d 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -143,6 +143,12 @@ extern void vfp_sync_state(struct thread_info *thread); #define TIF_MEMDIE 18 #define TIF_FREEZE 19 #define TIF_RESTORE_SIGMASK 20 +#ifdef CONFIG_IPIPE +#define TIF_MMSWITCH_INT 21 +#ifdef CONFIG_ARM_FCSE +#define TIF_SWITCHED 22 +#endif /* CONFIG_ARM_FCSE */ +#endif /* CONFIG_IPIPE */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -152,6 +158,12 @@ extern void vfp_sync_state(struct thread_info *thread); #define _TIF_USING_IWMMXT (1 << TIF_USING_IWMMXT) #define _TIF_FREEZE (1 << TIF_FREEZE) #define _TIF_RESTORE_SIGMASK (1 << TIF_RESTORE_SIGMASK) +#ifdef CONFIG_IPIPE +#define _TIF_MMSWITCH_INT (1 << TIF_MMSWITCH_INT) +#ifdef CONFIG_ARM_FCSE +#define _TIF_SWITCHED (1 << TIF_SWITCHED) +#endif /* CONFIG_ARM_FCSE */ +#endif /* CONFIG_IPIPE */ /* * Change these and you break ASM code in entry-common.S diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h index c2f1605..70a81b1 100644 --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -210,6 +210,7 @@ #ifndef __ASSEMBLY__ #include <linux/sched.h> +#include <asm/fcse.h> struct cpu_tlb_fns { void (*flush_user_range)(unsigned long, unsigned long, struct vm_area_struct *); @@ -384,7 +385,8 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) const int zero = 0; const unsigned int __tlb_flag = __cpu_tlb_flags; - uaddr = (uaddr & PAGE_MASK) | ASID(vma->vm_mm); + uaddr = (fcse_va_to_mva(vma->vm_mm, uaddr) & PAGE_MASK) + | ASID(vma->vm_mm); if (tlb_flag(TLB_WB)) dsb(); @@ -505,7 +507,15 @@ static inline void clean_pmd_entry(pmd_t *pmd) /* * Convert calls to our calling convention. */ -#define local_flush_tlb_range(vma,start,end) __cpu_flush_user_tlb_range(start,end,vma) +#define local_flush_tlb_range(vma, start, end) \ + ({ \ + struct mm_struct *_mm = (vma)->vm_mm; \ + unsigned long _start, _end; \ + _start = fcse_va_to_mva(_mm, start); \ + _end = fcse_va_to_mva(_mm, end); \ + __cpu_flush_user_tlb_range(_start, _end, vma); \ + }) + #define local_flush_tlb_kernel_range(s,e) __cpu_flush_kern_tlb_range(s,e) #ifndef CONFIG_SMP diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile index dd00f74..ca557ee 100644 --- a/arch/arm/kernel/Makefile +++ b/arch/arm/kernel/Makefile @@ -20,7 +20,7 @@ obj-y := compat.o elf.o entry-armv.o entry-common.o irq.o \ obj-$(CONFIG_OC_ETM) += etm.o obj-$(CONFIG_ISA_DMA_API) += dma.o -obj-$(CONFIG_ARCH_ACORN) += ecard.o +obj-$(CONFIG_ARCH_ACORN) += ecard.o obj-$(CONFIG_FIQ) += fiq.o obj-$(CONFIG_MODULES) += armksyms.o module.o obj-$(CONFIG_ARTHUR) += arthur.o @@ -55,5 +55,7 @@ endif head-y := head$(MMUEXT).o obj-$(CONFIG_DEBUG_LL) += debug.o obj-$(CONFIG_EARLY_PRINTK) += early_printk.o +obj-$(CONFIG_ARM_FCSE) += fcse.o +obj-$(CONFIG_IPIPE) += ipipe.o extra-y := $(head-y) init_task.o vmlinux.lds diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 6c5cf36..f485f8c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -4,6 +4,7 @@ * Copyright (C) 1996,1997,1998 Russell King. * ARM700 fix by Matthew Godbolt (linux-user@willothewisp.demon.co.uk) * nommu support by Hyok S. Choi (hyok.choi@samsung.com) + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -29,14 +30,22 @@ * Interrupt handling. Preserves r7, r8, r9 */ .macro irq_handler +#ifdef CONFIG_IPIPE + mov r0, #2 +#endif get_irqnr_preamble r5, lr -1: get_irqnr_and_base r0, r6, r5, lr +1: get_irqnr_and_base r1, r6, r5, lr + movne r0, r1 movne r1, sp @ @ routine called with r0 = irq number, r1 = struct pt_regs * @ adrne lr, BSYM(1b) +#ifdef CONFIG_IPIPE + bne __ipipe_grab_irq +#else bne asm_do_IRQ +#endif #ifdef CONFIG_SMP /* @@ -45,18 +54,22 @@ * this macro assumes that irqstat (r6) and base (r5) are * preserved from get_irqnr_and_base above */ - test_for_ipi r0, r6, r5, lr + test_for_ipi r1, r6, r5, lr movne r0, sp adrne lr, BSYM(1b) bne do_IPI #ifdef CONFIG_LOCAL_TIMERS - test_for_ltirq r0, r6, r5, lr + test_for_ltirq r1, r6, r5, lr movne r0, sp adrne lr, BSYM(1b) bne do_local_timer #endif #endif +#ifdef CONFIG_IPIPE + cmp r0, #2 + bleq __ipipe_check_root_interruptible +#endif .endm @@ -78,6 +91,21 @@ mov r1, #\reason .endm + .macro fcse_dabt_mva_to_va +#ifdef CONFIG_ARM_FCSE + @ + @ If FCSE is enabled the data abort fault address must be + @ converted from MVA to VA. This must happen before the irqs are + @ enabled if PREEMPT is enabled, as context switches may + @ change the FCSE pid. + @ + mrc p15, 0, r2, c13, c0, 0 + eor r2, r2, r0 + tst r2, #0xfe000000 + moveq r0, r2 +#endif + .endm + __pabt_invalid: inv_entry BAD_PREFETCH b common_invalid @@ -193,6 +221,7 @@ __dabt_svc: #else bl CPU_DABORT_HANDLER #endif + fcse_dabt_mva_to_va @ @ set desired IRQ state, then call main handler @@ -220,14 +249,26 @@ __irq_svc: #ifdef CONFIG_PREEMPT get_thread_info tsk +#ifndef CONFIG_IPIPE ldr r8, [tsk, #TI_PREEMPT] @ get preempt count add r7, r8, #1 @ increment it str r7, [tsk, #TI_PREEMPT] #endif +#endif irq_handler +#ifdef CONFIG_IPIPE + cmp r0, #0 + ldreq r4, [sp, #S_PSR] @ irqs are already disabled + beq __ipipe_fast_svc_irq_exit +#endif + #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE str r8, [tsk, #TI_PREEMPT] @ restore preempt count +#else + ldr r8, [tsk, #TI_PREEMPT] +#endif ldr r0, [tsk, #TI_FLAGS] @ get flags teq r8, #0 @ if preempt count != 0 movne r0, #0 @ force flags to 0 @@ -239,6 +280,9 @@ __irq_svc: tst r4, #PSR_I_BIT bleq trace_hardirqs_on #endif +#ifdef CONFIG_IPIPE +__ipipe_fast_svc_irq_exit: +#endif svc_exit r4 @ return from exception UNWIND(.fnend ) ENDPROC(__irq_svc) @@ -248,12 +292,16 @@ ENDPROC(__irq_svc) #ifdef CONFIG_PREEMPT svc_preempt: mov r8, lr +#ifndef CONFIG_IPIPE 1: bl preempt_schedule_irq @ irq en/disable is done inside +#else /* CONFIG_IPIPE */ +1: bl __ipipe_preempt_schedule_irq @ irq en/disable is done inside +#endif /* CONFIG_IPIPE */ ldr r0, [tsk, #TI_FLAGS] @ get new tasks TI_FLAGS tst r0, #_TIF_NEED_RESCHED moveq pc, r8 @ go again b 1b -#endif +#endif /* CONFIG_PREEMPT */ .align 5 __und_svc: @@ -266,6 +314,17 @@ __und_svc: svc_entry #endif +#ifdef CONFIG_IPIPE + /* Beware, do_undefinstr in arch/arm/traps.c will not signal the trap + when in svc mode because of this hunk. */ + mov r4, r2 + mov r0, #7 @ r0 = IPIPE_TRAP_UNDEFINSTR + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + bne 1f + mov r2, r4 +#endif /* CONFIG_IPIPE */ @ @ call emulation code, which returns using r9 if it has emulated @ the instruction, or the more conventional lr if we are to treat @@ -433,6 +492,7 @@ __dabt_usr: #else bl CPU_DABORT_HANDLER #endif + fcse_dabt_mva_to_va @ @ IRQs on, then call the main handler @@ -451,13 +511,22 @@ __irq_usr: get_thread_info tsk #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE ldr r8, [tsk, #TI_PREEMPT] @ get preempt count add r7, r8, #1 @ increment it str r7, [tsk, #TI_PREEMPT] #endif +#endif irq_handler +#ifdef CONFIG_IPIPE + cmp r0, #0 + bne __ipipe_usr_irq_continue + slow_restore_user_regs @ Fast exit path over non-root domains +__ipipe_usr_irq_continue: +#endif #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE ldr r0, [tsk, #TI_PREEMPT] str r8, [tsk, #TI_PREEMPT] teq r0, r7 @@ -465,6 +534,7 @@ __irq_usr: THUMB( movne r0, #0 ) THUMB( strne r0, [r0] ) #endif +#endif #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_on #endif @@ -580,7 +650,7 @@ call_fpe: mov r7, #1 strb r7, [r10, #TI_USED_CP + 10] @ mark CP#10 as used strb r7, [r10, #TI_USED_CP + 11] @ mark CP#11 as used - b do_vfp @ let VFP handler handle this + b _do_vfp @ let VFP handler handle this 1: #endif tst r0, #0x08000000 @ only CDP/CPRT/LDC/STC have bit 27 @@ -626,8 +696,8 @@ call_fpe: movw_pc lr @ CP#8 movw_pc lr @ CP#9 #ifdef CONFIG_VFP - W(b) do_vfp @ CP#10 (VFP) - W(b) do_vfp @ CP#11 (VFP) + W(b) _do_vfp @ CP#10 (VFP) + W(b) _do_vfp @ CP#11 (VFP) #else movw_pc lr @ CP#10 (VFP) movw_pc lr @ CP#11 (VFP) @@ -662,11 +732,43 @@ call_fpe: #endif do_fpe: +#ifdef CONFIG_IPIPE + mov r4, r0 + mov r5, r2 + mov r6, lr + mov r0, #5 @ r0 = IPIPE_TRAP_FPU + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + movne pc, r9 + mov lr, r6 + mov r2, r5 + mov r0, r4 +#endif enable_irq ldr r4, .LCfp add r10, r10, #TI_FPSTATE @ r10 = workspace ldr pc, [r4] @ Call FP module USR entry point +#ifdef CONFIG_VFP +_do_vfp: +#ifdef CONFIG_IPIPE + mov r4, r0 + mov r5, r2 + mov r6, lr + mov r0, #6 @ r0 = IPIPE_TRAP_VFP + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + movne pc, r9 + mov lr, r6 + mov r2, r5 + mov r0, r4 +#endif + b do_vfp +#endif + + /* * The FP module is called with these registers set: * r0 = instruction @@ -715,6 +817,13 @@ __pabt_usr: ENTRY(ret_from_exception) UNWIND(.fnstart ) UNWIND(.cantunwind ) +#ifdef CONFIG_IPIPE + bl __ipipe_check_root + cmp r0, #0 + bne 1f + slow_restore_user_regs @ Fast exit path over non-root domains +1: +#endif /* CONFIG_IPIPE */ get_thread_info tsk mov why, #0 b ret_to_user @@ -752,7 +861,11 @@ ENTRY(__switch_to) add r4, r2, #TI_CPU_SAVE ldr r0, =thread_notify_head mov r1, #THREAD_NOTIFY_SWITCH +#ifndef CONFIG_IPIPE bl atomic_notifier_call_chain +#else /* !CONFIG_IPIPE */ + bl __ipipe_switch_to_notifier_call_chain +#endif /* !CONFIG_IPIPE */ THUMB( mov ip, r4 ) mov r0, r5 ARM( ldmia r4, {r4 - sl, fp, sp, pc} ) @ Load all regs saved previously @@ -795,6 +908,18 @@ ENDPROC(__switch_to) */ THUMB( .arm ) +#ifdef CONFIG_IPIPE +/* We use the same mechanism as Linux user helpers to store variables related to + TSC emulation, so that they can be used in user-space. */ + .align 5 + .globl __ipipe_tsc_area_start + +__ipipe_tsc_area_start: + .rep 12 + .word 0 + .endr +#endif /* CONFIG_IPIPE */ + .macro usr_ret, reg #ifdef CONFIG_ARM_THUMB bx \reg @@ -836,7 +961,7 @@ __kuser_helper_start: * * #define __kernel_dmb() \ * asm volatile ( "mov r0, #0xffff0fff; mov lr, pc; sub pc, r0, #95" \ - * : : : "r0", "lr","cc" ) + * : : : "r0", "lr","cc" ) */ __kuser_memory_barrier: @ 0xffff0fa0 @@ -1004,7 +1129,7 @@ kuser_cmpxchg_fixup: * #define __kernel_get_tls() \ * ({ register unsigned int __val asm("r0"); \ * asm( "mov r0, #0xffff0fff; mov lr, pc; sub pc, r0, #31" \ - * : "=r" (__val) : : "lr","cc" ); \ + * : "=r" (__val) : : "lr","cc" ); \ * __val; }) */ diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S index 2c1db77..57e3bf1 100644 --- a/arch/arm/kernel/entry-common.S +++ b/arch/arm/kernel/entry-common.S @@ -2,6 +2,7 @@ * linux/arch/arm/kernel/entry-common.S * * Copyright (C) 2000 Russell King + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -22,6 +23,11 @@ * possible here, and this includes saving r0 back into the SVC * stack. */ +#ifdef CONFIG_IPIPE +__ipipe_ret_fast_syscall: + ldr r0, [sp, #S_R0+S_OFF] @ returned r0 + /* fall through */ +#endif ret_fast_syscall: UNWIND(.fnstart ) UNWIND(.cantunwind ) @@ -30,10 +36,19 @@ ret_fast_syscall: tst r1, #_TIF_WORK_MASK bne fast_work_pending +fast_restore_user_regs: /* perform architecture specific actions before user return */ arch_ret_to_user r1, lr restore_user_regs fast = 1, offset = S_OFF + +#ifdef CONFIG_IPIPE +__ipipe_fast_exit_syscall: + ldr r0, [sp, #S_R0+S_OFF] @ returned r0 + disable_irq @ disable interrupts + b fast_restore_user_regs +#endif /* CONFIG_IPIPE */ + UNWIND(.fnend ) /* @@ -46,13 +61,16 @@ work_pending: bne work_resched tst r1, #_TIF_SIGPENDING|_TIF_NOTIFY_RESUME beq no_work_pending + enable_irq_cond mov r0, sp @ 'regs' mov r2, why @ 'syscall' bl do_notify_resume b ret_slow_syscall @ Check work again work_resched: + enable_irq_cond bl schedule + /* * "slow" syscall return path. "why" tells us if this was a real syscall. */ @@ -63,16 +81,14 @@ ret_slow_syscall: tst r1, #_TIF_WORK_MASK bne work_pending no_work_pending: - /* perform architecture specific actions before user return */ - arch_ret_to_user r1, lr - - restore_user_regs fast = 0, offset = 0 + slow_restore_user_regs ENDPROC(ret_to_user) /* * This is how we return from a fork. */ ENTRY(ret_from_fork) + enable_irq_cond bl schedule_tail get_thread_info tsk ldr r1, [tsk, #TI_FLAGS] @ check for syscall tracing @@ -169,8 +185,8 @@ ftrace_stub: *----------------------------------------------------------------------------- */ - /* If we're optimising for StrongARM the resulting code won't - run on an ARM7 and we can save a couple of instructions. + /* If we're optimising for StrongARM the resulting code won't + run on an ARM7 and we can save a couple of instructions. --pb */ #ifdef CONFIG_CPU_ARM710 #define A710(code...) code @@ -275,6 +291,18 @@ ENTRY(vector_swi) #endif stmdb sp!, {r4, r5} @ push fifth and sixth args +#ifdef CONFIG_IPIPE + stmfd sp!, {r0-r3, ip} + sub sp, sp, #4 + add r1, sp, #S_OFF+24 + mov r0, scno + bl __ipipe_syscall_root + cmp r0, #0 + add sp, sp, #4 + ldmfd sp!, {r0-r3, ip} + blt __ipipe_ret_fast_syscall + bgt __ipipe_fast_exit_syscall +#endif /* CONFIG_IPIPE */ tst ip, #_TIF_SYSCALL_TRACE @ are we tracing syscalls? bne __sys_trace @@ -286,7 +314,7 @@ ENTRY(vector_swi) 2: mov why, #0 @ no longer a real syscall cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE) eor r0, scno, #__NR_SYSCALL_BASE @ put OS number back - bcs arm_syscall + bcs arm_syscall b sys_ni_syscall @ not private func ENDPROC(vector_swi) @@ -322,6 +350,9 @@ __sys_trace_return: __cr_alignment: .word cr_alignment #endif +#ifdef CONFIG_IPIPE + .word __ipipe_syscall_root +#endif .ltorg /* @@ -475,3 +506,28 @@ ENTRY(sys_oabi_call_table) #endif + +#ifdef CONFIG_FRAME_POINTER + + .text + .align 0 + .type arm_return_addr %function + .global arm_return_addr + +arm_return_addr: + mov ip, r0 + mov r0, fp +3: + cmp r0, #0 + beq 1f @ frame list hit end, bail + cmp ip, #0 + beq 2f @ reached desired frame + ldr r0, [r0, #-12] @ else continue, get next fp + sub ip, ip, #1 + b 3b +2: + ldr r0, [r0, #-4] @ get target return address +1: + mov pc, lr + +#endif diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 7e9ed1e..f465901 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -22,7 +22,7 @@ @ #define S_OFF 8 -/* +/* * The SWI code relies on the fact that R0 is at the bottom of the stack * (due to slow/fast restore user regs). */ @@ -163,6 +163,12 @@ .endm #endif /* !CONFIG_THUMB2_KERNEL */ + .macro slow_restore_user_regs + /* perform architecture specific actions before user return */ + arch_ret_to_user r1, lr + restore_user_regs fast = 0, offset = 0 + .endm + /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6. diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c new file mode 100644 index 0000000..1b9550c --- /dev/null +++ b/arch/arm/kernel/fcse.c @@ -0,0 +1,465 @@ +#include <linux/bitops.h> +#include <linux/memory.h> +#include <linux/spinlock.h> +#include <linux/mm.h> +#include <linux/kernel_stat.h> +#include <linux/mman.h> +#include <linux/dcache.h> +#include <linux/fs.h> + +#include <asm/fcse.h> +#include <asm/cacheflush.h> +#include <asm/tlbflush.h> + +#define NR_PIDS (TASK_SIZE / FCSE_PID_TASK_SIZE) +#define PIDS_LONGS ((NR_PIDS + BITS_PER_LONG - 1) / BITS_PER_LONG) + +static IPIPE_DEFINE_SPINLOCK(fcse_lock); +static unsigned long fcse_pids_bits[PIDS_LONGS]; +unsigned long fcse_pids_cache_dirty[PIDS_LONGS]; +EXPORT_SYMBOL(fcse_pids_cache_dirty); + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static unsigned random_pid; +struct fcse_user fcse_pids_user[NR_PIDS]; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + +static inline void fcse_pid_reference_inner(unsigned pid) +{ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (++fcse_pids_user[pid].count == 1) +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __set_bit(pid, fcse_pids_bits); +} + +static inline void fcse_pid_dereference(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#ifndef CONFIG_IPIPE + /* FIXME: Why ? */ + FCSE_BUG_ON(fcse_mm_in_cache(mm)); +#endif /* CONFIG_IPIPE */ + if (--fcse_pids_user[fcse_pid].count == 0) + __clear_bit(fcse_pid, fcse_pids_bits); + + /* + * The following means we suppose that by the time this + * function is called, this mm is out of cache: + * - when the caller is destroy_context, exit_mmap is called + * by mmput before, which flushes the cache; + * - when the caller is fcse_relocate_mm_to_pid from + * fcse_switch_mm_inner, we only relocate when the mm is out + * of cache; + * - when the caller is fcse_relocate_mm_to_pid from + * fcse_relocate_mm_to_null_pid, we flush the cache in this + * function. + */ + if (fcse_pids_user[fcse_pid].mm == mm) { + fcse_pids_user[fcse_pid].mm = NULL; + __clear_bit(fcse_pid, fcse_pids_cache_dirty); + } +#else /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __clear_bit(fcse_pid, fcse_pids_bits); + __clear_bit(fcse_pid, fcse_pids_cache_dirty); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +} + +static inline unsigned find_free_pid(unsigned long bits[]) +{ + unsigned fcse_pid; + + fcse_pid = find_next_zero_bit(bits, NR_PIDS, 1); + if (fcse_pid == NR_PIDS) + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ + if (!test_bit(0, bits)) + fcse_pid = 0; + + return fcse_pid; +} + +void fcse_pid_free(struct mm_struct *mm) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_dereference(mm); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +int fcse_pid_alloc(struct mm_struct *mm) +{ + unsigned long flags; + unsigned fcse_pid; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid = find_free_pid(fcse_pids_bits); + if (fcse_pid == NR_PIDS) { + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if(++random_pid == NR_PIDS) + random_pid = 0; + fcse_pid = random_pid; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + spin_unlock_irqrestore(&fcse_lock, flags); +#ifdef CONFIG_ARM_FCSE_MESSAGES + printk(KERN_WARNING "FCSE: system would exceed the %lu processes limit.\n", + NR_PIDS); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + return -EAGAIN; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ + } + fcse_pid_reference_inner(fcse_pid); +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + FCSE_BUG_ON(fcse_mm_in_cache(mm)); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + spin_unlock_irqrestore(&fcse_lock, flags); + + return fcse_pid; +} + +static inline void fcse_clear_dirty_all(void) +{ + switch(ARRAY_SIZE(fcse_pids_cache_dirty)) { + case 4: + fcse_pids_cache_dirty[3] = 0UL; + case 3: + fcse_pids_cache_dirty[2] = 0UL; + case 2: + fcse_pids_cache_dirty[1] = 0UL; + case 1: + fcse_pids_cache_dirty[0] = 0UL; + } +} + +unsigned fcse_flush_all_start(struct mm_struct *mm) +{ + if (!cache_is_vivt()) + return 0; + +#ifndef CONFIG_ARM_FCSE_PREEMPT_FLUSH + preempt_disable(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + +#if defined(CONFIG_IPIPE) + clear_ti_thread_flag(current_thread_info(), TIF_SWITCHED); +#elif defined(CONFIG_ARM_FCSE_PREEMPT_FLUSH) + return nr_context_switches(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + + return 0; +} + +noinline int +fcse_flush_all_done(struct mm_struct *mm, unsigned seq, unsigned dirty) +{ + unsigned long flags; + int done = 1; + + if (!cache_is_vivt()) + return 1; + + spin_lock_irqsave(&fcse_lock, flags); +#if defined(CONFIG_IPIPE) + if (test_ti_thread_flag(current_thread_info(), TIF_SWITCHED)) { + if (fcse_mm_in_cache(mm)) { + done = 0; + goto ret; + } + } else +#elif defined(CONFIG_ARM_FCSE_PREEMPT_FLUSH) + if (seq != nr_context_switches()) { + if (fcse_mm_in_cache(mm)) { + done = 0; + goto ret; + } + } else +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + fcse_clear_dirty_all(); + + if (dirty && current->mm != &init_mm && current->mm) { + unsigned fcse_pid = + current->mm->context.fcse.pid >> FCSE_PID_SHIFT; + __set_bit(fcse_pid, fcse_pids_cache_dirty); + } + + ret: + spin_unlock_irqrestore(&fcse_lock, flags); +#ifndef CONFIG_ARM_FCSE_PREEMPT_FLUSH + preempt_enable(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + return done; +} + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +/* Called with preemption disabled, mm->mmap_sem being held for writing. */ +static noinline int fcse_relocate_mm_to_pid(struct mm_struct *mm, int fcse_pid) +{ + const unsigned len = pgd_index(FCSE_TASK_SIZE) * sizeof(pgd_t); + unsigned long flags; + pgd_t *from, *to; + + spin_lock_irqsave(&fcse_lock, flags); +#if defined(CONFIG_ARM_FCSE_DYNPID) + /* pid == -1 means find a free pid. */ + if (fcse_pid == -1) { + fcse_pid = find_free_pid(fcse_pids_bits); + if (fcse_pid == NR_PIDS) { + fcse_pid = find_free_pid(fcse_pids_cache_dirty); + if (unlikely(fcse_pid == NR_PIDS)) { + spin_unlock_irqrestore(&fcse_lock, flags); + return -ENOENT; + } + } + } +#endif /* CONFIG_ARM_FCSE_DYNPID */ + fcse_pid_dereference(mm); + fcse_pid_reference_inner(fcse_pid); + fcse_pids_user[fcse_pid].mm = mm; + __set_bit(fcse_pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); + + from = pgd_offset(mm, 0); + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; + to = pgd_offset(mm, 0); + + memcpy(to, from, len); + memset(from, '\0', len); + barrier(); + clean_dcache_area(from, len); + clean_dcache_area(to, len); + + return fcse_pid; +} + +int fcse_switch_mm_inner(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned fcse_pid = next->context.fcse.pid >> FCSE_PID_SHIFT; + unsigned flush_needed, reused_pid = 0; + unsigned long flags; + + if (unlikely(next == &init_mm)) { + spin_lock_irqsave(&fcse_lock, flags); + goto is_flush_needed; + } + +#ifdef CONFIG_ARM_FCSE_DYNPID + /* + * If the next mm's pid is currently in use, and not by that + * mm, try and find a new, free, pid. + */ + if (unlikely(fcse_pids_user[fcse_pid].mm != next) + && test_bit(fcse_pid, fcse_pids_cache_dirty) + && !rwsem_is_locked(&next->mmap_sem) + && fcse_pids_user[fcse_pid].mm + && !next->context.fcse.large + && !next->core_state) { + int new_fcse_pid = fcse_relocate_mm_to_pid(next, -1); + if (new_fcse_pid >= 0) + fcse_pid = new_fcse_pid; + } +#endif /* CONFIG_ARM_FCSE_DYNPID */ + + spin_lock_irqsave(&fcse_lock, flags); + if (fcse_pids_user[fcse_pid].mm != next) { + if (fcse_pids_user[fcse_pid].mm) + reused_pid = test_bit(fcse_pid, fcse_pids_cache_dirty); + fcse_pids_user[fcse_pid].mm = next; + } + + is_flush_needed: + flush_needed = reused_pid + || next->context.fcse.high_pages + || !prev + || prev->context.fcse.shared_dirty_pages + || prev->context.fcse.high_pages; + + fcse_pid_set(fcse_pid << FCSE_PID_SHIFT); + if (flush_needed) + fcse_clear_dirty_all(); + if (next != &init_mm) + __set_bit(fcse_pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); + + return flush_needed; +} +EXPORT_SYMBOL_GPL(fcse_switch_mm_inner); + +void fcse_pid_reference(unsigned fcse_pid) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_reference_inner(fcse_pid); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +/* Called with mm->mmap_sem write-locked. */ +static noinline void fcse_relocate_mm_to_null_pid(struct mm_struct *mm) +{ + unsigned long seq; + + if (!cache_is_vivt()) + return; + + preempt_disable(); + if (fcse_mm_in_cache(mm)) + do { + preempt_enable(); + + seq = fcse_flush_all_start(mm); + flush_cache_all(); + + preempt_disable(); + } while (!fcse_flush_all_done(mm, seq, 0)); + + fcse_relocate_mm_to_pid(mm, 0); + barrier(); + flush_tlb_mm(mm); + fcse_pid_set(0); + + preempt_enable(); +} +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + +unsigned long +fcse_check_mmap_inner(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned long stack_reserved = + current->signal->rlim[RLIMIT_STACK].rlim_cur; + unsigned long stack_base = PAGE_ALIGN(mm->start_stack) - stack_reserved; + + /* We enfore the RLIMIT_STACK stack size, and here, the return + address would fall in that reserved stack area */ + if ((unsigned long)(addr + len - stack_base) < stack_reserved) { + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED) + && !mm->context.fcse.large) + return TASK_UNMAPPED_BASE; + + /* Forcibly restart from above the stack */ + if (!(fl & MAP_FIXED)) + return PAGE_ALIGN(mm->start_stack); + + /* If MAP_FIXED is set, we encroach upon the reserved + stack area. No choice. */ + } + + /* Address above 32MB */ + if (addr + len > FCSE_TASK_SIZE && !mm->context.fcse.high_pages) { + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED)) + return TASK_UNMAPPED_BASE; + + if (!mm->context.fcse.large) { + /* Ok, the process is going to be larger than 32MB */ +#ifdef CONFIG_ARM_FCSE_MSSAGES + printk(KERN_INFO "FCSE: process %u(%s) VM exceeds 32MB.\n", + current->pid, current->comm); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + mm->context.fcse.large = 1; + } + if (mm->context.fcse.pid) + fcse_relocate_mm_to_null_pid(mm); + } + + return addr; + +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + /* Address above 32MB */ + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED)) + return TASK_UNMAPPED_BASE; + + /* Fail, no 32MB processes in guaranteed mode. */ +#ifdef CONFIG_ARM_FCSE_MESSAGES + printk(KERN_WARNING "FCSE: process %u(%s) VM would exceed the 32MB limit.\n", + current->pid, current->comm); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + return -ENOMEM; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +} + +#ifdef CONFIG_ARM_FCSE_MESSAGES +#define addr_in_vma(vma, addr) \ + ({ \ + struct vm_area_struct *_vma = (vma); \ + ((unsigned long)((addr) - _vma->vm_start) \ + < (unsigned long)((_vma->vm_end - _vma->vm_start))); \ + }) + +#ifdef CONFIG_DEBUG_USER +static noinline void +dump_vmas(struct mm_struct *mm, unsigned long addr, struct pt_regs *regs) +{ + struct vm_area_struct *vma; + char path[128]; + + printk("mappings:\n"); + down_read(&mm->mmap_sem); + for(vma = mm->mmap; vma; vma = vma->vm_next) { + struct file *file = vma->vm_file; + int flags = vma->vm_flags; + const char *name; + + printk("0x%08lx-0x%08lx %c%c%c%c 0x%08llx ", + vma->vm_start, + vma->vm_end, + flags & VM_READ ? 'r' : '-', + flags & VM_WRITE ? 'w' : '-', + flags & VM_EXEC ? 'x' : '-', + flags & VM_MAYSHARE ? 's' : 'p', + ((loff_t)vma->vm_pgoff) << PAGE_SHIFT); + + if (file) + name = d_path(&file->f_path, path, sizeof(path)); + else if ((name = arch_vma_name(vma))) + ; + else if (!vma->vm_mm) + name = "[vdso]"; + else if (vma->vm_start <= mm->start_brk + && vma->vm_end >= mm->brk) + name = "[heap]"; + else if (vma->vm_start <= mm->start_stack && + vma->vm_end >= mm->start_stack) + name = "[stack]"; + else + name = ""; + printk("%s", name); + if (addr_in_vma(vma, regs->ARM_pc)) + printk(" <- PC"); + if (addr_in_vma(vma, regs->ARM_sp)) + printk(" <- SP"); + if (addr_in_vma(vma, addr)) + printk("%s fault", + (addr_in_vma(vma, regs->ARM_pc) + || addr_in_vma(vma, regs->ARM_sp) + ? "," : " <-")); + printk("\n"); + } + up_read(&mm->mmap_sem); +} +#endif /* CONFIG_DEBUG_USER */ + +void fcse_notify_segv(struct mm_struct *mm, + unsigned long addr, struct pt_regs *regs) +{ +#if defined(CONFIG_DEBUG_USER) + if (user_debug & UDBG_SEGV) + dump_vmas(mm, addr, regs); +#endif /* CONFIG_DEBUG_USER */ + + down_read(&mm->mmap_sem); + if (find_vma(mm, addr) == find_vma(mm, regs->ARM_sp)) + printk(KERN_INFO "FCSE: process %u(%s) probably overflowed stack at 0x%08lx.\n", + current->pid, current->comm, regs->ARM_pc); + up_read(&mm->mmap_sem); +} +#endif /* CONFIG_ARM_FCSE_MESSAGES */ diff --git a/arch/arm/kernel/ipipe.c b/arch/arm/kernel/ipipe.c new file mode 100644 index 0000000..41b1013 --- /dev/null +++ b/arch/arm/kernel/ipipe.c @@ -0,0 +1,549 @@ +/* -*- linux-c -*- + * linux/arch/arm/kernel/ipipe.c + * + * Copyright (C) 2002-2005 Philippe Gerum. + * Copyright (C) 2004 Wolfgang Grandegger (Adeos/arm port over 2.4). + * Copyright (C) 2005 Heikki Lindholm (PowerPC 970 fixes). + * Copyright (C) 2005 Stelian Pop. + * Copyright (C) 2006-2008 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Architecture-dependent I-PIPE support for ARM. + */ + +#include <linux/kernel.h> +#include <linux/smp.h> +#include <linux/sched.h> +#include <linux/slab.h> +#include <linux/bitops.h> +#include <linux/interrupt.h> +#include <linux/module.h> +#include <linux/errno.h> +#include <linux/kallsyms.h> +#include <linux/kprobes.h> +#include <linux/ipipe_trace.h> +#include <asm/system.h> +#include <asm/atomic.h> +#include <asm/hardirq.h> +#include <asm/io.h> +#include <asm/irq.h> +#include <asm/unistd.h> +#include <asm/mach/irq.h> +#include <asm/mmu_context.h> + +/* Next tick date (timebase value). */ +DEFINE_PER_CPU(struct pt_regs, __ipipe_tick_regs); +DEFINE_PER_CPU(struct mm_struct *,ipipe_active_mm); +EXPORT_PER_CPU_SYMBOL(ipipe_active_mm); +#ifdef __IPIPE_FEATURE_PIC_MUTE +__ipipe_irqbits_t __ipipe_irqbits; +IPIPE_DEFINE_SPINLOCK(__ipipe_irqbits_lock); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +extern struct irq_desc irq_desc[]; +asmlinkage void asm_do_IRQ(unsigned int irq, struct pt_regs *regs); + +#ifdef CONFIG_SMP + +static cpumask_t __ipipe_cpu_sync_map; + +static cpumask_t __ipipe_cpu_lock_map; + +static IPIPE_DEFINE_SPINLOCK(__ipipe_cpu_barrier); + +static atomic_t __ipipe_critical_count = ATOMIC_INIT(0); + +static void (*__ipipe_cpu_sync) (void); + +/* Always called with hw interrupts off. */ + + +void __ipipe_do_critical_sync(unsigned irq) +{ + int cpu = ipipe_processor_id(); + + cpu_set(cpu, __ipipe_cpu_sync_map); + + /* + * Now we are in sync with the lock requestor running on another + * CPU. Enter a spinning wait until he releases the global + * lock. + */ + spin_lock(&__ipipe_cpu_barrier); + + /* Got it. Now get out. */ + + if (__ipipe_cpu_sync) + /* Call the sync routine if any. */ + __ipipe_cpu_sync(); + + spin_unlock(&__ipipe_cpu_barrier); + + cpu_clear(cpu, __ipipe_cpu_sync_map); +} + +#endif /* CONFIG_SMP */ + +/* + * ipipe_trigger_irq() -- Push the interrupt at front of the pipeline + * just like if it has been actually received from a hw source. Also + * works for virtual interrupts. + */ +int ipipe_trigger_irq(unsigned irq) +{ + unsigned long flags; + +/* FIXME */ +#ifdef CONFIG_IPIPE_DEBUG + if (irq >= IPIPE_NR_IRQS || + (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))) + return -EINVAL; +#else + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + if (ipipe_virtual_irq_p(irq)) { + if (!test_bit(irq - IPIPE_VIRQ_BASE, + &__ipipe_virtual_irq_map)) + return -EINVAL; + } else if (irq_to_desc(irq) == NULL) + return -EINVAL; +#endif + + local_irq_save_hw(flags); + __ipipe_handle_irq(irq, NULL); + local_irq_restore_hw(flags); + + return 1; +} + +int ipipe_get_sysinfo(struct ipipe_sysinfo *info) +{ + info->ncpus = num_online_cpus(); + info->cpufreq = ipipe_cpu_freq(); + info->archdep.tmirq = __ipipe_mach_timerint; + info->archdep.tmfreq = info->cpufreq; + __ipipe_mach_get_tscinfo(&info->archdep.tsc); + + return 0; +} + +static void __ipipe_ack_irq(unsigned irq, struct irq_desc *desc) +{ + desc->ipipe_ack(irq, desc); +} + +static void __ipipe_ack_timerirq(unsigned irq, struct irq_desc *desc) +{ + desc->ipipe_ack(irq, desc); + __ipipe_mach_acktimer(); + desc->ipipe_end(irq, desc); +} + +void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq) +{ + unsigned long flags; + irq_desc[irq].status &= ~IRQ_DISABLED; +#ifdef __IPIPE_FEATURE_PIC_MUTE + if (ipd == &ipipe_root) + return; + + spin_lock_irqsave(&__ipipe_irqbits_lock, flags); + __ipipe_irqbits[irq / BITS_PER_LONG] &= ~(1 << (irq % BITS_PER_LONG)); + spin_unlock_irqrestore(&__ipipe_irqbits_lock, flags); +#else + (void) flags; +#endif /* __IPIPE_FEATURE_PIC_MUTE */ +} +EXPORT_SYMBOL(__ipipe_enable_irqdesc); + +#ifdef __IPIPE_FEATURE_PIC_MUTE +void __ipipe_disable_irqdesc(struct ipipe_domain *ipd, unsigned irq) +{ + unsigned long flags; + + if (ipd == &ipipe_root) + return; + + spin_lock_irqsave(&__ipipe_irqbits_lock, flags); + __ipipe_irqbits[irq / BITS_PER_LONG] |= 1 << (irq % BITS_PER_LONG); + spin_unlock_irqrestore(&__ipipe_irqbits_lock, flags); +} +EXPORT_SYMBOL(__ipipe_disable_irqdesc); +EXPORT_SYMBOL(ipipe_mute_pic); +EXPORT_SYMBOL(ipipe_unmute_pic); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +/* + * __ipipe_enable_pipeline() -- We are running on the boot CPU, hw + * interrupts are off, and secondary CPUs are still lost in space. + */ +void __ipipe_enable_pipeline(void) +{ + unsigned long flags; + unsigned irq; + +/* We do not want "wfi" to be called in arm926ejs based processor, as + this causes the I-cache to be disabled when idle. */ +#ifdef CONFIG_CPU_ARM926T + disable_hlt(); +#endif + + flags = ipipe_critical_enter(NULL); + + /* First, virtualize all interrupts from the root domain. */ + + for (irq = 0; irq < NR_IRQS; irq++) + ipipe_virtualize_irq(ipipe_root_domain, + irq, + (ipipe_irq_handler_t)&asm_do_IRQ, NULL, + ((irq == __ipipe_mach_timerint) + ? &__ipipe_ack_timerirq + : &__ipipe_ack_irq), + IPIPE_HANDLE_MASK | IPIPE_PASS_MASK); + + ipipe_critical_exit(flags); +} + +/* + * ipipe_critical_enter() -- Grab the superlock excluding all CPUs + * but the current one from a critical section. This lock is used when + * we must enforce a global critical section for a single CPU in a + * possibly SMP system whichever context the CPUs are running. + */ +unsigned long ipipe_critical_enter(void (*syncfn) (void)) +{ + unsigned long flags; + + local_irq_save_hw(flags); + +#ifdef CONFIG_SMP + if (likely(num_online_cpus() > 1)) { /* We might be running a SMP-kernel on a UP box... */ + int cpu = ipipe_processor_id(); + cpumask_t lock_map; + + if (!cpu_test_and_set(cpu, __ipipe_cpu_lock_map)) { + while (cpu_test_and_set(BITS_PER_LONG - 1, + __ipipe_cpu_lock_map)) { + int n = 0; + do { + cpu_relax(); + } while (++n < cpu); + } + + spin_lock(&__ipipe_cpu_barrier); + + __ipipe_cpu_sync = syncfn; + + /* Send the sync IPI to all processors but the current one. */ + send_IPI_allbutself(IPIPE_CRITICAL_VECTOR); + + cpus_andnot(lock_map, cpu_online_map, + __ipipe_cpu_lock_map); + + while (!cpus_equal(__ipipe_cpu_sync_map, lock_map)) + cpu_relax(); + } + + atomic_inc(&__ipipe_critical_count); + } +#endif /* CONFIG_SMP */ + + return flags; +} + +/* ipipe_critical_exit() -- Release the superlock. */ + +void ipipe_critical_exit(unsigned long flags) +{ +#ifdef CONFIG_SMP + if (likely(num_online_cpus() > 1)) { /* We might be running a SMP-kernel on a UP box... */ + if (atomic_dec_and_test(&__ipipe_critical_count)) { + spin_unlock(&__ipipe_cpu_barrier); + + while (!cpus_empty(__ipipe_cpu_sync_map)) + cpu_relax(); + + cpu_clear(ipipe_processor_id(), __ipipe_cpu_lock_map); + cpu_clear(BITS_PER_LONG - 1, __ipipe_cpu_lock_map); + } + } +#endif /* CONFIG_SMP */ + + local_irq_restore_hw(flags); +} + +#ifdef CONFIG_PREEMPT + +asmlinkage void __sched __ipipe_preempt_schedule_irq(void) +{ + extern asmlinkage void __sched preempt_schedule_irq(void); + struct ipipe_percpu_domain_data *p; + unsigned long flags; + /* + * preempt_schedule_irq expects the root stage to be stalled. + */ +#ifdef CONFIG_IPIPE_DEBUG + BUG_ON(!irqs_disabled_hw()); +#endif + local_irq_save(flags); + local_irq_enable_hw(); + preempt_schedule_irq(); /* Ok, may reschedule now. */ + local_irq_disable_hw(); + + /* + * Flush any pending interrupt that may have been logged after + * preempt_schedule_irq() stalled the root stage before + * returning to us, and now. + */ + p = ipipe_root_cpudom_ptr(); + if (unlikely(__ipipe_ipending_p(p))) { + add_preempt_count(PREEMPT_ACTIVE); + clear_bit(IPIPE_STALL_FLAG, &p->status); + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + sub_preempt_count(PREEMPT_ACTIVE); + } + + __local_irq_restore_nosync(flags); +} + +#endif /* CONFIG_PREEMPT */ + +asmlinkage int __ipipe_check_root(void) +{ + return ipipe_root_domain_p; +} + +asmlinkage int __ipipe_check_root_interruptible(void) +{ + return ipipe_root_domain_p && !__ipipe_test_root(); +} + +__kprobes int +__ipipe_switch_to_notifier_call_chain(struct atomic_notifier_head *nh, + unsigned long val, void *v) +{ + unsigned long flags; + int rc; + + local_irq_save(flags); + rc = atomic_notifier_call_chain(nh, val, v); + __local_irq_restore_nosync(flags); + + return rc; +} + +asmlinkage int __ipipe_syscall_root(unsigned long scno, struct pt_regs *regs) +{ + struct ipipe_percpu_domain_data *p; + unsigned long flags, origr7; + + /* We use r7 to pass the syscall number to the other domains */ + origr7 = regs->ARM_r7; + regs->ARM_r7 = __NR_SYSCALL_BASE + scno; + /* + * This routine either returns: + * 0 -- if the syscall is to be passed to Linux; + * >0 -- if the syscall should not be passed to Linux, and no + * tail work should be performed; + * <0 -- if the syscall should not be passed to Linux but the + * tail work has to be performed (for handling signals etc). + */ + + WARN_ON_ONCE(irqs_disabled_hw()); + + if (!__ipipe_syscall_watched_p(current, regs->ARM_r7) || + !__ipipe_event_monitored_p(IPIPE_EVENT_SYSCALL)) + goto done; + + if (__ipipe_dispatch_event(IPIPE_EVENT_SYSCALL,regs) > 0){ + if (ipipe_root_domain_p && !in_atomic()) { + /* + * Sync pending VIRQs before _TIF_NEED_RESCHED + * is tested. + */ + local_irq_save_hw(flags); + p = ipipe_root_cpudom_ptr(); + if (__ipipe_ipending_p(p)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOVIRT); + local_irq_restore_hw(flags); + regs->ARM_r7 = origr7; + return -1; + } + regs->ARM_r7 = origr7; + return 1; + } +done: + regs->ARM_r7 = origr7; + return 0; +} + +/* + * __ipipe_handle_irq() -- IPIPE's generic IRQ handler. An optimistic + * interrupt protection log is maintained here for each domain. Hw + * interrupts are off on entry. + */ +int __ipipe_handle_irq(int irq, struct pt_regs *regs) +{ + struct ipipe_domain *this_domain, *next_domain; + struct list_head *head, *pos; + int m_ack; + + m_ack = (regs == NULL); + +#ifdef CONFIG_IPIPE_DEBUG + if (irq >= IPIPE_NR_IRQS) { + printk(KERN_ERR "I-pipe: spurious interrupt %d\n", irq); + goto finalize_nosync; + } +#endif + + this_domain = ipipe_current_domain; + + if (test_bit(IPIPE_STICKY_FLAG, &this_domain->irqs[irq].control)) + head = &this_domain->p_link; + else { + head = __ipipe_pipeline.next; + next_domain = list_entry(head, struct ipipe_domain, p_link); + if (likely(test_bit(IPIPE_WIRED_FLAG, &next_domain->irqs[irq].control))) { + if (!m_ack && next_domain->irqs[irq].acknowledge != NULL) + next_domain->irqs[irq].acknowledge(irq, irq_desc + irq); + __ipipe_dispatch_wired(next_domain, irq); + goto finalize_nosync; + } + } + + /* Ack the interrupt. */ + + pos = head; + + while (pos != &__ipipe_pipeline) { + next_domain = list_entry(pos, struct ipipe_domain, p_link); + prefetch(next_domain); + + /* + * For each domain handling the incoming IRQ, mark it + * as pending in its log. + */ + if (test_bit(IPIPE_HANDLE_FLAG, &next_domain->irqs[irq].control)) { + /* + * Domains that handle this IRQ are polled for + * acknowledging it by decreasing priority + * order. The interrupt must be made pending + * _first_ in the domain's status flags before + * the PIC is unlocked. + */ + __ipipe_set_irq_pending(next_domain, irq); + + if (!m_ack && next_domain->irqs[irq].acknowledge) { + next_domain->irqs[irq].acknowledge(irq, irq_desc + irq); + m_ack = 1; + } + } + + /* + * If the domain does not want the IRQ to be passed + * down the interrupt pipe, exit the loop now. + */ + + if (!test_bit(IPIPE_PASS_FLAG, &next_domain->irqs[irq].control)) + break; + + pos = next_domain->p_link.next; + } + + /* + * If the interrupt preempted the head domain, then do not + * even try to walk the pipeline, unless an interrupt is + * pending for it. + */ + if (test_bit(IPIPE_AHEAD_FLAG, &this_domain->flags) && + ipipe_head_cpudom_var(irqpend_himap) == 0) + goto finalize_nosync; + + /* + * Now walk the pipeline, yielding control to the highest + * priority domain that has pending interrupt(s) or + * immediately to the current domain if the interrupt has been + * marked as 'sticky'. This search does not go beyond the + * current domain in the pipeline. + */ + + __ipipe_walk_pipeline(head); + +finalize_nosync: + if (!ipipe_root_domain_p || __ipipe_test_root()) + return 0; + +#ifdef CONFIG_SMP + /* + * Prevent a spurious rescheduling from being triggered on + * preemptible kernels along the way out through + * ret_from_intr. + */ + if (!regs) + __set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); +#endif /* CONFIG_SMP */ + + return 1; +} + +asmlinkage int __ipipe_grab_irq(int irq, struct pt_regs *regs) +{ + int status; + +#ifdef irq_finish + /* AT91 specific workaround */ + irq_finish(irq); +#endif /* irq_finish */ + + if (irq == __ipipe_mach_timerint) { + /* + * Given our deferred dispatching model for regular IRQs, we + * only record CPU regs for the last timer interrupt, so that + * the timer handler charges CPU times properly. It is assumed + * that other interrupt handlers don't actually care for such + * information. + */ + __raw_get_cpu_var(__ipipe_tick_regs).ARM_cpsr = + (ipipe_root_domain_p + ? regs->ARM_cpsr + : regs->ARM_cpsr | PSR_I_BIT); + __raw_get_cpu_var(__ipipe_tick_regs).ARM_pc = regs->ARM_pc; + } + + ipipe_trace_irq_entry(irq); + status = __ipipe_handle_irq(irq, regs); + ipipe_trace_irq_exit(irq); + + return status; +} + +EXPORT_SYMBOL_GPL(show_stack); +EXPORT_SYMBOL_GPL(init_mm); +#ifndef MULTI_CPU +EXPORT_SYMBOL_GPL(cpu_do_switch_mm); +#endif +EXPORT_SYMBOL_GPL(__check_kvm_seq); +#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) +EXPORT_SYMBOL_GPL(tasklist_lock); +#endif /* CONFIG_SMP || CONFIG_DEBUG_SPINLOCK */ + +#ifdef CONFIG_CPU_HAS_ASID +EXPORT_SYMBOL_GPL(__new_context); +EXPORT_SYMBOL_GPL(cpu_last_asid); +#endif /* CONFIG_CPU_HAS_ASID */ diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index b7cb45b..b759267 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -121,8 +121,10 @@ asmlinkage void __exception asm_do_IRQ(unsigned int irq, struct pt_regs *regs) generic_handle_irq(irq); } +#ifndef CONFIG_IPIPE /* AT91 specific workaround */ irq_finish(irq); +#endif /* !CONFIG_IPIPE */ irq_exit(); set_irq_regs(old_regs); diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index ba2adef..f612624 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -127,8 +127,15 @@ EXPORT_SYMBOL_GPL(arm_pm_restart); */ static void default_idle(void) { - if (!need_resched()) + if (!need_resched()) { +#ifdef CONFIG_IPIPE + __ipipe_unstall_root(); +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF + ipipe_trace_end(0x8000000E); +#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */ +#endif /* CONFIG_IPIPE */ arch_idle(); + } local_irq_enable(); } @@ -148,6 +155,7 @@ void cpu_idle(void) /* endless idle loop with no priority at all */ while (1) { tick_nohz_stop_sched_tick(1); + ipipe_suspend_domain(); leds_event(led_idle_start); while (!need_resched()) { #ifdef CONFIG_HOTPLUG_CPU diff --git a/arch/arm/kernel/ptrace.c b/arch/arm/kernel/ptrace.c index a2ea385..487ab4c 100644 --- a/arch/arm/kernel/ptrace.c +++ b/arch/arm/kernel/ptrace.c @@ -479,6 +479,10 @@ void ptrace_break(struct task_struct *tsk, struct pt_regs *regs) static int break_trap(struct pt_regs *regs, unsigned int instr) { + + if (ipipe_trap_notify(IPIPE_TRAP_BREAK,regs)) + return 0; + ptrace_break(current, regs); return 0; } diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 57162af..b891257 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -37,6 +37,7 @@ #include <asm/ptrace.h> #include <asm/localtimer.h> #include <asm/smp_plat.h> +#include <asm/fcse.h> /* * as from 2.5, kernels no longer have an init_tasks structure @@ -638,7 +639,7 @@ void flush_tlb_all(void) void flush_tlb_mm(struct mm_struct *mm) { if (tlb_ops_need_broadcast()) - on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, mm_cpumask(mm)); + on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, fcse_tlb_mask(mm)); else local_flush_tlb_mm(mm); } @@ -649,7 +650,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) struct tlb_args ta; ta.ta_vma = vma; ta.ta_start = uaddr; - on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_page(vma, uaddr); } @@ -672,7 +673,7 @@ void flush_tlb_range(struct vm_area_struct *vma, ta.ta_vma = vma; ta.ta_start = start; ta.ta_end = end; - on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_range(vma, start, end); } diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 3f361a7..efe92a5 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -331,6 +331,11 @@ asmlinkage void __exception do_undefinstr(struct pt_regs *regs) siginfo_t info; void __user *pc; + /* In kernel mode, the trap was already signaled at the beginning of + * __und_svc in arch/arm/kernel/entry-armv.S */ + if (user_mode(regs) && ipipe_trap_notify(IPIPE_TRAP_UNDEFINSTR, regs)) + return; + /* * According to the ARM ARM, PC is 2 or 4 bytes ahead, * depending whether we're in Thumb mode or not. @@ -728,13 +733,22 @@ void __init trap_init(void) return; } +#ifdef CONFIG_IPIPE +void *__ipipe_tsc_area; +#endif /* CONFIG_IPIPE */ + void __init early_trap_init(void) { unsigned long vectors = CONFIG_VECTORS_BASE; extern char __stubs_start[], __stubs_end[]; extern char __vectors_start[], __vectors_end[]; +#ifndef CONFIG_IPIPE extern char __kuser_helper_start[], __kuser_helper_end[]; int kuser_sz = __kuser_helper_end - __kuser_helper_start; +#else /* !CONFIG_IPIPE */ + extern char __ipipe_tsc_area_start[], __kuser_helper_end[]; + int kuser_sz = __kuser_helper_end - __ipipe_tsc_area_start; +#endif /* !CONFIG_IPIPE */ /* * Copy the vectors, stubs and kuser helpers (in entry-armv.S) @@ -743,7 +757,13 @@ void __init early_trap_init(void) */ memcpy((void *)vectors, __vectors_start, __vectors_end - __vectors_start); memcpy((void *)vectors + 0x200, __stubs_start, __stubs_end - __stubs_start); +#ifndef CONFIG_IPIPE memcpy((void *)vectors + 0x1000 - kuser_sz, __kuser_helper_start, kuser_sz); +#else /* !CONFIG_IPIPE */ + BUG_ON(0x1000 - kuser_sz < 0x200 + __stubs_end - __stubs_start); + memcpy((void *)vectors + 0x1000 - kuser_sz, __ipipe_tsc_area_start, kuser_sz); + __ipipe_tsc_area = (void *)vectors + 0x1000 - kuser_sz; +#endif /* !CONFIG_IPIPE */ /* * Copy signal return handlers into the vector page, and diff --git a/arch/arm/mach-at91/Kconfig b/arch/arm/mach-at91/Kconfig index 0b2ee95..2308048 100644 --- a/arch/arm/mach-at91/Kconfig +++ b/arch/arm/mach-at91/Kconfig @@ -406,6 +406,19 @@ endif # ---------------------------------------------------------- +comment "Adeos I-pipe Options" + +config IPIPE_AT91_TC + depends on IPIPE + int "AT91 TC used as time base by Adeos I-pipe" + default 0 + help + When Adeos interrupt pipeline is enabled, TC0 is used by default + as time base, but you can use TC1 or TC2 by setting this variable to 1 + or 2. This should only be needed to avoid conflicts with other drivers. + +# ---------------------------------------------------------- + comment "AT91 Board Options" config MTD_AT91_DATAFLASH_CARD diff --git a/arch/arm/mach-at91/Makefile b/arch/arm/mach-at91/Makefile index 709fbad..09cfdd0 100644 --- a/arch/arm/mach-at91/Makefile +++ b/arch/arm/mach-at91/Makefile @@ -83,3 +83,14 @@ obj-$(CONFIG_CPU_IDLE) += cpuidle.o ifeq ($(CONFIG_PM_DEBUG),y) CFLAGS_pm.o += -DDEBUG endif + +ifeq ($(CONFIG_IPIPE),y) +obj-y := $(filter-out at91rm9200_time.o at91sam926x_time.o at91x40_time.o, $(obj-y)) +obj-$(CONFIG_ARCH_AT91RM9200) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9260) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9261) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9263) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9RL) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9G20) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91X40) += at91_ipipe_time.o +endif diff --git a/arch/arm/mach-at91/at91_ipipe_time.c b/arch/arm/mach-at91/at91_ipipe_time.c new file mode 100644 index 0000000..a31515c --- /dev/null +++ b/arch/arm/mach-at91/at91_ipipe_time.c @@ -0,0 +1,419 @@ +/* + * linux/arch/arm/mach-at91/at91_ipipe_time.c + * + * Copyright (C) 2007 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> + * + * Adaptation to AT91SAM926x: + * Copyright (C) 2007 Gregory CLEMENT, Adeneo + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include <linux/interrupt.h> +#include <linux/irq.h> +#include <linux/kernel.h> +#include <linux/clockchips.h> +#include <linux/clk.h> +#include <linux/stringify.h> +#include <linux/err.h> +#include <linux/console.h> +#include <linux/module.h> +#include <linux/ipipe.h> + +#include <asm/io.h> +#include <asm/mach/time.h> + +#include <mach/hardware.h> +#include <mach/at91_st.h> +#include <mach/at91_tc.h> +#include <mach/at91_pit.h> +#include "clock.h" + +#if defined(CONFIG_ARCH_AT91RM9200) +#define AT91_ID_TC0 AT91RM9200_ID_TC0 +#define AT91_ID_TC1 AT91RM9200_ID_TC1 +#define AT91_ID_TC2 AT91RM9200_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9G20) +#define AT91_ID_TC0 AT91SAM9260_ID_TC0 +#define AT91_ID_TC1 AT91SAM9260_ID_TC1 +#define AT91_ID_TC2 AT91SAM9260_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9261) +#define AT91_ID_TC0 AT91SAM9261_ID_TC0 +#define AT91_ID_TC1 AT91SAM9261_ID_TC1 +#define AT91_ID_TC2 AT91SAM9261_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9263) +#define AT91_ID_TC0 AT91SAM9263_ID_TCB +#define AT91_ID_TC1 AT91SAM9263_ID_TCB +#define AT91_ID_TC2 AT91SAM9263_ID_TCB +#elif defined(CONFIG_ARCH_AT91SAM9RL) +#define AT91_ID_TC0 AT91SAM9RL_ID_TC0 +#define AT91_ID_TC1 AT91SAM9RL_ID_TC1 +#define AT91_ID_TC2 AT91SAM9RL_ID_TC2 +#elif defined(CONFIG_ARCH_AT91X40) +#define AT91_ID_TC0 AT91X40_ID_TC0 +#define AT91_ID_TC1 AT91X40_ID_TC1 +#define AT91_ID_TC2 AT91X40_ID_TC2 +#else +#error "AT91 processor unsupported by Adeos" +#endif + +#if (CONFIG_IPIPE_AT91_TC==0) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC0 +#elif (CONFIG_IPIPE_AT91_TC==1) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC1 +#elif (CONFIG_IPIPE_AT91_TC==2) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC2 +#else +#error IPIPE_AT91_TC must be 0, 1 or 2. +#endif + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ + +#define TCNXCNS(timer,v) ((v) << ((timer)<<1)) +#define AT91_TC_REG_MASK (0xffff) + +static unsigned long next_match; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + union { + struct { + unsigned short mid; + unsigned short low; + }; + unsigned long mid_low; + }; + }; +#else /* __LITTLE_ENDIAN */ + struct { + union { + struct { + unsigned short low; + unsigned short mid; + }; + unsigned long mid_low; + }; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; +static unsigned max_delta_ticks, min_delta_ticks; +static struct clock_event_device clkevt; +static int tc_timer_clock; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = + (unsigned *) + (AT91_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC + AT91_TC_CV); + info->u.fr.mask = AT91_TC_REG_MASK; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static inline unsigned int at91_tc_read(unsigned int reg_offset) +{ + unsigned long addr = + (AT91_VA_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC); + + return readl((void __iomem *)(addr + reg_offset)); +} + +static inline void at91_tc_write(unsigned int reg_offset, unsigned long value) +{ + unsigned long addr = + (AT91_VA_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC); + + writel(value, (void __iomem *)(addr + reg_offset)); +} + +#define read_CV() at91_tc_read(AT91_TC_CV) +#define read_RC() at91_tc_read(AT91_TC_RC) +#define write_RC(value) at91_tc_write(AT91_TC_RC, value) + +int __ipipe_mach_timerint = KERNEL_TIMER_IRQ_NUM; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static void ipipe_mach_update_tsc(unsigned short stamp); + +static int at91_timer_initialized; + +/* + * IRQ handler for the timer. + */ +static irqreturn_t at91_timer_interrupt(int irq, void *dev_id) +{ + clkevt.event_handler(&clkevt); + return IRQ_HANDLED; +} + +static struct irqaction at91_timer_irq = { + .name = "at91_tick", + .flags = IRQF_DISABLED | IRQF_TIMER, + .handler = &at91_timer_interrupt +}; + +static void ipipe_mach_update_tsc(unsigned short stamp) +{ + union tsc_reg *local_tsc; + + local_tsc = &tsc[ipipe_processor_id()]; + if (unlikely(stamp < local_tsc->low)) + local_tsc->full += AT91_TC_REG_MASK + 1; + local_tsc->low = stamp; +} + +void __ipipe_mach_acktimer(void) +{ + at91_tc_read(AT91_TC_SR); + + if (unlikely(!__ipipe_mach_timerstolen)) { + ipipe_mach_update_tsc(read_CV()); + next_match = (next_match + __ipipe_mach_ticks_per_jiffy) + & AT91_TC_REG_MASK; + write_RC(next_match); + } +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(at91_timer_initialized)) { + union tsc_reg *local_tsc, before, after; + unsigned short stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(after.full): "r"(local_tsc), "m"(*local_tsc)); + do { + before = after; + stamp = read_CV(); + barrier(); + __asm__ ("ldmia %1, %M0\n": + "=r"(after.full): + "r"(local_tsc), "m"(*local_tsc)); + } while (after.mid_low != before.mid_low); + if (unlikely(stamp < before.low)) + before.full += AT91_TC_REG_MASK + 1; + before.low = stamp; + return before.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +cycle_t at91_ipipe_get_clocksource(struct clocksource *cs) +{ + return (cycle_t)__ipipe_mach_get_tsc(); +} + +static struct clocksource clksrc = { + .name = "at91_tc" __stringify(CONFIG_IPIPE_AT91_TC), + .rating = 250, + .read = at91_ipipe_get_clocksource, + .mask = CLOCKSOURCE_MASK(64), + .shift = 20, + .flags = CLOCK_SOURCE_IS_CONTINUOUS, +}; + +static void +at91_tc_set_mode(enum clock_event_mode mode, struct clock_event_device *dev) +{ + /* Disable the channel */ + at91_tc_write(AT91_TC_CCR, AT91_TC_CLKDIS); + + /* Disable all interrupts. */ + at91_tc_write(AT91_TC_IDR, ~0ul); + + if (mode == CLOCK_EVT_MODE_PERIODIC) { + unsigned long v; + +#ifndef CONFIG_ARCH_AT91SAM9263 + clk_enable(clk_get(NULL, "tc" + __stringify(CONFIG_IPIPE_AT91_TC) "_clk")); +#else /* AT91SAM9263 */ + clk_enable(clk_get(NULL, "tcb_clk")); +#endif /* AT91SAM9263 */ + + /* No Sync. */ + at91_tc_write(AT91_TC_BCR, 0); + + /* program NO signal on XCN */ + v = readl((void __iomem *) (AT91_VA_BASE_TCB0 + AT91_TC_BMR)); + v &= ~TCNXCNS(CONFIG_IPIPE_AT91_TC, 3); + v |= TCNXCNS(CONFIG_IPIPE_AT91_TC, 1); /* AT91_TC_TCNXCNS_NONE */ + writel(v, (void __iomem *) (AT91_VA_BASE_TCB0 + AT91_TC_BMR)); + + /* Use the clock selected by at91_timer_init as input clock. */ + at91_tc_write(AT91_TC_CMR, tc_timer_clock); + + /* Load the TC register C. */ + next_match = __ipipe_mach_ticks_per_jiffy; + write_RC(next_match); + + /* Enable CPCS interrupt. */ + at91_tc_write(AT91_TC_IER, AT91_TC_CPCS); + + /* Enable the channel. */ + at91_tc_write(AT91_TC_CCR, AT91_TC_CLKEN | AT91_TC_SWTRG); + + at91_timer_initialized = 1; + } +} + +/* + * Reprogram the timer + */ +void __ipipe_mach_set_dec(unsigned long delay) +{ + unsigned short stamp; + unsigned long flags; + + if (delay > max_delta_ticks) + delay = max_delta_ticks; + + if (likely(delay > min_delta_ticks)) { + local_irq_save_hw(flags); + stamp = read_CV(); + ipipe_mach_update_tsc(stamp); + write_RC((stamp + delay) & AT91_TC_REG_MASK); + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(KERNEL_TIMER_IRQ_NUM); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clkevt.name); +} + +static struct clock_event_device clkevt = { + .name = "at91_tc" __stringify(CONFIG_IPIPE_AT91_TC), + .features = CLOCK_EVT_FEAT_PERIODIC, + .shift = 32, + .rating = 250, + .set_mode = at91_tc_set_mode, +}; + +void __ipipe_mach_release_timer(void) +{ + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return (read_RC() - read_CV()) & AT91_TC_REG_MASK; +} + +void __init at91_timer_init(void) +{ + unsigned char tc_divisors[] = { 2, 8, 32, 128, 0, }; + unsigned master_freq, divisor = 0, divided_freq = 0; + unsigned long long wrap_ns; + + /* Disable (boot loader) timer interrupts. */ +#if defined(CONFIG_ARCH_AT91RM9200) + at91_sys_write(AT91_ST_IDR, AT91_ST_PITS | AT91_ST_WDOVF | AT91_ST_RTTINC | AT91_ST_ALMS); + (void) at91_sys_read(AT91_ST_SR); /* Clear any pending interrupts */ +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9261) \ + || defined(CONFIG_ARCH_AT91SAM9263) || defined(CONFIG_ARCH_AT91SAM9RL) \ + || defined(CONFIG_ARCH_AT91SAM9G20) + at91_sys_write(AT91_PIT_MR, 0); + + /* Clear any pending interrupts */ + (void) at91_sys_read(AT91_PIT_PIVR); +#endif /* CONFIG_ARCH_AT91SAM926x */ + + master_freq = clk_get_rate(clk_get(NULL, "mck")); + /* Find the first frequency above 1 MHz */ + for (tc_timer_clock = ARRAY_SIZE(tc_divisors) - 1; + tc_timer_clock >= 0; tc_timer_clock--) { + divisor = tc_divisors[tc_timer_clock]; + divided_freq = (divisor + ? master_freq / divisor : AT91_SLOW_CLOCK); + if (divided_freq > 1000000) + break; + } + + wrap_ns = (unsigned long long) (AT91_TC_REG_MASK + 1) * NSEC_PER_SEC; + do_div(wrap_ns, divided_freq); + + if (divided_freq < 1000000) + printk(KERN_INFO "AT91 I-pipe warning: could not find a" + " frequency greater than 1MHz\n"); + + printk(KERN_INFO "AT91 I-pipe timer: div: %u, freq: %u.%06u MHz, wrap: " + "%u.%06u ms\n", divisor, + divided_freq / 1000000, divided_freq % 1000000, + (unsigned) wrap_ns / 1000000, (unsigned) wrap_ns % 1000000); + + /* Add a 1ms margin. It means that when an interrupt occurs, update_tsc + must be called within 1ms. update_tsc is called by acktimer when no + higher domain handles the timer, and called through set_dec when a + higher domain handles the timer. */ + wrap_ns -= 1000000; + /* Set up the interrupt. */ + setup_irq(KERNEL_TIMER_IRQ_NUM, &at91_timer_irq); + +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; +#endif /* CONFIG_SMP */ + + clkevt.mult = div_sc(divided_freq, NSEC_PER_SEC, clkevt.shift); + clkevt.max_delta_ns = wrap_ns; + clkevt.min_delta_ns = 2000; + clkevt.cpumask = cpumask_of(0); + clockevents_register_device(&clkevt); + + clksrc.mult = clocksource_hz2mult(divided_freq, clksrc.shift); + clocksource_register(&clksrc); + + __ipipe_mach_ticks_per_jiffy = (divided_freq + HZ/2) / HZ; + max_delta_ticks = (wrap_ns * clkevt.mult) >> clkevt.shift; + min_delta_ticks = ((unsigned long long) clkevt.min_delta_ns + * clkevt.mult) >> clkevt.shift; +} + +#ifdef CONFIG_ARCH_AT91RM9200 +struct sys_timer at91rm9200_timer = { +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9261) \ + || defined(CONFIG_ARCH_AT91SAM9263) || defined(CONFIG_ARCH_AT91SAM9RL) \ + || defined(CONFIG_ARCH_AT91SAM9G20) +struct sys_timer at91sam926x_timer = { +#elif defined(CONFIG_ARCH_AT91X40) +struct sys_timer at91x40_timer = { +#else +#error "Unknown machine" +#endif + .init = at91_timer_init, + .suspend = NULL, + .resume = NULL, +}; diff --git a/arch/arm/mach-at91/at91rm9200.c b/arch/arm/mach-at91/at91rm9200.c index 2e9ecad..2dea6d7 100644 --- a/arch/arm/mach-at91/at91rm9200.c +++ b/arch/arm/mach-at91/at91rm9200.c @@ -33,7 +33,14 @@ static struct map_desc at91rm9200_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91RM9200_BASE_EMAC), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ + }, { .virtual = AT91_IO_VIRT_BASE - AT91RM9200_SRAM_SIZE, .pfn = __phys_to_pfn(AT91RM9200_SRAM_BASE), .length = AT91RM9200_SRAM_SIZE, @@ -300,6 +307,7 @@ void __init at91rm9200_initialize(unsigned long main_clock, unsigned short banks * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91rm9200_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller (FIQ) */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -332,6 +340,42 @@ static unsigned int at91rm9200_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller (IRQ4) */ 0, /* Advanced Interrupt Controller (IRQ5) */ 0 /* Advanced Interrupt Controller (IRQ6) */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 6, /* System Peripheral */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, /* Parallel IO Controller D */ + 5, /* USART 0 */ + 5, /* USART 1 */ + 5, /* USART 2 */ + 5, /* USART 3 */ + 0, /* Multimedia Card Interface */ + 3, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 5, /* Serial Peripheral Interface */ + 4, /* Serial Synchronous Controller */ + 4, /* Serial Synchronous Controller */ + 4, /* Serial Synchronous Controller */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 0, /* Timer Counter 3 */ + 0, /* Timer Counter 4 */ + 0, /* Timer Counter 5 */ + 2, /* USB Host port */ + 2, /* Ethernet MAC */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0 /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91rm9200_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9260.c b/arch/arm/mach-at91/at91sam9260.c index 0894f10..63f05ad 100644 --- a/arch/arm/mach-at91/at91sam9260.c +++ b/arch/arm/mach-at91/at91sam9260.c @@ -31,6 +31,13 @@ static struct map_desc at91sam9260_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ } }; @@ -350,6 +357,7 @@ void __init at91sam9260_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9260_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -382,6 +390,42 @@ static unsigned int at91sam9260_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 7, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, /* Analog-to-Digital Converter */ + 6, /* USART 0 */ + 6, /* USART 1 */ + 6, /* USART 2 */ + 0, /* Multimedia Card Interface */ + 4, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 6, /* Serial Peripheral Interface 0 */ + 6, /* Serial Peripheral Interface 1 */ + 5, /* Serial Synchronous Controller */ + 0, + 0, + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 3, /* USB Host port */ + 3, /* Ethernet */ + 0, /* Image Sensor Interface */ + 6, /* USART 3 */ + 6, /* USART 4 */ + 6, /* USART 5 */ + 7, /* Timer Counter 3 */ + 7, /* Timer Counter 4 */ + 7, /* Timer Counter 5 */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9260_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9261.c b/arch/arm/mach-at91/at91sam9261.c index 4ecf379..8a13d5f 100644 --- a/arch/arm/mach-at91/at91sam9261.c +++ b/arch/arm/mach-at91/at91sam9261.c @@ -31,6 +31,13 @@ static struct map_desc at91sam9261_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, }; @@ -306,6 +313,7 @@ void __init at91sam9261_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9261_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -338,6 +346,42 @@ static unsigned int at91sam9261_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 7, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, + 6, /* USART 0 */ + 6, /* USART 1 */ + 6, /* USART 2 */ + 0, /* Multimedia Card Interface */ + 4, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 6, /* Serial Peripheral Interface 0 */ + 6, /* Serial Peripheral Interface 1 */ + 5, /* Serial Synchronous Controller 0 */ + 5, /* Serial Synchronous Controller 1 */ + 5, /* Serial Synchronous Controller 2 */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 3, /* USB Host port */ + 3, /* LCD Controller */ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9261_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9263.c b/arch/arm/mach-at91/at91sam9263.c index 942792d..c043f3a 100644 --- a/arch/arm/mach-at91/at91sam9263.c +++ b/arch/arm/mach-at91/at91sam9263.c @@ -30,6 +30,13 @@ static struct map_desc at91sam9263_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, { .virtual = AT91_IO_VIRT_BASE - AT91SAM9263_SRAM0_SIZE, .pfn = __phys_to_pfn(AT91SAM9263_SRAM0_BASE), @@ -311,6 +318,7 @@ void __init at91sam9263_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9263_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller (FIQ) */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -343,6 +351,42 @@ static unsigned int at91sam9263_default_irq_priority[NR_AIC_IRQS] __initdata = { 2, /* USB Host port */ 0, /* Advanced Interrupt Controller (IRQ0) */ 0, /* Advanced Interrupt Controller (IRQ1) */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller (FIQ) */ + 6, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C, D and E */ + 0, + 0, + 5, /* USART 0 */ + 5, /* USART 1 */ + 5, /* USART 2 */ + 0, /* Multimedia Card Interface 0 */ + 0, /* Multimedia Card Interface 1 */ + 3, /* CAN */ + 0, /* Two-Wire Interface */ + 5, /* Serial Peripheral Interface 0 */ + 5, /* Serial Peripheral Interface 1 */ + 4, /* Serial Synchronous Controller 0 */ + 4, /* Serial Synchronous Controller 1 */ + 5, /* AC97 Controller */ + 7, /* Timer Counter 0, 1 and 2 */ + 0, /* Pulse Width Modulation Controller */ + 2, /* Ethernet */ + 0, + 0, /* 2D Graphic Engine */ + 2, /* USB Device Port */ + 0, /* Image Sensor Interface */ + 2, /* LDC Controller */ + 0, /* DMA Controller */ + 0, + 2, /* USB Host port */ + 0, /* Advanced Interrupt Controller (IRQ0) */ + 0, /* Advanced Interrupt Controller (IRQ1) */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9263_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9rl.c b/arch/arm/mach-at91/at91sam9rl.c index 211c5c1..9cdad31 100644 --- a/arch/arm/mach-at91/at91sam9rl.c +++ b/arch/arm/mach-at91/at91sam9rl.c @@ -30,6 +30,13 @@ static struct map_desc at91sam9rl_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, }; @@ -303,6 +310,7 @@ void __init at91sam9rl_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9rl_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -335,6 +343,42 @@ static unsigned int at91sam9rl_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, 0, 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 6, /* System Peripherals */ + 1, /* Parallel IO Controller A */ + 1, /* Parallel IO Controller B */ + 1, /* Parallel IO Controller C */ + 1, /* Parallel IO Controller D */ + 4, /* USART 0 */ + 4, /* USART 1 */ + 4, /* USART 2 */ + 4, /* USART 3 */ + 0, /* Multimedia Card Interface */ + 5, /* Two-Wire Interface 0 */ + 5, /* Two-Wire Interface 1 */ + 4, /* Serial Peripheral Interface */ + 3, /* Serial Synchronous Controller 0 */ + 3, /* Serial Synchronous Controller 1 */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 0, + 0, /* Touch Screen Controller */ + 0, /* DMA Controller */ + 2, /* USB Device High speed port */ + 2, /* LCD Controller */ + 5, /* AC97 Controller */ + 0, + 0, + 0, + 0, + 0, + 0, + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9rl_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/gpio.c b/arch/arm/mach-at91/gpio.c index ae4772e..9b3c914 100644 --- a/arch/arm/mach-at91/gpio.c +++ b/arch/arm/mach-at91/gpio.c @@ -19,12 +19,20 @@ #include <linux/list.h> #include <linux/module.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <mach/at91_pio.h> #include <mach/gpio.h> #include <asm/gpio.h> +#ifdef CONFIG_IPIPE +#include <asm/irq.h> + +#ifdef __IPIPE_FEATURE_PIC_MUTE +DEFINE_PER_CPU(__ipipe_irqbits_t, __ipipe_muted_irqs); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ +#endif /* CONFIG_IPIPE */ #include "generic.h" @@ -375,6 +383,10 @@ static int gpio_irq_type(unsigned pin, unsigned type) static struct irq_chip gpio_irqchip = { .name = "GPIO", +#ifdef CONFIG_IPIPE + .ack = gpio_irq_mask, + .mask_ack = gpio_irq_mask, +#endif /* CONFIG_IPIPE */ .mask = gpio_irq_mask, .unmask = gpio_irq_unmask, .set_type = gpio_irq_type, @@ -422,7 +434,7 @@ static void gpio_irq_handler(unsigned irq, struct irq_desc *desc) gpio_irq_mask(pin); } else - generic_handle_irq(pin); + ipipe_handle_irq_cond(pin); } pin++; gpio++; @@ -433,6 +445,38 @@ static void gpio_irq_handler(unsigned irq, struct irq_desc *desc) /* now it may re-trigger */ } +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) +void ipipe_mute_pic(void) +{ + unsigned long unmasked, muted; + unsigned i; + + for (i = 0; i < gpio_banks; i++) { + unmasked = __raw_readl(gpio_chip[i].regbase + PIO_IMR); + muted = unmasked & __ipipe_irqbits[i + 1]; + __raw_get_cpu_var(__ipipe_muted_irqs)[i + 1] = muted; + __raw_writel(muted, gpio_chip[i].regbase + PIO_IDR); + } + + unmasked = at91_sys_read(AT91_AIC_IMR); + muted = unmasked & __ipipe_irqbits[0]; + __raw_get_cpu_var(__ipipe_muted_irqs)[0] = muted; + at91_sys_write(AT91_AIC_IDCR, muted); +} + +void ipipe_unmute_pic(void) +{ + unsigned i; + + at91_sys_write(AT91_AIC_IECR, __raw_get_cpu_var(__ipipe_muted_irqs)[0]); + for (i = 0; i < gpio_banks; i++) { + unsigned long muted; + muted = __raw_get_cpu_var(__ipipe_muted_irqs)[i + 1]; + __raw_writel(muted, gpio_chip[i].regbase + PIO_IER); + } +} +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ + /*--------------------------------------------------------------------------*/ #ifdef CONFIG_DEBUG_FS @@ -509,6 +553,10 @@ void __init at91_gpio_irq_setup(void) unsigned pioc, pin; struct at91_gpio_chip *this, *prev; +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) + memset(__ipipe_irqbits, ~0, sizeof(__ipipe_irqbits)); +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ + for (pioc = 0, pin = PIN_BASE, this = gpio_chip, prev = NULL; pioc++ < gpio_banks; prev = this, this++) { @@ -538,6 +586,9 @@ void __init at91_gpio_irq_setup(void) set_irq_chip_data(id, this); set_irq_chained_handler(id, gpio_irq_handler); +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) + __ipipe_irqbits[0] &= ~(1 << id); +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ } pr_info("AT91: %d gpio irqs in %d banks\n", pin - PIN_BASE, gpio_banks); } diff --git a/arch/arm/mach-at91/include/mach/hardware.h b/arch/arm/mach-at91/include/mach/hardware.h index a0df8b0..05ff930 100644 --- a/arch/arm/mach-at91/include/mach/hardware.h +++ b/arch/arm/mach-at91/include/mach/hardware.h @@ -63,6 +63,25 @@ #define AT91_VA_BASE_SYS AT91_IO_P2V(AT91_BASE_SYS) #define AT91_VA_BASE_EMAC AT91_IO_P2V(AT91RM9200_BASE_EMAC) +#ifdef CONFIG_IPIPE +#if defined(CONFIG_ARCH_AT91RM9200) +#define AT91_BASE_TCB0 AT91RM9200_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9G20) +#define AT91_BASE_TCB0 AT91SAM9260_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9261) +#define AT91_BASE_TCB0 AT91SAM9261_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9263) +#define AT91_BASE_TCB0 AT91SAM9263_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9RL) +#define AT91_BASE_TCB0 AT91SAM9RL_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91X40) +#define AT91_BASE_TCB0 (AT91_BASE_SYS + AT91_TC) +#else +#error "AT91 processor unsupported by Adeos" +#endif +#define AT91_VA_BASE_TCB0 AT91_IO_P2V(AT91_BASE_TCB0) +#endif + /* Internal SRAM is mapped below the IO devices */ #define AT91_SRAM_MAX SZ_1M #define AT91_VIRT_BASE (AT91_IO_VIRT_BASE - AT91_SRAM_MAX) diff --git a/arch/arm/mach-at91/include/mach/irqs.h b/arch/arm/mach-at91/include/mach/irqs.h index 36bd55f..cedaab2 100644 --- a/arch/arm/mach-at91/include/mach/irqs.h +++ b/arch/arm/mach-at91/include/mach/irqs.h @@ -45,4 +45,6 @@ /* FIQ is AIC source 0. */ #define FIQ_START AT91_ID_FIQ +#define __IPIPE_FEATURE_PIC_MUTE + #endif diff --git a/arch/arm/mach-at91/include/mach/timex.h b/arch/arm/mach-at91/include/mach/timex.h index 31ac2d9..29a70e2 100644 --- a/arch/arm/mach-at91/include/mach/timex.h +++ b/arch/arm/mach-at91/include/mach/timex.h @@ -82,6 +82,6 @@ #define AT91X40_MASTER_CLOCK 40000000 #define CLOCK_TICK_RATE (AT91X40_MASTER_CLOCK) -#endif +#endif /* arch specific */ #endif diff --git a/arch/arm/mach-at91/irq.c b/arch/arm/mach-at91/irq.c index da3494a..d375427 100644 --- a/arch/arm/mach-at91/irq.c +++ b/arch/arm/mach-at91/irq.c @@ -121,6 +121,9 @@ static struct irq_chip at91_aic_chip = { .name = "AIC", .ack = at91_aic_mask_irq, .mask = at91_aic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = at91_aic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = at91_aic_unmask_irq, .set_type = at91_aic_set_type, .set_wake = at91_aic_set_wake, diff --git a/arch/arm/mach-integrator/core.c b/arch/arm/mach-integrator/core.c index a0f60e5..84c99a7 100644 --- a/arch/arm/mach-integrator/core.c +++ b/arch/arm/mach-integrator/core.c @@ -2,6 +2,7 @@ * linux/arch/arm/mach-integrator/core.c * * Copyright (C) 2000-2003 Deep Blue Solutions Ltd + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2, as @@ -230,53 +231,62 @@ EXPORT_SYMBOL(cm_control); /* * How long is the timer interval? */ -#define TIMER_INTERVAL (TICKS_PER_uSEC * mSEC_10) -#if TIMER_INTERVAL >= 0x100000 -#define TICKS2USECS(x) (256 * (x) / TICKS_PER_uSEC) -#elif TIMER_INTERVAL >= 0x10000 -#define TICKS2USECS(x) (16 * (x) / TICKS_PER_uSEC) -#else #define TICKS2USECS(x) ((x) / TICKS_PER_uSEC) -#endif static unsigned long timer_reload; +static unsigned long timer_interval; +static unsigned long timer_lxlost; +static int tscok; + +#ifdef CONFIG_IPIPE +int __ipipe_mach_timerint = IRQ_TIMERINT1; +static unsigned long long __ipipe_mach_tsc; +static IPIPE_DEFINE_SPINLOCK(timer_lock); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} +#endif /* - * Returns number of ms since last clock interrupt. Note that interrupts - * will have been disabled by do_gettimeoffset() + * Called with IRQ disabled from do_gettimeofday(). */ -unsigned long integrator_gettimeoffset(void) +static inline unsigned long integrator_getticksoffset(void) { - unsigned long ticks1, ticks2, status; + unsigned long ticks; - /* - * Get the current number of ticks. Note that there is a race - * condition between us reading the timer and checking for - * an interrupt. We get around this by ensuring that the - * counter has not reloaded between our two reads. - */ - ticks2 = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - do { - ticks1 = ticks2; - status = __raw_readl(VA_IC_BASE + IRQ_RAW_STATUS); - ticks2 = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - } while (ticks2 > ticks1); + if (!tscok) + return 0; - /* - * Number of ticks since last interrupt. - */ - ticks1 = timer_reload - ticks2; + ticks = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - /* - * Interrupt pending? If so, we've reloaded once already. - */ - if (status & (1 << IRQ_TIMERINT1)) - ticks1 += timer_reload; + if (ticks > timer_reload) + ticks = 0xffff + timer_reload - ticks; + else + ticks = timer_reload - ticks; + if (timer_interval < 0x10000) + return ticks; + else if (timer_interval < 0x100000) + return ticks * 16; + else + return ticks * 256; +} + +/* + * Returns number of ms since last clock interrupt. Note that interrupts + * will have been disabled by do_gettimeoffset() + */ +unsigned long integrator_gettimeoffset(void) +{ /* * Convert the ticks to usecs */ - return TICKS2USECS(ticks1); + return TICKS2USECS(timer_lxlost + integrator_getticksoffset()); } /* @@ -285,11 +295,22 @@ unsigned long integrator_gettimeoffset(void) static irqreturn_t integrator_timer_interrupt(int irq, void *dev_id) { + timer_lxlost = 0; + +#ifdef CONFIG_IPIPE /* - * clear the interrupt + * If Linux is the only domain, ack the timer and reprogram it */ - writel(1, TIMER1_VA_BASE + TIMER_INTCLR); + if (!__ipipe_mach_timerstolen) { + __ipipe_mach_tsc += integrator_getticksoffset(); +#else + writel(1, TIMER1_VA_BASE + TIMER_INTCLR); +#endif + writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); +#ifdef CONFIG_IPIPE + } +#endif timer_tick(); return IRQ_HANDLED; @@ -301,24 +322,30 @@ static struct irqaction integrator_timer_irq = { .handler = integrator_timer_interrupt, }; -/* - * Set up timer interrupt, and return the current time in seconds. - */ -void __init integrator_time_init(unsigned long reload, unsigned int ctrl) +static inline void set_dec(unsigned long reload) { - unsigned int timer_ctrl = TIMER_CTRL_ENABLE | TIMER_CTRL_PERIODIC; + unsigned int ctrl = TIMER_CTRL_ENABLE | TIMER_CTRL_IE; timer_reload = reload; - timer_ctrl |= ctrl; + timer_interval = reload; - if (timer_reload > 0x100000) { + if (timer_reload >= 0x100000) { timer_reload >>= 8; - timer_ctrl |= TIMER_CTRL_DIV256; - } else if (timer_reload > 0x010000) { + ctrl |= TIMER_CTRL_DIV256; + } else if (timer_reload >= 0x010000) { timer_reload >>= 4; - timer_ctrl |= TIMER_CTRL_DIV16; + ctrl |= TIMER_CTRL_DIV16; } + writel(ctrl, TIMER1_VA_BASE + TIMER_CTRL); + writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); +} + +/* + * Set up timer interrupt, and return the current time in seconds. + */ +void __init integrator_time_init(unsigned long reload, unsigned int ctrl) +{ /* * Initialise to a known state (all timers off) */ @@ -326,12 +353,60 @@ void __init integrator_time_init(unsigned long reload, unsigned int ctrl) writel(0, TIMER1_VA_BASE + TIMER_CTRL); writel(0, TIMER2_VA_BASE + TIMER_CTRL); - writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); - writel(timer_reload, TIMER1_VA_BASE + TIMER_VALUE); - writel(timer_ctrl, TIMER1_VA_BASE + TIMER_CTRL); + set_dec(reload); /* * Make irqs happen for the system timer */ setup_irq(IRQ_TIMERINT1, &integrator_timer_irq); + + tscok = 1; +} + +#ifdef CONFIG_IPIPE + +void __ipipe_mach_acktimer(void) +{ + writel(1, TIMER1_VA_BASE + TIMER_INTCLR); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + unsigned long long result; + unsigned long flags; + + local_irq_save_hw_notrace(flags); + spin_lock(&timer_lock); + result = __ipipe_mach_tsc + integrator_getticksoffset(); + spin_unlock(&timer_lock); + local_irq_restore_hw_notrace(flags); + return result; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +void __ipipe_mach_set_dec(unsigned long reload) +{ + unsigned long ticks; + unsigned long flags; + + spin_lock_irqsave(&timer_lock, flags); + ticks = integrator_getticksoffset(); + __ipipe_mach_tsc += ticks; + timer_lxlost += ticks; + + set_dec(reload); + spin_unlock_irqrestore(&timer_lock, flags); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return readl(TIMER1_VA_BASE + TIMER_VALUE); } +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-integrator/include/mach/entry-macro.S b/arch/arm/mach-integrator/include/mach/entry-macro.S index 7649c57..ab3623d 100644 --- a/arch/arm/mach-integrator/include/mach/entry-macro.S +++ b/arch/arm/mach-integrator/include/mach/entry-macro.S @@ -28,7 +28,11 @@ teq \irqstat, #0 ldreq \irqstat, [\base, #(INTEGRATOR_HDR_IC_OFFSET+IRQ_STATUS)] moveq \irqnr, #IRQ_CIC_START - +#ifdef CONFIG_IPIPE + tst \irqstat, #0x00000040 @ check IRQ_TIMERINT1 first + movne \irqnr, #6 + bne 1003f +#endif /* CONFIG_IPIPE */ 1001: tst \irqstat, #15 bne 1002f add \irqnr, \irqnr, #4 diff --git a/arch/arm/mach-integrator/include/mach/irqs.h b/arch/arm/mach-integrator/include/mach/irqs.h index 1fbe6d1..dea6390 100644 --- a/arch/arm/mach-integrator/include/mach/irqs.h +++ b/arch/arm/mach-integrator/include/mach/irqs.h @@ -79,4 +79,3 @@ #define IRQ_SIC_END 46 #define NR_IRQS 47 - diff --git a/arch/arm/mach-integrator/include/mach/platform.h b/arch/arm/mach-integrator/include/mach/platform.h index e00a262..3b39ce3 100644 --- a/arch/arm/mach-integrator/include/mach/platform.h +++ b/arch/arm/mach-integrator/include/mach/platform.h @@ -417,7 +417,7 @@ * Timer definitions * * Only use timer 1 & 2 - * (both run at 24MHz and will need the clock divider set to 16). + * (both run at 1MHZ on /CP and at 24MHz on /AP) * * Timer 0 runs at bus frequency and therefore could vary and currently * uHAL can't handle that. @@ -430,7 +430,12 @@ #define MAX_TIMER 2 #define MAX_PERIOD 699050 + +#ifdef CONFIG_ARCH_INTEGRATOR_CP +#define TICKS_PER_uSEC 1 +#else #define TICKS_PER_uSEC 24 +#endif /* * These are useconds NOT ticks. diff --git a/arch/arm/mach-integrator/include/mach/timex.h b/arch/arm/mach-integrator/include/mach/timex.h index 1dcb420..a04653a 100644 --- a/arch/arm/mach-integrator/include/mach/timex.h +++ b/arch/arm/mach-integrator/include/mach/timex.h @@ -21,6 +21,6 @@ */ /* - * ?? + * Timer rate */ -#define CLOCK_TICK_RATE (50000000 / 16) +#define CLOCK_TICK_RATE (1000000) diff --git a/arch/arm/mach-integrator/integrator_cp.c b/arch/arm/mach-integrator/integrator_cp.c index 3f35293..91d2eaa 100644 --- a/arch/arm/mach-integrator/integrator_cp.c +++ b/arch/arm/mach-integrator/integrator_cp.c @@ -2,6 +2,7 @@ * linux/arch/arm/mach-integrator/integrator_cp.c * * Copyright (C) 2003 Deep Blue Solutions Ltd + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -21,6 +22,7 @@ #include <linux/amba/clcd.h> #include <linux/amba/mmci.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <asm/clkdev.h> #include <mach/clkdev.h> @@ -161,6 +163,9 @@ static struct irq_chip cic_chip = { .name = "CIC", .ack = cic_mask_irq, .mask = cic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = cic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = cic_unmask_irq, }; @@ -180,6 +185,9 @@ static struct irq_chip pic_chip = { .name = "PIC", .ack = pic_mask_irq, .mask = pic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = pic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = pic_unmask_irq, }; @@ -199,6 +207,9 @@ static struct irq_chip sic_chip = { .name = "SIC", .ack = sic_mask_irq, .mask = sic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = sic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = sic_unmask_irq, }; @@ -218,7 +229,7 @@ sic_handle_irq(unsigned int irq, struct irq_desc *desc) irq += IRQ_SIC_START; - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } while (status); } @@ -571,9 +582,14 @@ static void __init intcp_init(void) #define TIMER_CTRL_IE (1 << 5) /* Interrupt Enable */ +#ifdef CONFIG_IPIPE +unsigned int __ipipe_mach_ticks_per_jiffy = 1000000 * TICKS_PER_uSEC / HZ; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); +#endif + static void __init intcp_timer_init(void) { - integrator_time_init(1000000 / HZ, TIMER_CTRL_IE); + integrator_time_init(1000000 * TICKS_PER_uSEC / HZ, TIMER_CTRL_IE); } static struct sys_timer cp_timer = { diff --git a/arch/arm/mach-ixp4xx/common.c b/arch/arm/mach-ixp4xx/common.c index 3bbf40f..cfb1f44 100644 --- a/arch/arm/mach-ixp4xx/common.c +++ b/arch/arm/mach-ixp4xx/common.c @@ -45,6 +45,65 @@ static void __init ixp4xx_clocksource_init(void); static void __init ixp4xx_clockevent_init(void); static struct clock_event_device clockevent_ixp4xx; +#ifdef CONFIG_IPIPE +#include <linux/ipipe.h> + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif + +int __ipipe_mach_timerint = IRQ_IXP4XX_TIMER1; +int __ipipe_mach_timerstolen = 0; +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_timerint); +EXPORT_SYMBOL(__ipipe_mach_timerstolen); +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int ixp4xx_timer_initialized; + +#define ONE_SHOT_ENABLE (IXP4XX_OST_ENABLE|IXP4XX_OST_ONE_SHOT) + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = + (unsigned *) + (IXP4XX_TIMER_BASE_PHYS + IXP4XX_OSTS_OFFSET); + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + /************************************************************************* * IXP4xx chipset I/O mapping *************************************************************************/ @@ -269,8 +328,12 @@ static irqreturn_t ixp4xx_timer_interrupt(int irq, void *dev_id) { struct clock_event_device *evt = dev_id; +#ifndef CONFIG_IPIPE /* Clear Pending Interrupt by writing '1' to it */ *IXP4XX_OSST = IXP4XX_OSST_TIMER_1_PEND; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ evt->event_handler(evt); @@ -295,6 +358,14 @@ void __init ixp4xx_timer_init(void) /* Reset time-stamp counter */ *IXP4XX_OSTS = 0; +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + ixp4xx_timer_initialized = 1; +#endif /* Connect the interrupt handler and enable the interrupt */ setup_irq(IRQ_IXP4XX_TIMER1, &ixp4xx_timer_irq); @@ -470,6 +541,13 @@ static void ixp4xx_set_mode(enum clock_event_mode mode, *IXP4XX_OSRT1 = osrt | opts; } +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_ixp4xx.name); +} +#endif /* CONFIG_IPIPE */ + static struct clock_event_device clockevent_ixp4xx = { .name = "ixp4xx timer1", .features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT, @@ -491,3 +569,98 @@ static void __init ixp4xx_clockevent_init(void) clockevents_register_device(&clockevent_ixp4xx); } + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + /* Clear Pending Interrupt by writing '1' to it */ + *IXP4XX_OSST = IXP4XX_OSST_TIMER_1_PEND; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = *IXP4XX_OSTS; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(ixp4xx_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = *IXP4XX_OSTS; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + return result.full; + } + + return 0; +} + +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + * + * The timer is aperiodic (most of the time) when running Xenomai, so + * __ipipe_mach_set_dec is called for each timer tick and programs the + * timer hardware for the next tick. + * + */ +#define MIN_DELAY 333 /* 5 usec with the 66.66 MHz system clock */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + unsigned long flags; + if (delay > MIN_DELAY) { + local_irq_save_hw(flags); + *IXP4XX_OSRT1 = delay | ONE_SHOT_ENABLE; + local_irq_restore_hw(flags); + } else { + ipipe_trigger_irq(IRQ_IXP4XX_TIMER1); + } +} + +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +/* + * This returns the number of clock ticks remaining. + */ +unsigned long __ipipe_mach_get_dec(void) +{ + return(*IXP4XX_OST1); /* remaining */ +} + +EXPORT_SYMBOL(__ipipe_mach_get_dec); + +void __ipipe_mach_release_timer(void) +{ + unsigned long flags; + + local_irq_save_hw(flags); + ixp4xx_set_mode(clockevent_ixp4xx.mode, &clockevent_ixp4xx); + if (clockevent_ixp4xx.mode == CLOCK_EVT_MODE_ONESHOT) + ixp4xx_set_next_event(LATCH, &clockevent_ixp4xx); + local_irq_restore_hw(flags); +} + +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx25/devices.c b/arch/arm/mach-mx25/devices.c index 9fdeea1..89f74e8 100644 --- a/arch/arm/mach-mx25/devices.c +++ b/arch/arm/mach-mx25/devices.c @@ -19,6 +19,7 @@ #include <linux/platform_device.h> #include <linux/dma-mapping.h> #include <linux/gpio.h> +#include <mach/common.h> #include <mach/mx25.h> #include <mach/irqs.h> @@ -438,3 +439,14 @@ struct platform_device mx25_fec_device = { .num_resources = ARRAY_SIZE(mx25_fec_resources), .resource = mx25_fec_resources, }; + +#ifdef CONFIG_IPIPE +static int post_cpu_init(void) +{ + ipipe_mach_allow_hwtimer_uaccess(MX25_AIPS1_BASE_ADDR_VIRT, + MX25_AIPS2_BASE_ADDR_VIRT); + return 0; +} + +postcore_initcall(post_cpu_init); +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx3/devices.c b/arch/arm/mach-mx3/devices.c index 6adb586..9040c13 100644 --- a/arch/arm/mach-mx3/devices.c +++ b/arch/arm/mach-mx3/devices.c @@ -22,6 +22,7 @@ #include <linux/platform_device.h> #include <linux/serial.h> #include <linux/gpio.h> +#include <mach/common.h> #include <mach/hardware.h> #include <mach/irqs.h> #include <mach/common.h> @@ -603,3 +604,15 @@ static int mx3_devices_init(void) } subsys_initcall(mx3_devices_init); + +#ifdef CONFIG_IPIPE +static int post_cpu_init(void) +{ + if (cpu_is_mx31()) + ipipe_mach_allow_hwtimer_uaccess(AIPS1_BASE_ADDR_VIRT, + AIPS2_BASE_ADDR_VIRT); + return 0; +} + +postcore_initcall(post_cpu_init); +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx3/mx31ads.c b/arch/arm/mach-mx3/mx31ads.c index 938c549..b7c2139 100644 --- a/arch/arm/mach-mx3/mx31ads.c +++ b/arch/arm/mach-mx3/mx31ads.c @@ -25,6 +25,7 @@ #include <linux/gpio.h> #include <linux/i2c.h> #include <linux/irq.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach-types.h> @@ -134,7 +135,7 @@ static void mx31ads_expio_irq_handler(u32 irq, struct irq_desc *desc) if ((int_valid & 1) == 0) continue; - generic_handle_irq(expio_irq); + ipipe_handle_irq_cond(expio_irq); } } diff --git a/arch/arm/mach-mx3/mx31pdk.c b/arch/arm/mach-mx3/mx31pdk.c index 18715f1..aa47303 100644 --- a/arch/arm/mach-mx3/mx31pdk.c +++ b/arch/arm/mach-mx3/mx31pdk.c @@ -23,6 +23,7 @@ #include <linux/gpio.h> #include <linux/smsc911x.h> #include <linux/platform_device.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach-types.h> @@ -108,7 +109,7 @@ static void mx31pdk_expio_irq_handler(uint32_t irq, struct irq_desc *desc) for (; int_valid != 0; int_valid >>= 1, expio_irq++) { if ((int_valid & 1) == 0) continue; - generic_handle_irq(expio_irq); + ipipe_handle_irq_cond(expio_irq); } } diff --git a/arch/arm/mach-omap2/irq.c b/arch/arm/mach-omap2/irq.c index 26aeef5..5ed22fa 100644 --- a/arch/arm/mach-omap2/irq.c +++ b/arch/arm/mach-omap2/irq.c @@ -148,6 +148,7 @@ static struct irq_chip omap_irq_chip = { .name = "INTC", .ack = omap_mask_ack_irq, .mask = omap_mask_irq, + .mask_ack = omap_mask_ack_irq, .unmask = omap_unmask_irq, }; @@ -167,8 +168,14 @@ static void __init omap_irq_bank_init_one(struct omap_irq_bank *bank) while (!(intc_bank_read_reg(bank, INTC_SYSSTATUS) & 0x1)) /* Wait for reset to complete */; +#ifndef CONFIG_IPIPE /* Enable autoidle */ intc_bank_write_reg(1 << 0, bank, INTC_SYSCONFIG); +#else /* !CONFIG_IPIPE */ + /* Disable autoidle */ + intc_bank_write_reg(0, bank, INTC_SYSCONFIG); + intc_bank_write_reg(0x2, bank, INTC_IDLE); +#endif /* CONFIG_IPIPE */ } int omap_irq_pending(void) diff --git a/arch/arm/mach-omap2/timer-gp.c b/arch/arm/mach-omap2/timer-gp.c index cd04dea..389df09 100644 --- a/arch/arm/mach-omap2/timer-gp.c +++ b/arch/arm/mach-omap2/timer-gp.c @@ -35,6 +35,7 @@ #include <linux/irq.h> #include <linux/clocksource.h> #include <linux/clockchips.h> +#include <linux/ipipe.h> #include <asm/mach/time.h> #include <plat/dmtimer.h> @@ -48,13 +49,74 @@ static struct clock_event_device clockevent_gpt; static u8 __initdata gptimer_id = 1; static u8 __initdata inited; struct omap_dm_timer *gptimer_wakeup; +#ifndef CONFIG_OMAP_32K_TIMER +static struct omap_dm_timer *gpt_clocksource; +#endif /* !CONFIG_OMAP_32K_TIMER */ + +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int omap2_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *) + omap_dm_timer_get_phys_counter_addr(gpt_clocksource); + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ static irqreturn_t omap2_gp_timer_interrupt(int irq, void *dev_id) { - struct omap_dm_timer *gpt = (struct omap_dm_timer *)dev_id; struct clock_event_device *evt = &clockevent_gpt; +#ifndef CONFIG_IPIPE + struct omap_dm_timer *gpt = (struct omap_dm_timer *)dev_id; omap_dm_timer_write_status(gpt, OMAP_TIMER_INT_OVERFLOW); +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ evt->event_handler(evt); return IRQ_HANDLED; @@ -126,6 +188,13 @@ int __init omap2_gp_clockevent_set_gptimer(u8 id) return 0; } +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_gpt.name); +} +#endif /* CONFIG_IPIPE */ + static void __init omap2_gp_clockevent_init(void) { u32 tick_rate; @@ -158,6 +227,12 @@ static void __init omap2_gp_clockevent_init(void) gptimer_id, tick_rate); omap2_gp_timer_irq.dev_id = (void *)gptimer; + +#ifdef CONFIG_IPIPE + __ipipe_mach_timerint = omap_dm_timer_get_irq(gptimer); + __ipipe_mach_ticks_per_jiffy = (tick_rate + HZ / 2) / HZ; +#endif /* CONFIG_IPIPE */ + setup_irq(omap_dm_timer_get_irq(gptimer), &omap2_gp_timer_irq); omap_dm_timer_set_int_enable(gptimer, OMAP_TIMER_INT_OVERFLOW); @@ -168,7 +243,6 @@ static void __init omap2_gp_clockevent_init(void) clockevent_gpt.min_delta_ns = clockevent_delta2ns(3, &clockevent_gpt); /* Timer internal resynch latency. */ - clockevent_gpt.cpumask = cpumask_of(0); clockevents_register_device(&clockevent_gpt); } @@ -176,10 +250,10 @@ static void __init omap2_gp_clockevent_init(void) /* Clocksource code */ #ifdef CONFIG_OMAP_32K_TIMER -/* +/* * When 32k-timer is enabled, don't use GPTimer for clocksource * instead, just leave default clocksource which uses the 32k - * sync counter. See clocksource setup in see plat-omap/common.c. + * sync counter. See clocksource setup in see plat-omap/common.c. */ static inline void __init omap2_gp_clocksource_init(void) {} @@ -187,7 +261,6 @@ static inline void __init omap2_gp_clocksource_init(void) {} /* * clocksource */ -static struct omap_dm_timer *gpt_clocksource; static cycle_t clocksource_read_cycles(struct clocksource *cs) { return (cycle_t)omap_dm_timer_read_counter(gpt_clocksource); @@ -225,6 +298,16 @@ static void __init omap2_gp_clocksource_init(void) clocksource_gpt.mult = clocksource_khz2mult(tick_rate/1000, clocksource_gpt.shift); + +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + omap2_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + if (clocksource_register(&clocksource_gpt)) printk(err2, clocksource_gpt.name); } @@ -245,3 +328,79 @@ static void __init omap2_gp_timer_init(void) struct sys_timer omap_timer = { .init = omap2_gp_timer_init, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + omap_dm_timer_write_status(gptimer, OMAP_TIMER_INT_OVERFLOW); + omap_dm_timer_read_status(gptimer); +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + if (likely(omap2_timer_initialized)) { + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = omap_dm_timer_read_counter(gpt_clocksource); + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); + } +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(omap2_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n" : + "=r"(result.full) : "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = omap_dm_timer_read_counter(gpt_clocksource); + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > 3) + omap_dm_timer_set_load_start(gptimer, 0, 0xffffffff - delay); + else + ipipe_trigger_irq(__ipipe_mach_timerint); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + struct clock_event_device *ckdev = &clockevent_gpt; + ckdev->set_mode(ckdev->mode, ckdev); + if (ckdev->mode == CLOCK_EVT_MODE_ONESHOT) + ckdev->set_next_event(__ipipe_mach_ticks_per_jiffy, ckdev); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return 0xffffffff - omap_dm_timer_read_counter(gptimer); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-pxa/irq.c b/arch/arm/mach-pxa/irq.c index 1beb40f..3b27f7b 100644 --- a/arch/arm/mach-pxa/irq.c +++ b/arch/arm/mach-pxa/irq.c @@ -51,6 +51,9 @@ static struct irq_chip pxa_internal_irq_chip = { .name = "SC", .ack = pxa_mask_irq, .mask = pxa_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = pxa_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = pxa_unmask_irq, }; diff --git a/arch/arm/mach-pxa/leds-idp.c b/arch/arm/mach-pxa/leds-idp.c index 8b9c171..144cdc1 100644 --- a/arch/arm/mach-pxa/leds-idp.c +++ b/arch/arm/mach-pxa/leds-idp.c @@ -13,6 +13,7 @@ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/leds-lubbock.c b/arch/arm/mach-pxa/leds-lubbock.c index e26d5ef..706cced 100644 --- a/arch/arm/mach-pxa/leds-lubbock.c +++ b/arch/arm/mach-pxa/leds-lubbock.c @@ -12,6 +12,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/leds-mainstone.c b/arch/arm/mach-pxa/leds-mainstone.c index db4af5e..c204cd4 100644 --- a/arch/arm/mach-pxa/leds-mainstone.c +++ b/arch/arm/mach-pxa/leds-mainstone.c @@ -11,6 +11,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/lpd270.c b/arch/arm/mach-pxa/lpd270.c index 1373c22..f931637 100644 --- a/arch/arm/mach-pxa/lpd270.c +++ b/arch/arm/mach-pxa/lpd270.c @@ -24,6 +24,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/pwm_backlight.h> +#include <linux/ipipe.h> #include <asm/types.h> #include <asm/setup.h> @@ -124,7 +125,7 @@ static void lpd270_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear useless edge notification */ if (likely(pending)) { irq = LPD270_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); pending = __raw_readw(LPD270_INT_STATUS) & lpd270_irq_enabled; diff --git a/arch/arm/mach-pxa/lubbock.c b/arch/arm/mach-pxa/lubbock.c index 98ee7e5..5dbef64 100644 --- a/arch/arm/mach-pxa/lubbock.c +++ b/arch/arm/mach-pxa/lubbock.c @@ -22,6 +22,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/smc91x.h> +#include <linux/ipipe.h> #include <linux/spi/spi.h> #include <linux/spi/ads7846.h> @@ -161,7 +162,7 @@ static void lubbock_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear our parent irq */ if (likely(pending)) { irq = LUBBOCK_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = LUB_IRQ_SET_CLR & lubbock_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/mainstone.c b/arch/arm/mach-pxa/mainstone.c index 851ee0f..9c1428a 100644 --- a/arch/arm/mach-pxa/mainstone.c +++ b/arch/arm/mach-pxa/mainstone.c @@ -27,6 +27,7 @@ #include <linux/gpio_keys.h> #include <linux/pwm_backlight.h> #include <linux/smc91x.h> +#include <linux/ipipe.h> #include <asm/types.h> #include <asm/setup.h> @@ -165,7 +166,7 @@ static void mainstone_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear useless edge notification */ if (likely(pending)) { irq = MAINSTONE_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = MST_INTSETCLR & mainstone_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/pcm990-baseboard.c b/arch/arm/mach-pxa/pcm990-baseboard.c index d5255ae..cf0ec2a 100644 --- a/arch/arm/mach-pxa/pcm990-baseboard.c +++ b/arch/arm/mach-pxa/pcm990-baseboard.c @@ -24,6 +24,7 @@ #include <linux/platform_device.h> #include <linux/i2c.h> #include <linux/pwm_backlight.h> +#include <linux/ipipe.h> #include <media/soc_camera.h> @@ -263,7 +264,7 @@ static void pcm990_irq_handler(unsigned int irq, struct irq_desc *desc) GPIO_bit(PCM990_CTRL_INT_IRQ_GPIO); if (likely(pending)) { irq = PCM027_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = (~PCM990_INTSETCLR) & pcm990_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/time.c b/arch/arm/mach-pxa/time.c index 750c448..2b7e7f8 100644 --- a/arch/arm/mach-pxa/time.c +++ b/arch/arm/mach-pxa/time.c @@ -24,6 +24,60 @@ #include <asm/mach/time.h> #include <mach/regs-ost.h> +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint = IRQ_OST0; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int pxa_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *) 0x40A00010; + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + /* * This is PXA's sched_clock implementation. This has a resolution * of at least 308 ns and a maximum value of 208 days. @@ -66,8 +120,12 @@ pxa_ost0_interrupt(int irq, void *dev_id) struct clock_event_device *c = dev_id; /* Disarm the compare/match, signal the event. */ +#ifndef CONFIG_IPIPE OIER &= ~OIER_E0; OSSR = OSSR_M0; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ c->event_handler(c); return IRQ_HANDLED; @@ -79,8 +137,8 @@ pxa_osmr0_set_next_event(unsigned long delta, struct clock_event_device *dev) unsigned long flags, next, oscr; raw_local_irq_save(flags); - OIER |= OIER_E0; next = OSCR + delta; + OIER |= OIER_E0; OSMR0 = next; oscr = OSCR; raw_local_irq_restore(flags); @@ -125,6 +183,13 @@ static struct clock_event_device ckevt_pxa_osmr0 = { .set_mode = pxa_osmr0_set_mode, }; +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, ckevt_pxa_osmr0.name); +} +#endif /* CONFIG_IPIPE */ + static cycle_t pxa_read_oscr(struct clocksource *cs) { return OSCR; @@ -168,6 +233,15 @@ static void __init pxa_timer_init(void) setup_irq(IRQ_OST0, &pxa_ost0_irq); +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + pxa_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + clocksource_register(&cksrc_pxa_oscr0); clockevents_register_device(&ckevt_pxa_osmr0); } @@ -213,3 +287,81 @@ struct sys_timer pxa_timer = { .suspend = pxa_timer_suspend, .resume = pxa_timer_resume, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + OSSR = OSSR_M0; /* Clear match on timer 0 */ + OIER &= ~OIER_E0; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = OSCR; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(pxa_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = OSCR; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > MIN_OSCR_DELTA) { + unsigned long flags; + + local_irq_save_hw(flags); + OSMR0 = delay + OSCR; + OIER |= OIER_E0; + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(IRQ_OST0); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + pxa_osmr0_set_mode(ckevt_pxa_osmr0.mode, &ckevt_pxa_osmr0); + if (ckevt_pxa_osmr0.mode == CLOCK_EVT_MODE_ONESHOT) + pxa_osmr0_set_next_event(LATCH, &ckevt_pxa_osmr0); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return OSMR0 - OSCR; +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-pxa/viper.c b/arch/arm/mach-pxa/viper.c index 89f258c..51e1d64 100644 --- a/arch/arm/mach-pxa/viper.c +++ b/arch/arm/mach-pxa/viper.c @@ -41,6 +41,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/mtd/physmap.h> +#include <linux/ipipe.h> #include <mach/pxa25x.h> #include <mach/audio.h> @@ -285,7 +286,7 @@ static void viper_irq_handler(unsigned int irq, struct irq_desc *desc) if (likely(pending)) { irq = viper_bit_to_irq(__ffs(pending)); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = viper_irq_pending(); } while (pending); diff --git a/arch/arm/mach-s3c2410/include/mach/irqs.h b/arch/arm/mach-s3c2410/include/mach/irqs.h index 6c12c63..06ccc4e 100644 --- a/arch/arm/mach-s3c2410/include/mach/irqs.h +++ b/arch/arm/mach-s3c2410/include/mach/irqs.h @@ -3,6 +3,8 @@ * Copyright (c) 2003-2005 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. diff --git a/arch/arm/mach-s3c2440/irq.c b/arch/arm/mach-s3c2440/irq.c index 0c049b9..23a5c90 100644 --- a/arch/arm/mach-s3c2440/irq.c +++ b/arch/arm/mach-s3c2440/irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -25,6 +27,7 @@ #include <linux/ioport.h> #include <linux/sysdev.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -57,10 +60,10 @@ static void s3c_irq_demux_wdtac97(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_S3C2440_WDT); + ipipe_handle_irq_cond(IRQ_S3C2440_WDT); } if (subsrc & 2) { - generic_handle_irq(IRQ_S3C2440_AC97); + ipipe_handle_irq_cond(IRQ_S3C2440_AC97); } } } diff --git a/arch/arm/mach-sa1100/irq.c b/arch/arm/mach-sa1100/irq.c index 3093d46..04a56d2 100644 --- a/arch/arm/mach-sa1100/irq.c +++ b/arch/arm/mach-sa1100/irq.c @@ -15,6 +15,7 @@ #include <linux/irq.h> #include <linux/ioport.h> #include <linux/sysdev.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach/irq.h> @@ -125,7 +126,7 @@ sa1100_high_gpio_handler(unsigned int irq, struct irq_desc *desc) mask >>= 11; do { if (mask & 1) - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); mask >>= 1; irq++; } while (mask); @@ -217,6 +218,9 @@ static struct irq_chip sa1100_normal_chip = { .name = "SC", .ack = sa1100_mask_irq, .mask = sa1100_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = sa1100_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = sa1100_unmask_irq, .set_wake = sa1100_set_wake, }; diff --git a/arch/arm/mach-sa1100/leds-assabet.c b/arch/arm/mach-sa1100/leds-assabet.c index 64e9b4b..0d4e5a4 100644 --- a/arch/arm/mach-sa1100/leds-assabet.c +++ b/arch/arm/mach-sa1100/leds-assabet.c @@ -10,6 +10,7 @@ * - Red - on if system is not idle */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-badge4.c b/arch/arm/mach-sa1100/leds-badge4.c index cf1e384..e41140f 100644 --- a/arch/arm/mach-sa1100/leds-badge4.c +++ b/arch/arm/mach-sa1100/leds-badge4.c @@ -11,6 +11,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-cerf.c b/arch/arm/mach-sa1100/leds-cerf.c index 259b48e..3ab9eff 100644 --- a/arch/arm/mach-sa1100/leds-cerf.c +++ b/arch/arm/mach-sa1100/leds-cerf.c @@ -4,6 +4,7 @@ * Author: ??? */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-hackkit.c b/arch/arm/mach-sa1100/leds-hackkit.c index 2bce137..0a4fe91 100644 --- a/arch/arm/mach-sa1100/leds-hackkit.c +++ b/arch/arm/mach-sa1100/leds-hackkit.c @@ -10,6 +10,7 @@ * as cpu led, the green one is used as timer led. */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-lart.c b/arch/arm/mach-sa1100/leds-lart.c index 0505a1f..f03128d 100644 --- a/arch/arm/mach-sa1100/leds-lart.c +++ b/arch/arm/mach-sa1100/leds-lart.c @@ -10,6 +10,7 @@ * pace of the LED. */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-simpad.c b/arch/arm/mach-sa1100/leds-simpad.c index d50f4ee..2c154e3 100644 --- a/arch/arm/mach-sa1100/leds-simpad.c +++ b/arch/arm/mach-sa1100/leds-simpad.c @@ -4,6 +4,7 @@ * Author: Juergen Messerer <juergen.messerer@siemens.ch> */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/time.c b/arch/arm/mach-sa1100/time.c index b9cbb56..90c62f1 100644 --- a/arch/arm/mach-sa1100/time.c +++ b/arch/arm/mach-sa1100/time.c @@ -14,19 +14,78 @@ #include <linux/irq.h> #include <linux/timex.h> #include <linux/clockchips.h> +#include <linux/ipipe.h> #include <asm/mach/time.h> #include <mach/hardware.h> #define MIN_OSCR_DELTA 2 +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint = IRQ_OST0; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int sa1100_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *)0x90000010; + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + static irqreturn_t sa1100_ost0_interrupt(int irq, void *dev_id) { struct clock_event_device *c = dev_id; /* Disarm the compare/match, signal the event. */ +#ifndef CONFIG_IPIPE OIER &= ~OIER_E0; OSSR = OSSR_M0; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ c->event_handler(c); return IRQ_HANDLED; @@ -77,7 +136,14 @@ static struct clock_event_device ckevt_sa1100_osmr0 = { .set_mode = sa1100_osmr0_set_mode, }; -static cycle_t sa1100_read_oscr(struct clocksource *s) +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, ckevt_sa1100_osmr0.name); +} +#endif /* CONFIG_IPIPE */ + +static cycle_t sa1100_read_oscr(struct clocksource *cs) { return OSCR; } @@ -116,6 +182,15 @@ static void __init sa1100_timer_init(void) setup_irq(IRQ_OST0, &sa1100_timer_irq); +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + sa1100_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + clocksource_register(&cksrc_sa1100_oscr); clockevents_register_device(&ckevt_sa1100_osmr0); } @@ -156,3 +231,80 @@ struct sys_timer sa1100_timer = { .suspend = sa1100_timer_suspend, .resume = sa1100_timer_resume, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + OSSR = OSSR_M0; /* Clear match on timer 0 */ + OIER &= ~OIER_E0; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = OSCR; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(sa1100_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = OSCR; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > MIN_OSCR_DELTA) { + unsigned long flags; + + local_irq_save_hw(flags); + OSMR0 = delay + OSCR; + OIER |= OIER_E0; + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(IRQ_OST0); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + sa1100_osmr0_set_mode(ckevt_sa1100_osmr0.mode, &ckevt_sa1100_osmr0); + if (ckevt_sa1100_osmr0.mode == CLOCK_EVT_MODE_ONESHOT) + sa1100_osmr0_set_next_event(LATCH, &ckevt_sa1100_osmr0); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return OSMR0 - OSCR; +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index baf6384..9c6421c 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -781,3 +781,83 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARM_FCSE + bool "Fast Context Switch Extension (EXPERIMENTAL)" + depends on EXPERIMENTAL && !SMP && (CPU_32v4 || CPU_32v4T || CPU_32v5) + help + The Fast Context Switch Extension (FCSE for short) is an extension of + some ARM processors which allows to switch contexts between processes + without flushing cache and saves a few tens of microseconds in the + worst case. + + Enabling this option makes linux use the FCSE. + + We propose two modes: + - the guaranteed mode: we guarantee that there will never be any cache + flush when switching context, but this means that there can not be + more than 95 running processes in the system, each with a virtual + memory space smaller than 32MB, and that the shared memory + mappings do not use cache; + - the best effort mode: we allow some cache flushes to happen from + time to time, but do not limit the number of processes or the + virtual memory space available for each process, and the shared + memory mappings use cache. + +if ARM_FCSE + +choice + prompt "FCSE mode" + default ARM_FCSE_BEST_EFFORT + help + This option allow setting which FCSE mode will be used. + +config ARM_FCSE_GUARANTEED + bool "guaranteed" + help + Select guaranteed mode. + +config ARM_FCSE_BEST_EFFORT + bool "best effort" + help + Select best-effort mode. + +endchoice + +config ARM_FCSE_DYNPID + bool "Dynamic FCSE pid allocation" + depends on ARM_FCSE_BEST_EFFORT && !IPIPE + help + When this option is enabled, the kernel may decide to change the FCSE + pid of a process when switching context. This helps reducing the + number of cache flushes, but adds some overhead to the context + switches. + +config ARM_FCSE_PREEMPT_FLUSH + bool "Preemptible cache flushes" + help + When FCSE is enabled, some cache flushes happen with preemption + disabled, with this option, we make them preemptible, this results in + better response time, but may result in the need to restart a cache + flush if it was preempted, so in a global higher cpu consumption. + +config ARM_FCSE_MESSAGES + bool "help messages" + help + When FCSE is enabled in best-effort mode, due to the VM space + reduction, a too large stack size limit may result in processes + exceeding the 32MB limit too easily. A too small stack size may result + in stack overflows. Enabling this option will print messages in these + situations to assist you in tuning the stack size limit. + + In guaranteed mode, this option will cause message to be printed if + one of the hard limits (95 proceses, 32 MB VM space) is exceeded. + +config ARM_FCSE_DEBUG + bool "FCSE debug" + select ARM_FCSE_MESSAGES + help + This option enables some internal debug checks. It has a high + overhead, and is only useful for debugging the FCSE code. + +endif diff --git a/arch/arm/mm/alignment.c b/arch/arm/mm/alignment.c index 62820ed..510a94f 100644 --- a/arch/arm/mm/alignment.c +++ b/arch/arm/mm/alignment.c @@ -726,6 +726,9 @@ do_alignment(unsigned long addr, unsigned int fsr, struct pt_regs *regs) int isize = 4; int thumb2_32b = 0; + if (ipipe_trap_notify(IPIPE_TRAP_ALIGNMENT,regs)) + return 0; + instrptr = instruction_pointer(regs); fs = get_fs(); diff --git a/arch/arm/mm/copypage-v4mc.c b/arch/arm/mm/copypage-v4mc.c index 7370a71..3a7928a 100644 --- a/arch/arm/mm/copypage-v4mc.c +++ b/arch/arm/mm/copypage-v4mc.c @@ -44,7 +44,7 @@ static DEFINE_SPINLOCK(minicache_lock); * instruction. If your processor does not supply this, you have to write your * own copy_user_highpage that does the right thing. */ -static void __naked +static void notrace __naked mc_copy_user_page(void *from, void *to) { asm volatile( diff --git a/arch/arm/mm/copypage-xscale.c b/arch/arm/mm/copypage-xscale.c index 76824d3..db642ce 100644 --- a/arch/arm/mm/copypage-xscale.c +++ b/arch/arm/mm/copypage-xscale.c @@ -42,7 +42,7 @@ static DEFINE_SPINLOCK(minicache_lock); * Dcache aliasing issue. The writes will be forwarded to the write buffer, * and merged as appropriate. */ -static void __naked +static void notrace __naked mc_copy_user_page(void *from, void *to) { /* diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index 10e0680..4413c0e 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -73,6 +73,10 @@ void show_pte(struct mm_struct *mm, unsigned long addr) if (!mm) mm = &init_mm; +#ifdef CONFIG_ARM_FCSE + printk(KERN_ALERT "fcse pid: %ld, 0x%08lx\n", + mm->context.fcse.pid >> FCSE_PID_SHIFT, mm->context.fcse.pid); +#endif /* CONFIG_ARM_FCSE */ printk(KERN_ALERT "pgd = %p\n", mm->pgd); pgd = pgd_offset(mm, addr); printk(KERN_ALERT "[%08lx] *pgd=%08lx", addr, pgd_val(*pgd)); @@ -161,11 +165,21 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr, if (user_debug & UDBG_SEGV) { printk(KERN_DEBUG "%s: unhandled page fault (%d) at 0x%08lx, code 0x%03x\n", tsk->comm, sig, addr, fsr); +#ifdef CONFIG_ARM_FCSE_DYNPID + /* Disable preemption to avoid page tables changing under our + feet */ + preempt_disable(); +#endif /* CONFIG_ARM_FCSE_DYNPID */ show_pte(tsk->mm, addr); +#ifdef CONFIG_ARM_FCSE_DYNPID + preempt_enable(); +#endif /* CONFIG_ARM_FCSE_DYNPID */ show_regs(regs); } #endif + fcse_notify_segv(tsk->mm, addr, regs); + tsk->thread.address = addr; tsk->thread.error_code = fsr; tsk->thread.trap_no = 14; @@ -266,6 +280,9 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (notify_page_fault(regs, fsr)) return 0; + if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs)) + return 0; + tsk = current; mm = tsk->mm; @@ -386,6 +403,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr, if (addr < TASK_SIZE) return do_page_fault(addr, fsr, regs); + if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs)) + return 0; + index = pgd_index(addr); /* @@ -429,6 +449,10 @@ do_translation_fault(unsigned long addr, unsigned int fsr, static int do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + + if (ipipe_trap_notify(IPIPE_TRAP_SECTION,regs)) + return 0; + do_bad_area(addr, fsr, regs); return 0; } @@ -439,6 +463,9 @@ do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) static int do_bad(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + if (ipipe_trap_notify(IPIPE_TRAP_DABT,regs)) + return 0; + return 1; } @@ -514,6 +541,9 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (!inf->fn(addr, fsr & ~FSR_LNX_PF, regs)) return; + if (ipipe_trap_notify(IPIPE_TRAP_UNKNOWN,regs)) + return; + printk(KERN_ALERT "Unhandled fault: %s (0x%03x) at 0x%08lx\n", inf->name, fsr, addr); @@ -579,3 +609,42 @@ do_PrefetchAbort(unsigned long addr, unsigned int ifsr, struct pt_regs *regs) arm_notify_die("", regs, &info, ifsr, 0); } +#ifdef CONFIG_IPIPE +extern spinlock_t pgd_lock; +extern struct page *pgd_list; + +static void vmalloc_sync_one(pgd_t *pgd, unsigned long addr) +{ + unsigned int index = pgd_index(addr); + pgd_t *pgd_k; + pmd_t *pmd, *pmd_k; + + pgd += index; + pgd_k = init_mm.pgd + index; + + if (!pgd_present(*pgd)) + set_pgd(pgd, *pgd_k); + + pmd_k = pmd_offset(pgd_k, addr); + pmd = pmd_offset(pgd, addr); + + copy_pmd(pmd, pmd_k); +} + +void __ipipe_pin_range_globally(unsigned long start, unsigned long end) +{ + unsigned long next, addr = start; + + do { + unsigned long flags; + struct page *page; + + next = pgd_addr_end(addr, end); + spin_lock_irqsave(&pgd_lock, flags); + for (page = pgd_list; page; page = (struct page *)page->index) + vmalloc_sync_one(page_address(page), addr); + spin_unlock_irqrestore(&pgd_lock, flags); + + } while (addr = next, addr != end); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mm/ioremap.c b/arch/arm/mm/ioremap.c index 0ab75c6..32fdbe6 100644 --- a/arch/arm/mm/ioremap.c +++ b/arch/arm/mm/ioremap.c @@ -316,6 +316,7 @@ __arm_ioremap_pfn(unsigned long pfn, unsigned long offset, size_t size, } flush_cache_vmap(addr, addr + size); + __ipipe_pin_range_globally(addr, addr + size); return (void __iomem *) (offset + addr); } EXPORT_SYMBOL(__arm_ioremap_pfn); diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index f5abc51..fb22056 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -29,7 +29,8 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - unsigned long start_addr; + unsigned long start_addr = addr; + #ifdef CONFIG_CPU_V6 unsigned int cache_type; int do_align = 0, aliasing = 0; @@ -50,6 +51,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, #define aliasing 0 #endif +#ifdef CONFIG_ARM_FCSE + start_addr = addr; +#endif /* CONFIG_ARM_FCSE */ /* * We enforce the MAP_FIXED case. */ @@ -57,7 +61,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, if (aliasing && flags & MAP_SHARED && (addr - (pgoff << PAGE_SHIFT)) & (SHMLBA - 1)) return -EINVAL; - return addr; + goto found_addr; } if (len > TASK_SIZE) @@ -72,13 +76,13 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, vma = find_vma(mm, addr); if (TASK_SIZE - len >= addr && (!vma || addr + len <= vma->vm_start)) - return addr; + goto found_addr; } if (len > mm->cached_hole_size) { - start_addr = addr = mm->free_area_cache; + start_addr = addr = mm->free_area_cache; } else { - start_addr = addr = TASK_UNMAPPED_BASE; - mm->cached_hole_size = 0; + start_addr = addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; } full_search: @@ -106,14 +110,31 @@ full_search: * Remember the place where we stopped the search: */ mm->free_area_cache = addr + len; - return addr; + goto found_addr; } if (addr + mm->cached_hole_size < vma->vm_start) - mm->cached_hole_size = vma->vm_start - addr; + mm->cached_hole_size = vma->vm_start - addr; addr = vma->vm_end; if (do_align) addr = COLOUR_ALIGN(addr, pgoff); } + + found_addr: +#ifdef CONFIG_ARM_FCSE + { + unsigned long new_addr = fcse_check_mmap_addr(mm, start_addr, + addr, len, flags); + if (new_addr != addr) { + addr = new_addr; + if (!(addr & ~PAGE_MASK)) { + start_addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; + goto full_search; + } + } + } +#endif /* CONFIG_ARM_FCSE */ + return addr; } diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 2690146..16c48a3 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -18,6 +18,41 @@ #define FIRST_KERNEL_PGD_NR (FIRST_USER_PGD_NR + USER_PTRS_PER_PGD) +#ifdef CONFIG_IPIPE +/* Copied from arch/i386/mm/pgdtable.c, maintains the list of pgds for the + implementation of ipipe_pin_range_globally in arch/arm/mm/fault.c. */ +DEFINE_SPINLOCK(pgd_lock); +struct page *pgd_list; + +#define pgd_list_lock(flags) spin_lock_irqsave(&pgd_lock, flags) +#define pgd_list_unlock(flags) spin_unlock_irqrestore(&pgd_lock, flags) + +static inline void pgd_list_add(pgd_t *pgd) +{ + struct page *page = virt_to_page(pgd); + page->index = (unsigned long)pgd_list; + if (pgd_list) + set_page_private(pgd_list, (unsigned long)&page->index); + pgd_list = page; + set_page_private(page, (unsigned long)&pgd_list); +} + +static inline void pgd_list_del(pgd_t *pgd) +{ + struct page *next, **pprev, *page = virt_to_page(pgd); + next = (struct page *)page->index; + pprev = (struct page **)page_private(page); + *pprev = next; + if (next) + set_page_private(next, (unsigned long)pprev); +} +#else /* !CONFIG_IPIPE */ +#define pgd_list_lock(flags) ((void) (flags)) +#define pgd_list_unlock(flags) ((void) (flags)) +#define pgd_list_add(pgd) do { } while (0) +#define pgd_list_del(pgd) do { } while (0) +#endif /* !CONFIG_IPIPE */ + /* * need to get a 16k page for level 1 */ @@ -26,6 +61,7 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) pgd_t *new_pgd, *init_pgd; pmd_t *new_pmd, *init_pmd; pte_t *new_pte, *init_pte; + unsigned long flags; new_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, 2); if (!new_pgd) @@ -37,12 +73,20 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) * Copy over the kernel and IO PGD entries */ init_pgd = pgd_offset_k(0); + pgd_list_lock(flags); memcpy(new_pgd + FIRST_KERNEL_PGD_NR, init_pgd + FIRST_KERNEL_PGD_NR, (PTRS_PER_PGD - FIRST_KERNEL_PGD_NR) * sizeof(pgd_t)); + pgd_list_add(new_pgd); + pgd_list_unlock(flags); clean_dcache_area(new_pgd, PTRS_PER_PGD * sizeof(pgd_t)); if (!vectors_high()) { +#ifdef CONFIG_ARM_FCSE + /* FCSE does not work without high vectors. */ + BUG(); +#endif /* CONFIG_ARM_FCSE */ + /* * On ARM, first page must always be allocated since it * contains the machine vectors. @@ -74,6 +118,7 @@ no_pgd: void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) { + unsigned long flags; pmd_t *pmd; pgtable_t pte; @@ -81,7 +126,7 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) return; /* pgd is always present and good */ - pmd = pmd_off(pgd, 0); + pmd = pmd_off(pgd + pgd_index(fcse_va_to_mva(mm, 0)), 0); if (pmd_none(*pmd)) goto free; if (pmd_bad(*pmd)) { @@ -95,5 +140,8 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) pte_free(mm, pte); pmd_free(mm, pmd); free: + pgd_list_lock(flags); + pgd_list_del(pgd); + pgd_list_unlock(flags); free_pages((unsigned long) pgd, 2); } diff --git a/arch/arm/mm/proc-arm920.S b/arch/arm/mm/proc-arm920.S index 471669e..e474ecc 100644 --- a/arch/arm/mm/proc-arm920.S +++ b/arch/arm/mm/proc-arm920.S @@ -322,6 +322,11 @@ ENTRY(cpu_arm920_dcache_clean_area) ENTRY(cpu_arm920_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 3f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -339,6 +344,10 @@ ENTRY(cpu_arm920_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* !CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +3: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-arm926.S b/arch/arm/mm/proc-arm926.S index 64db6e2..ce7aa93 100644 --- a/arch/arm/mm/proc-arm926.S +++ b/arch/arm/mm/proc-arm926.S @@ -338,6 +338,11 @@ ENTRY(cpu_arm926_dcache_clean_area) ENTRY(cpu_arm926_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -347,6 +352,10 @@ ENTRY(cpu_arm926_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-feroceon.S b/arch/arm/mm/proc-feroceon.S index dbc3938..d8da8d6 100644 --- a/arch/arm/mm/proc-feroceon.S +++ b/arch/arm/mm/proc-feroceon.S @@ -422,6 +422,12 @@ ENTRY(cpu_feroceon_dcache_clean_area) .align 5 ENTRY(cpu_feroceon_switch_mm) #ifdef CONFIG_MMU +#ifndef CONFIG_ARM_FCSE_GUARANTEED +#ifdef CONFIG_ARM_FCSE + cmp r2, #0 + mov r2, lr + beq 2f +#else /* !CONFIG_ARM_FCSE */ /* * Note: we wish to call __flush_whole_cache but we need to preserve * lr to do so. The only way without touching main memory is to @@ -429,12 +435,19 @@ ENTRY(cpu_feroceon_switch_mm) * compensate locally for the skipped ops if it is not set. */ mov r2, lr @ abuse r2 to preserve lr +#endif /* !CONFIG_ARM_FCSE */ bl __flush_whole_cache @ if r2 contains the VM_EXEC bit then the next 2 ops are done already tst r2, #VM_EXEC mcreq p15, 0, ip, c7, c5, 0 @ invalidate I cache mcreq p15, 0, ip, c7, c10, 4 @ drain WB +#ifdef CONFIG_ARM_FCSE +2: +#endif +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + mov r2, lr +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs mov pc, r2 diff --git a/arch/arm/mm/proc-xscale.S b/arch/arm/mm/proc-xscale.S index 93df472..f794a86 100644 --- a/arch/arm/mm/proc-xscale.S +++ b/arch/arm/mm/proc-xscale.S @@ -416,9 +416,18 @@ ENTRY(cpu_xscale_dcache_clean_area) */ .align 5 ENTRY(cpu_xscale_switch_mm) +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED clean_d_cache r1, r2 mcr p15, 0, ip, c7, c5, 0 @ Invalidate I cache & BTB mcr p15, 0, ip, c7, c10, 4 @ Drain Write (& Fill) Buffer +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs cpwait_ret lr, ip diff --git a/arch/arm/plat-mxc/cpu.c b/arch/arm/plat-mxc/cpu.c index 386e0d5..0bb140e 100644 --- a/arch/arm/plat-mxc/cpu.c +++ b/arch/arm/plat-mxc/cpu.c @@ -1,5 +1,5 @@ - #include <linux/module.h> +#include <linux/io.h> unsigned int __mxc_cpu_type; EXPORT_SYMBOL(__mxc_cpu_type); @@ -9,3 +9,29 @@ void mxc_set_cpu_type(unsigned int type) __mxc_cpu_type = type; } +#ifdef CONFIG_IPIPE +void ipipe_mach_allow_hwtimer_uaccess(unsigned long aips1, unsigned long aips2) +{ + volatile unsigned long aips_reg; + + /* + * S/W workaround: Clear the off platform peripheral modules + * Supervisor Protect bit for SDMA to access them. + */ + __raw_writel(0x0, aips1 + 0x40); + __raw_writel(0x0, aips1 + 0x44); + __raw_writel(0x0, aips1 + 0x48); + __raw_writel(0x0, aips1 + 0x4C); + aips_reg = __raw_readl(aips1 + 0x50); + aips_reg &= 0x00FFFFFF; + __raw_writel(aips_reg, aips1 + 0x50); + + __raw_writel(0x0, aips2 + 0x40); + __raw_writel(0x0, aips2 + 0x44); + __raw_writel(0x0, aips2 + 0x48); + __raw_writel(0x0, aips2 + 0x4C); + aips_reg = __raw_readl(aips2 + 0x50); + aips_reg &= 0x00FFFFFF; + __raw_writel(aips_reg, aips2 + 0x50); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-mxc/gpio.c b/arch/arm/plat-mxc/gpio.c index d65ebe3..2eedd79 100644 --- a/arch/arm/plat-mxc/gpio.c +++ b/arch/arm/plat-mxc/gpio.c @@ -23,6 +23,7 @@ #include <linux/io.h> #include <linux/irq.h> #include <linux/gpio.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm-generic/bug.h> @@ -174,8 +175,7 @@ static void mxc_gpio_irq_handler(struct mxc_gpio_port *port, u32 irq_stat) if (port->both_edges & (1 << (gpio & 31))) mxc_flip_edge(port, gpio); - irq_desc[gpio_irq_no].handle_irq(gpio_irq_no, - &irq_desc[gpio_irq_no]); + ipipe_handle_irq_cond(gpio_irq_no); } } diff --git a/arch/arm/plat-mxc/include/mach/common.h b/arch/arm/plat-mxc/include/mach/common.h index 4bf1068..1cf801f 100644 --- a/arch/arm/plat-mxc/include/mach/common.h +++ b/arch/arm/plat-mxc/include/mach/common.h @@ -45,4 +45,8 @@ extern void mxc91231_power_off(void); extern void mxc91231_arch_reset(int, const char *); extern void mxc91231_prepare_idle(void); +#ifdef CONFIG_IPIPE +void ipipe_mach_allow_hwtimer_uaccess(unsigned long aips1, unsigned long aips2); +#endif + #endif diff --git a/arch/arm/plat-mxc/time.c b/arch/arm/plat-mxc/time.c index 844567e..bc60cac 100644 --- a/arch/arm/plat-mxc/time.c +++ b/arch/arm/plat-mxc/time.c @@ -25,6 +25,7 @@ #include <linux/irq.h> #include <linux/clockchips.h> #include <linux/clk.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach/time.h> @@ -57,6 +58,72 @@ #define MX3_TCN 0x24 #define MX3_TCMP 0x10 +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int mxc_timer_initialized; +static unsigned mxc_min_delay; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + if (cpu_is_mx1()) { +#ifdef CONFIG_ARCH_MX1 + info->u.fr.counter = (unsigned *) (TIM1_BASE_ADDR + MX1_2_TCN); +#endif + } else if (cpu_is_mx2()) { +#ifdef CONFIG_ARCH_MX2 + info->u.fr.counter = (unsigned *) (GPT1_BASE_ADDR + MX1_2_TCN); +#endif + } else if (cpu_is_mx3()) { +#ifdef CONFIG_ARCH_MX3 + info->u.fr.counter = (unsigned *) (GPT1_BASE_ADDR + MX3_TCN); +#endif + } + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); +#endif /* CONFIG_IPIPE */ + static struct clock_event_device clockevent_mxc; static enum clock_event_mode clockevent_mode = CLOCK_EVT_MODE_UNUSED; @@ -120,6 +187,9 @@ static int __init mxc_clocksource_init(struct clk *timer_clk) if (cpu_is_mx3() || cpu_is_mx25()) clocksource_mxc.read = mx3_get_cycles; +#ifdef CONFIG_IPIPE + __ipipe_mach_ticks_per_jiffy = (c + HZ/2) / HZ; +#endif /* CONFIG_IPIPE */ clocksource_mxc.mult = clocksource_hz2mult(c, clocksource_mxc.shift); clocksource_register(&clocksource_mxc); @@ -238,7 +308,11 @@ static irqreturn_t mxc_timer_interrupt(int irq, void *dev_id) else tstat = __raw_readl(timer_base + MX1_2_TSTAT); +#ifndef CONFIG_IPIPE gpt_irq_acknowledge(); +#else /* !CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* !CONFIG_IPIPE */ evt->event_handler(evt); @@ -309,4 +383,116 @@ void __init mxc_timer_init(struct clk *timer_clk, void __iomem *base, int irq) /* Make irqs happen */ setup_irq(irq, &mxc_timer_irq); + +#ifdef CONFIG_IPIPE + __ipipe_mach_timerint = irq; +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + mxc_min_delay = ((ipipe_cpu_freq() + 500000) / 1000000) ?: 1; + mxc_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ +} + +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_mxc.name); +} + +void __ipipe_mach_acktimer(void) +{ + gpt_irq_acknowledge(); +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + if (!cpu_is_mx3()) + stamp = __raw_readl(timer_base + MX1_2_TCN); + else + stamp = __raw_readl(timer_base + MX3_TCN); + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(mxc_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + if (!cpu_is_mx3()) { + __asm__ ("ldmia %1, %M0\n" + : "=r"(result.full), "+&r"(local_tsc) + : "m"(*local_tsc)); + barrier(); + stamp = __raw_readl(timer_base + MX1_2_TCN); + } else { + __asm__ ("ldmia %1, %M0\n" + : "=r"(result.full), "+&r"(local_tsc) + : "m"(*local_tsc)); + barrier(); + stamp = __raw_readl(timer_base + MX3_TCN); + } + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > mxc_min_delay) { + unsigned long flags; + + local_irq_save_hw(flags); + if (!cpu_is_mx3()) + mx1_2_set_next_event(delay, NULL); + else + mx3_set_next_event(delay, NULL); + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(__ipipe_mach_timerint); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + mxc_set_mode(clockevent_mxc.mode, &clockevent_mxc); + if (clockevent_mxc.mode == CLOCK_EVT_MODE_ONESHOT) + clockevent_mxc.set_next_event(LATCH, &clockevent_mxc); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + if (!cpu_is_mx3()) + return __raw_readl(timer_base + MX1_2_TCMP) + - __raw_readl(timer_base + MX1_2_TCN); + else + return __raw_readl(timer_base + MX3_TCMP) + - __raw_readl(timer_base + MX3_TCN); } +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-omap/Kconfig b/arch/arm/plat-omap/Kconfig index e2ea04a..7756c92 100644 --- a/arch/arm/plat-omap/Kconfig +++ b/arch/arm/plat-omap/Kconfig @@ -114,6 +114,7 @@ config OMAP_MPU_TIMER timer provides more intra-tick resolution than the 32KHz timer, but consumes more power. +if !IPIPE config OMAP_32K_TIMER bool "Use 32KHz timer" depends on ARCH_OMAP16XX || ARCH_OMAP24XX || ARCH_OMAP34XX || ARCH_OMAP4 @@ -123,6 +124,7 @@ config OMAP_32K_TIMER support for no tick during idle. The 32KHz timer provides less intra-tick resolution than OMAP_MPU_TIMER. The 32KHz timer is currently only available for OMAP16XX, 24XX, 34XX and OMAP4. +endif endchoice diff --git a/arch/arm/plat-omap/dmtimer.c b/arch/arm/plat-omap/dmtimer.c index 08ccf89..4d1abd7 100644 --- a/arch/arm/plat-omap/dmtimer.c +++ b/arch/arm/plat-omap/dmtimer.c @@ -475,6 +475,13 @@ int omap_dm_timer_get_irq(struct omap_dm_timer *timer) } EXPORT_SYMBOL_GPL(omap_dm_timer_get_irq); +#ifdef CONFIG_IPIPE +unsigned long omap_dm_timer_get_phys_counter_addr(struct omap_dm_timer *timer) +{ + return timer->phys_base + (OMAP_TIMER_COUNTER_REG & 0xff); +} +#endif /* CONFIG_IPIPE */ + #if defined(CONFIG_ARCH_OMAP1) /** diff --git a/arch/arm/plat-omap/gpio.c b/arch/arm/plat-omap/gpio.c index d2422c7..589a4bb 100644 --- a/arch/arm/plat-omap/gpio.c +++ b/arch/arm/plat-omap/gpio.c @@ -20,7 +20,9 @@ #include <linux/sysdev.h> #include <linux/err.h> #include <linux/clk.h> +#include <linux/irq.h> /* For irq_desc */ #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -1345,8 +1347,7 @@ static void gpio_irq_handler(unsigned int irq, struct irq_desc *desc) if (bank->toggle_mask & (1 << gpio_index)) _toggle_gpio_edge_triggering(bank, gpio_index); #endif - - generic_handle_irq(gpio_irq); + ipipe_handle_irq_cond(gpio_irq); } } /* if bank has any level sensitive GPIO pin interrupt diff --git a/arch/arm/plat-omap/include/plat/dmtimer.h b/arch/arm/plat-omap/include/plat/dmtimer.h index 20f1054..42a0b8d 100644 --- a/arch/arm/plat-omap/include/plat/dmtimer.h +++ b/arch/arm/plat-omap/include/plat/dmtimer.h @@ -56,6 +56,9 @@ void omap_dm_timer_enable(struct omap_dm_timer *timer); void omap_dm_timer_disable(struct omap_dm_timer *timer); int omap_dm_timer_get_irq(struct omap_dm_timer *timer); +#ifdef CONFIG_IPIPE +unsigned long omap_dm_timer_get_phys_counter_addr(struct omap_dm_timer *timer); +#endif /* CONFIG_IPIPE */ u32 omap_dm_timer_modify_idlect_mask(u32 inputmask); struct clk *omap_dm_timer_get_fclk(struct omap_dm_timer *timer); diff --git a/arch/arm/plat-pxa/gpio.c b/arch/arm/plat-pxa/gpio.c index 98548c6..e732e45 100644 --- a/arch/arm/plat-pxa/gpio.c +++ b/arch/arm/plat-pxa/gpio.c @@ -17,6 +17,7 @@ #include <linux/io.h> #include <linux/sysdev.h> #include <linux/slab.h> +#include <linux/ipipe.h> #include <mach/gpio.h> @@ -220,7 +221,7 @@ static void pxa_gpio_demux_handler(unsigned int irq, struct irq_desc *desc) while (n < BITS_PER_LONG) { loop = 1; - generic_handle_irq(gpio_to_irq(gpio_base + n)); + ipipe_handle_irq_cond(gpio_to_irq(gpio_base + n)); n = find_next_bit(&gedr, BITS_PER_LONG, n + 1); } } @@ -285,7 +286,7 @@ void __init pxa_init_gpio(int mux_irq, int start, int end, set_wake_t fn) for (irq = gpio_to_irq(start); irq <= gpio_to_irq(end); irq++) { set_irq_chip(irq, &pxa_muxed_gpio_chip); - set_irq_handler(irq, handle_edge_irq); + set_irq_handler(irq, handle_level_irq); set_irq_flags(irq, IRQF_VALID | IRQF_PROBE); } diff --git a/arch/arm/plat-s3c/time.c b/arch/arm/plat-s3c/time.c index 3b27b29..976fe03 100644 --- a/arch/arm/plat-s3c/time.c +++ b/arch/arm/plat-s3c/time.c @@ -3,6 +3,8 @@ * Copyright (C) 2003-2005 Simtec Electronics * Ben Dooks, <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -27,6 +29,7 @@ #include <linux/clk.h> #include <linux/io.h> #include <linux/platform_device.h> +#include <linux/module.h> #include <asm/system.h> #include <asm/leds.h> @@ -42,7 +45,6 @@ #include <plat/clock.h> #include <plat/cpu.h> -static unsigned long timer_startval; static unsigned long timer_usec_ticks; #ifndef TICK_MAX @@ -61,6 +63,39 @@ static unsigned long timer_usec_ticks; * Original patch by Dimitry Andric, updated by Ben Dooks */ +static unsigned long last_free_running_tcnt = 0; +static unsigned long free_running_tcon = 0; +static unsigned long timer_lxlost = 0; + +#ifdef CONFIG_IPIPE + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ + +unsigned int __ipipe_mach_ticks_per_jiffy; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +int __ipipe_mach_timerint = IRQ_TIMER4; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +static unsigned long long *tsc; +static unsigned *last_cnt; +static unsigned long timer_ackval = 1UL << (IRQ_TIMER4 - IRQ_EINT0); +static IPIPE_DEFINE_SPINLOCK(timer_lock); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_DECREMENTER; + info->u.dec.counter = (unsigned *)0x51000038; + info->u.dec.mask = 0xffff; + info->u.dec.last_cnt = last_cnt; + info->u.dec.tsc = tsc; +} +#endif /* CONFIG_IPIPE */ /* timer_mask_usec_ticks * @@ -91,38 +126,45 @@ static inline unsigned long timer_ticks_to_usec(unsigned long ticks) return res >> TIMER_USEC_SHIFT; } -/*** - * Returns microsecond since last clock interrupt. Note that interrupts - * will have been disabled by do_gettimeoffset() - * IRQs are disabled before entering here from do_gettimeofday() - */ - -static unsigned long s3c2410_gettimeoffset (void) +static inline unsigned long timer_freerunning_getvalue(void) { - unsigned long tdone; - unsigned long tval; - - /* work out how many ticks have gone since last timer interrupt */ + return __raw_readl(S3C2410_TCNTO(3)); +} - tval = __raw_readl(S3C2410_TCNTO(4)); - tdone = timer_startval - tval; +static inline unsigned long timer_freerunning_getticksoffset(unsigned long tval) +{ + long tdone; - /* check to see if there is an interrupt pending */ + tdone = last_free_running_tcnt - tval; + if (tdone < 0) + tdone += 0x10000; - if (s3c24xx_ostimer_pending()) { - /* re-read the timer, and try and fix up for the missed - * interrupt. Note, the interrupt may go off before the - * timer has re-loaded from wrapping. - */ + return tdone; +} - tval = __raw_readl(S3C2410_TCNTO(4)); - tdone = timer_startval - tval; +static inline unsigned long getticksoffset(void) +{ + return timer_freerunning_getticksoffset(timer_freerunning_getvalue()); +} - if (tval != 0) - tdone += timer_startval; - } +#ifdef CONFIG_IPIPE +static inline unsigned long getticksoffset_tscupdate(void) +{ + unsigned long tval; + unsigned long ticks; + + tval = timer_freerunning_getvalue(); + ticks = timer_freerunning_getticksoffset(tval); + last_free_running_tcnt = tval; + *tsc += ticks; + *last_cnt = last_free_running_tcnt; + return ticks; +} +#endif /* CONFIG_IPIPE */ - return timer_ticks_to_usec(tdone); +static unsigned long s3c2410_gettimeoffset (void) +{ + return timer_ticks_to_usec(timer_lxlost + getticksoffset()); } @@ -132,6 +174,13 @@ static unsigned long s3c2410_gettimeoffset (void) static irqreturn_t s3c2410_timer_interrupt(int irq, void *dev_id) { +#ifdef CONFIG_IPIPE + timer_lxlost = 0; + + if (!__ipipe_mach_timerstolen) + getticksoffset_tscupdate(); +#endif /* CONFIG_IPIPE */ + timer_tick(); return IRQ_HANDLED; } @@ -153,10 +202,10 @@ static struct clk *tdiv; static struct clk *timerclk; /* - * Set up timer interrupt, and return the current time in seconds. + * Set up timer interrupt. * - * Currently we only use timer4, as it is the only timer which has no - * other function that can be exploited externally + * Currently we use timer4 as event timer and timer3 as tick counter which + * permanently counts ticks without interrupt generation. */ static void s3c2410_timer_setup (void) { @@ -164,6 +213,7 @@ static void s3c2410_timer_setup (void) unsigned long tcnt; unsigned long tcfg1; unsigned long tcfg0; + unsigned long intmask; tcnt = TICK_MAX; /* default value for tcnt */ @@ -175,8 +225,8 @@ static void s3c2410_timer_setup (void) tcnt = 12000000 / HZ; tcfg1 = __raw_readl(S3C2410_TCFG1); - tcfg1 &= ~S3C2410_TCFG1_MUX4_MASK; - tcfg1 |= S3C2410_TCFG1_MUX4_TCLK1; + tcfg1 &= ~(S3C2410_TCFG1_MUX4_MASK | S3C2410_TCFG1_MUX3_MASK); + tcfg1 |= (S3C2410_TCFG1_MUX4_TCLK1 | S3C2410_TCFG1_MUX3_TCLK1); __raw_writel(tcfg1, S3C2410_TCFG1); } else { unsigned long pclk; @@ -210,6 +260,14 @@ static void s3c2410_timer_setup (void) tcfg0 = __raw_readl(S3C2410_TCFG0); tcfg1 = __raw_readl(S3C2410_TCFG1); +#ifdef CONFIG_IPIPE + tsc = (unsigned long long *)__ipipe_tsc_area; + last_cnt = (unsigned *)(tsc + 1); /* means: +8 bytes */ + barrier(); + + __ipipe_mach_ticks_per_jiffy = tcnt; +#endif /* CONFIG_IPIPE */ + /* timers reload after counting zero, so reduce the count by 1 */ tcnt--; @@ -226,23 +284,37 @@ static void s3c2410_timer_setup (void) __raw_writel(tcfg1, S3C2410_TCFG1); __raw_writel(tcfg0, S3C2410_TCFG0); - timer_startval = tcnt; - __raw_writel(tcnt, S3C2410_TCNTB(4)); - - /* ensure timer is stopped... */ + /* ensure timers are stopped... */ + tcon &= ~(0x3f<<17); + __raw_writel(tcon, S3C2410_TCON); - tcon &= ~(7<<20); - tcon |= S3C2410_TCON_T4RELOAD; - tcon |= S3C2410_TCON_T4MANUALUPD; + /* Mask timer3 interrupt. */ + intmask = __raw_readl(S3C2410_INTMSK); + intmask |= 1UL << (IRQ_TIMER3 - IRQ_EINT0); + __raw_writel(intmask, S3C2410_INTMSK); - __raw_writel(tcon, S3C2410_TCON); + /* Set timer values */ __raw_writel(tcnt, S3C2410_TCNTB(4)); __raw_writel(tcnt, S3C2410_TCMPB(4)); + __raw_writel(0xffff, S3C2410_TCNTB(3)); + __raw_writel(0xffff, S3C2410_TCMPB(3)); + + /* Set base tcon value for later programming of timer 4 by Xenomai. */ + free_running_tcon = tcon | S3C2410_TCON_T3RELOAD | S3C2410_TCON_T3START; + + /* Set auto reloads for both timers. */ + tcon |= S3C2410_TCON_T3RELOAD | S3C2410_TCON_T4RELOAD; - /* start the timer running */ - tcon |= S3C2410_TCON_T4START; - tcon &= ~S3C2410_TCON_T4MANUALUPD; + /* Manual update */ + __raw_writel(tcon | S3C2410_TCON_T3MANUALUPD + | S3C2410_TCON_T4MANUALUPD, S3C2410_TCON); + + tcon |= S3C2410_TCON_T3START | S3C2410_TCON_T4START; + /* Start timers.*/ __raw_writel(tcon, S3C2410_TCON); + + /* Save start value of timer 3 as begining of first period. */ + last_free_running_tcnt = 0xffff; } static void __init s3c2410_timer_resources(void) @@ -283,3 +355,58 @@ struct sys_timer s3c24xx_timer = { .offset = s3c2410_gettimeoffset, .resume = s3c2410_timer_setup }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + __raw_writel(timer_ackval, S3C2410_SRCPND); + __raw_writel(timer_ackval, S3C2410_INTPND); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + unsigned long long result; + unsigned long flags; + + local_irq_save_hw_notrace(flags); + spin_lock(&timer_lock); + result = *tsc + getticksoffset(); + spin_unlock(&timer_lock); + local_irq_restore_hw_notrace(flags); + return result; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +static inline void set_dec(unsigned long reload) +{ + __raw_writel(reload, S3C2410_TCNTB(4)); + /* Manual update */ + __raw_writel(free_running_tcon | S3C2410_TCON_T4MANUALUPD, S3C2410_TCON); + /* Start timer */ + __raw_writel(free_running_tcon | S3C2410_TCON_T4START, S3C2410_TCON); +} + +void __ipipe_mach_set_dec(unsigned long reload) +{ + unsigned long flags; + + spin_lock_irqsave(&timer_lock, flags); + timer_lxlost += getticksoffset_tscupdate(); + set_dec(reload); + spin_unlock_irqrestore(&timer_lock, flags); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + free_running_tcon |= S3C2410_TCON_T4RELOAD; + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy - 1); + free_running_tcon &= ~S3C2410_TCON_T4RELOAD; +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return __raw_readl(S3C2410_TCNTO(4)); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-s3c24xx/irq.c b/arch/arm/plat-s3c24xx/irq.c index ad0d44e..8444c8b 100644 --- a/arch/arm/plat-s3c24xx/irq.c +++ b/arch/arm/plat-s3c24xx/irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -23,6 +25,7 @@ #include <linux/interrupt.h> #include <linux/ioport.h> #include <linux/sysdev.h> +#include <linux/ipipe.h> #include <asm/irq.h> #include <asm/mach/irq.h> @@ -87,6 +90,9 @@ struct irq_chip s3c_irq_level_chip = { .name = "s3c-level", .ack = s3c_irq_maskack, .mask = s3c_irq_mask, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_maskack, +#endif /* CONFIG_IPIPE */ .unmask = s3c_irq_unmask, .set_wake = s3c_irq_wake }; @@ -283,6 +289,9 @@ static struct irq_chip s3c_irq_uart0 = { .mask = s3c_irq_uart0_mask, .unmask = s3c_irq_uart0_unmask, .ack = s3c_irq_uart0_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart0_ack, +#endif /* CONFIG_IPIPE */ }; /* UART1 */ @@ -310,6 +319,9 @@ static struct irq_chip s3c_irq_uart1 = { .mask = s3c_irq_uart1_mask, .unmask = s3c_irq_uart1_unmask, .ack = s3c_irq_uart1_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart1_ack, +#endif /* CONFIG_IPIPE */ }; /* UART2 */ @@ -337,6 +349,9 @@ static struct irq_chip s3c_irq_uart2 = { .mask = s3c_irq_uart2_mask, .unmask = s3c_irq_uart2_unmask, .ack = s3c_irq_uart2_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart2_ack, +#endif /* CONFIG_IPIPE */ }; /* ADC and Touchscreen */ @@ -385,10 +400,10 @@ static void s3c_irq_demux_adc(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_TC); + ipipe_handle_irq_cond(IRQ_TC); } if (subsrc & 2) { - generic_handle_irq(IRQ_ADC); + ipipe_handle_irq_cond(IRQ_ADC); } } } @@ -413,13 +428,13 @@ static void s3c_irq_demux_uart(unsigned int start) if (subsrc != 0) { if (subsrc & 1) - generic_handle_irq(start); + ipipe_handle_irq_cond(start); if (subsrc & 2) - generic_handle_irq(start+1); + ipipe_handle_irq_cond(start+1); if (subsrc & 4) - generic_handle_irq(start+2); + ipipe_handle_irq_cond(start+2); } } @@ -466,7 +481,7 @@ s3c_irq_demux_extint8(unsigned int irq, eintpnd &= ~(1<<irq); irq += (IRQ_EINT4 - 4); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } } @@ -489,7 +504,7 @@ s3c_irq_demux_extint4t7(unsigned int irq, irq += (IRQ_EINT4 - 4); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } } diff --git a/arch/arm/plat-s3c24xx/s3c244x-irq.c b/arch/arm/plat-s3c24xx/s3c244x-irq.c index a75c0c2..5dd2dd1 100644 --- a/arch/arm/plat-s3c24xx/s3c244x-irq.c +++ b/arch/arm/plat-s3c24xx/s3c244x-irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -25,6 +27,7 @@ #include <linux/ioport.h> #include <linux/sysdev.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -57,10 +60,10 @@ static void s3c_irq_demux_cam(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_S3C2440_CAM_C); + ipipe_handle_irq_cond(IRQ_S3C2440_CAM_C); } if (subsrc & 2) { - generic_handle_irq(IRQ_S3C2440_CAM_P); + ipipe_handle_irq_cond(IRQ_S3C2440_CAM_P); } } } @@ -89,6 +92,9 @@ static struct irq_chip s3c_irq_cam = { .mask = s3c_irq_cam_mask, .unmask = s3c_irq_cam_unmask, .ack = s3c_irq_cam_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_cam_ack, +#endif /* CONFIG_IPIPE */ }; static int s3c244x_irq_add(struct sys_device *sysdev) diff --git a/arch/arm/vfp/entry.S b/arch/arm/vfp/entry.S index 4fa9903..e46bae3 100644 --- a/arch/arm/vfp/entry.S +++ b/arch/arm/vfp/entry.S @@ -25,7 +25,6 @@ ENTRY(do_vfp) add r11, r4, #1 @ increment it str r11, [r10, #TI_PREEMPT] #endif - enable_irq ldr r4, .LCvfp ldr r11, [r10, #TI_CPU] @ CPU number add r10, r10, #TI_VFPSTATE @ r10 = workspace @@ -33,6 +32,7 @@ ENTRY(do_vfp) ENDPROC(do_vfp) ENTRY(vfp_null_entry) + enable_irq #ifdef CONFIG_PREEMPT get_thread_info r10 ldr r4, [r10, #TI_PREEMPT] @ get preempt count @@ -51,6 +51,7 @@ ENDPROC(vfp_null_entry) __INIT ENTRY(vfp_testing_entry) + enable_irq #ifdef CONFIG_PREEMPT get_thread_info r10 ldr r4, [r10, #TI_PREEMPT] @ get preempt count diff --git a/arch/arm/vfp/vfphw.S b/arch/arm/vfp/vfphw.S index 66dc2d0..b1b9c31 100644 --- a/arch/arm/vfp/vfphw.S +++ b/arch/arm/vfp/vfphw.S @@ -19,7 +19,7 @@ #include "../kernel/entry-header.S" .macro DBGSTR, str -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} add r0, pc, #4 bl printk @@ -31,7 +31,7 @@ .endm .macro DBGSTR1, str, arg -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} mov r1, \arg add r0, pc, #4 @@ -44,7 +44,7 @@ .endm .macro DBGSTR3, str, arg1, arg2, arg3 -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} mov r3, \arg3 mov r2, \arg2 @@ -87,6 +87,11 @@ ENTRY(vfp_support_entry) @ still there. In this case, we do @ not want to drop a pending exception. + enable_irq +#ifdef CONFIG_IPIPE + disable_irq + ldr r4, [r3, r11, lsl #2] @ reload last_VFP_context pointer +#endif VFPFMXR FPEXC, r5 @ enable VFP, disable any pending @ exceptions, so we can get at the @ rest of it @@ -139,6 +144,7 @@ check_for_exception: @ out before setting an FPEXC that @ stops us reading stuff VFPFMXR FPEXC, r1 @ restore FPEXC last + enable_irq_cond sub r2, r2, #4 str r2, [sp, #S_PC] @ retry the instruction #ifdef CONFIG_PREEMPT @@ -164,6 +170,7 @@ look_for_VFP_exceptions: @ Fall into hand on to next handler - appropriate coproc instr @ not recognised by VFP + enable_irq_cond DBGSTR "not VFP" #ifdef CONFIG_PREEMPT get_thread_info r10 diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index a63c4be..1fdff52 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -44,6 +44,7 @@ unsigned int VFP_arch; static void vfp_thread_flush(struct thread_info *thread) { union vfp_state *vfp = &thread->vfpstate; + unsigned long flags; unsigned int cpu; memset(vfp, 0, sizeof(union vfp_state)); @@ -56,22 +57,23 @@ static void vfp_thread_flush(struct thread_info *thread) * that the modification of last_VFP_context[] and hardware disable * are done for the same CPU and without preemption. */ - cpu = get_cpu(); + cpu = ipipe_get_cpu(flags); if (last_VFP_context[cpu] == vfp) last_VFP_context[cpu] = NULL; fmxr(FPEXC, fmrx(FPEXC) & ~FPEXC_EN); - put_cpu(); + ipipe_put_cpu(flags); } static void vfp_thread_exit(struct thread_info *thread) { /* release case: Per-thread VFP cleanup. */ union vfp_state *vfp = &thread->vfpstate; - unsigned int cpu = get_cpu(); + unsigned long flags; + unsigned int cpu = ipipe_get_cpu(flags); if (last_VFP_context[cpu] == vfp) last_VFP_context[cpu] = NULL; - put_cpu(); + ipipe_put_cpu(flags); } /* @@ -100,10 +102,13 @@ static void vfp_thread_exit(struct thread_info *thread) static int vfp_notifier(struct notifier_block *self, unsigned long cmd, void *v) { struct thread_info *thread = v; + unsigned long flags; if (likely(cmd == THREAD_NOTIFY_SWITCH)) { - u32 fpexc = fmrx(FPEXC); + u32 fpexc; + local_irq_save_hw_cond(flags); + fpexc = fmrx(FPEXC); #ifdef CONFIG_SMP unsigned int cpu = thread->cpu; @@ -130,6 +135,7 @@ static int vfp_notifier(struct notifier_block *self, unsigned long cmd, void *v) * old state. */ fmxr(FPEXC, fpexc & ~FPEXC_EN); + local_irq_restore_hw_cond(flags); return NOTIFY_DONE; } @@ -266,7 +272,7 @@ static u32 vfp_emulate_instruction(u32 inst, u32 fpscr, struct pt_regs *regs) */ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) { - u32 fpscr, orig_fpscr, fpsid, exceptions; + u32 fpscr, orig_fpscr, fpsid, exceptions, next_trigger = 0; pr_debug("VFP: bounce: trigger %08x fpexc %08x\n", trigger, fpexc); @@ -296,6 +302,7 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) /* * Synchronous exception, emulate the trigger instruction */ + local_irq_enable_hw_cond(); goto emulate; } @@ -308,13 +315,24 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) trigger = fmrx(FPINST); regs->ARM_pc -= 4; #endif - } else if (!(fpexc & FPEXC_DEX)) { + if (fpexc & FPEXC_FP2V) { + /* + * The barrier() here prevents fpinst2 being read + * before the condition above. + */ + barrier(); + next_trigger = fmrx(FPINST2); + } + } + local_irq_enable_hw_cond(); + + if (!(fpexc & (FPEXC_EX | FPEXC_DEX))) { /* * Illegal combination of bits. It can be caused by an * unallocated VFP instruction but with FPSCR.IXE set and not * on VFP subarch 1. */ - vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs); + vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs); goto exit; } @@ -348,18 +366,14 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) if (fpexc ^ (FPEXC_EX | FPEXC_FP2V)) goto exit; - /* - * The barrier() here prevents fpinst2 being read - * before the condition above. - */ - barrier(); - trigger = fmrx(FPINST2); + trigger = next_trigger; emulate: exceptions = vfp_emulate_instruction(trigger, orig_fpscr, regs); if (exceptions) vfp_raise_exceptions(exceptions, trigger, orig_fpscr, regs); exit: + local_irq_enable_hw_cond(); preempt_enable(); } diff --git a/drivers/i2c/busses/i2c-omap.c b/drivers/i2c/busses/i2c-omap.c index 0037e31..8d608a1 100644 --- a/drivers/i2c/busses/i2c-omap.c +++ b/drivers/i2c/busses/i2c-omap.c @@ -478,6 +478,10 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, if (msg->len == 0) return -EINVAL; +#ifdef CONFIG_IPIPE + disable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ + omap_i2c_write_reg(dev, OMAP_I2C_SA_REG, msg->addr); /* REVISIT: Could the STB bit of I2C_CON be used with probing? */ @@ -521,6 +525,9 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, /* Let the user know if i2c is in a bad state */ if (time_after(jiffies, delay)) { +#ifdef CONFIG_IPIPE + enable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ dev_err(dev->dev, "controller timed out " "waiting for start condition to finish\n"); return -ETIMEDOUT; @@ -532,6 +539,9 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, w &= ~OMAP_I2C_CON_STT; omap_i2c_write_reg(dev, OMAP_I2C_CON_REG, w); } +#ifdef CONFIG_IPIPE + enable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ /* * REVISIT: We should abort the transfer on signals, but the bus goes diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index e3551d2..d8f7317 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -46,7 +46,7 @@ config ATMEL_TCLIB config ATMEL_TCB_CLKSRC bool "TC Block Clocksource" - depends on ATMEL_TCLIB && GENERIC_TIME + depends on ATMEL_TCLIB && GENERIC_TIME && !IPIPE default y help Select this to get a high precision clocksource based on a diff --git a/drivers/pci/htirq.c b/drivers/pci/htirq.c index 737a1c4..15e81de 100644 --- a/drivers/pci/htirq.c +++ b/drivers/pci/htirq.c @@ -21,7 +21,7 @@ * With multiple simultaneous hypertransport irq devices it might pay * to make this more fine grained. But start with simple, stupid, and correct. */ -static DEFINE_SPINLOCK(ht_irq_lock); +static IPIPE_DEFINE_SPINLOCK(ht_irq_lock); struct ht_irq_cfg { struct pci_dev *dev; diff --git a/drivers/serial/8250.c b/drivers/serial/8250.c index e9b15c3..f93771a 100644 --- a/drivers/serial/8250.c +++ b/drivers/serial/8250.c @@ -3020,6 +3020,51 @@ static int serial8250_resume(struct platform_device *dev) return 0; } +#if defined(CONFIG_IPIPE_DEBUG) && defined(CONFIG_SERIAL_8250_CONSOLE) + +#include <stdarg.h> + +void __weak __ipipe_serial_debug(const char *fmt, ...) +{ + struct uart_8250_port *up = &serial8250_ports[0]; + unsigned int ier, count; + unsigned long flags; + char buf[128]; + va_list ap; + + va_start(ap, fmt); + vsprintf(buf, fmt, ap); + va_end(ap); + count = strlen(buf); + + touch_nmi_watchdog(); + + local_irq_save_hw(flags); + + /* + * First save the IER then disable the interrupts + */ + ier = serial_in(up, UART_IER); + + if (up->capabilities & UART_CAP_UUE) + serial_out(up, UART_IER, UART_IER_UUE); + else + serial_out(up, UART_IER, 0); + + uart_console_write(&up->port, buf, count, serial8250_console_putchar); + + /* + * Finally, wait for transmitter to become empty + * and restore the IER + */ + wait_for_xmitr(up, BOTH_EMPTY); + serial_out(up, UART_IER, ier); + + local_irq_restore_hw(flags); +} + +#endif + static struct platform_driver serial8250_isa_driver = { .probe = serial8250_probe, .remove = __devexit_p(serial8250_remove), diff --git a/fs/exec.c b/fs/exec.c index cce6bbd..1f2021d 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -714,6 +714,7 @@ static int exec_mmap(struct mm_struct *mm) { struct task_struct *tsk; struct mm_struct * old_mm, *active_mm; + unsigned long flags; /* Notify parent that we're no longer interested in the old VM */ tsk = current; @@ -736,8 +737,10 @@ static int exec_mmap(struct mm_struct *mm) task_lock(tsk); active_mm = tsk->active_mm; tsk->mm = mm; + ipipe_mm_switch_protect(flags); tsk->active_mm = mm; activate_mm(active_mm, mm); + ipipe_mm_switch_unprotect(flags); task_unlock(tsk); arch_pick_mmap_layout(mm); if (old_mm) { diff --git a/fs/proc/array.c b/fs/proc/array.c index 13b5d07..e2c2b55 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -144,6 +144,10 @@ static const char *task_state_array[] = { "x (dead)", /* 64 */ "K (wakekill)", /* 128 */ "W (waking)", /* 256 */ +#ifdef CONFIG_IPIPE + "A (atomic switch)", /* 512 */ + "N (wakeup disabled)", /* 1024 */ +#endif }; static inline const char *get_task_state(struct task_struct *tsk) diff --git a/include/asm-generic/atomic.h b/include/asm-generic/atomic.h index c99c64d..5d01b93 100644 --- a/include/asm-generic/atomic.h +++ b/include/asm-generic/atomic.h @@ -60,11 +60,11 @@ static inline int atomic_add_return(int i, atomic_t *v) unsigned long flags; int temp; - local_irq_save(flags); + local_irq_save_hw(flags); temp = v->counter; temp += i; v->counter = temp; - local_irq_restore(flags); + local_irq_restore_hw(flags); return temp; } @@ -82,11 +82,11 @@ static inline int atomic_sub_return(int i, atomic_t *v) unsigned long flags; int temp; - local_irq_save(flags); + local_irq_save_hw(flags); temp = v->counter; temp -= i; v->counter = temp; - local_irq_restore(flags); + local_irq_restore_hw(flags); return temp; } @@ -139,9 +139,9 @@ static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) unsigned long flags; mask = ~mask; - local_irq_save(flags); + local_irq_save_hw(flags); *addr &= mask; - local_irq_restore(flags); + local_irq_restore_hw(flags); } #define atomic_xchg(ptr, v) (xchg(&(ptr)->counter, (v))) diff --git a/include/asm-generic/bitops/atomic.h b/include/asm-generic/bitops/atomic.h index ecc44a8..5caf6e9 100644 --- a/include/asm-generic/bitops/atomic.h +++ b/include/asm-generic/bitops/atomic.h @@ -21,20 +21,20 @@ extern arch_spinlock_t __atomic_hash[ATOMIC_HASH_SIZE] __lock_aligned; * this is the substitute */ #define _atomic_spin_lock_irqsave(l,f) do { \ arch_spinlock_t *s = ATOMIC_HASH(l); \ - local_irq_save(f); \ + local_irq_save_hw(f); \ arch_spin_lock(s); \ } while(0) #define _atomic_spin_unlock_irqrestore(l,f) do { \ arch_spinlock_t *s = ATOMIC_HASH(l); \ arch_spin_unlock(s); \ - local_irq_restore(f); \ + local_irq_restore_hw(f); \ } while(0) #else -# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save(f); } while (0) -# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore(f); } while (0) +# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save_hw(f); } while (0) +# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore_hw(f); } while (0) #endif /* diff --git a/include/asm-generic/cmpxchg-local.h b/include/asm-generic/cmpxchg-local.h index b2ba2fc..ed01ab9 100644 --- a/include/asm-generic/cmpxchg-local.h +++ b/include/asm-generic/cmpxchg-local.h @@ -20,7 +20,7 @@ static inline unsigned long __cmpxchg_local_generic(volatile void *ptr, if (size == 8 && sizeof(unsigned long) != 8) wrong_size_cmpxchg(ptr); - local_irq_save(flags); + local_irq_save_hw(flags); switch (size) { case 1: prev = *(u8 *)ptr; if (prev == old) @@ -41,7 +41,7 @@ static inline unsigned long __cmpxchg_local_generic(volatile void *ptr, default: wrong_size_cmpxchg(ptr); } - local_irq_restore(flags); + local_irq_restore_hw(flags); return prev; } @@ -54,11 +54,11 @@ static inline u64 __cmpxchg64_local_generic(volatile void *ptr, u64 prev; unsigned long flags; - local_irq_save(flags); + local_irq_save_hw(flags); prev = *(u64 *)ptr; if (prev == old) *(u64 *)ptr = new; - local_irq_restore(flags); + local_irq_restore_hw(flags); return prev; } diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h index 8087b90..d5c9897 100644 --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -59,6 +59,20 @@ extern unsigned long __per_cpu_offset[NR_CPUS]; #define this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) #define __this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset) +#ifdef CONFIG_IPIPE +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) +extern int __ipipe_check_percpu_access(void); +#define __ipipe_local_cpu_offset \ + ({ \ + WARN_ON_ONCE(__ipipe_check_percpu_access()); \ + __my_cpu_offset; \ + }) +#else +#define __ipipe_local_cpu_offset __my_cpu_offset +#endif +#define __ipipe_get_cpu_var(var) \ + (*SHIFT_PERCPU_PTR(&per_cpu_var(var), __ipipe_local_cpu_offset)) +#endif /* CONFIG_IPIPE */ #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA extern void setup_per_cpu_areas(void); @@ -69,6 +83,7 @@ extern void setup_per_cpu_areas(void); #define per_cpu(var, cpu) (*((void)(cpu), &per_cpu_var(var))) #define __get_cpu_var(var) per_cpu_var(var) #define __raw_get_cpu_var(var) per_cpu_var(var) +#define __ipipe_get_cpu_var(var) __raw_get_cpu_var(var) #define this_cpu_ptr(ptr) per_cpu_ptr(ptr, 0) #define __this_cpu_ptr(ptr) this_cpu_ptr(ptr) diff --git a/include/asm-generic/resource.h b/include/asm-generic/resource.h index 587566f..66d60cd 100644 --- a/include/asm-generic/resource.h +++ b/include/asm-generic/resource.h @@ -58,6 +58,14 @@ #endif /* + * Limit the stack by to some sane default: root can always + * increase this limit if needed.. 8MB seems reasonable. + */ +#ifndef _STK_LIM +# define _STK_LIM (8*1024*1024) +#endif + +/* * RLIMIT_STACK default maximum - some architectures override it: */ #ifndef _STK_LIM_MAX diff --git a/include/linux/autoconf.h b/include/linux/autoconf.h new file mode 100644 index 0000000..9c633d6 --- /dev/null +++ b/include/linux/autoconf.h @@ -0,0 +1,552 @@ +/* + * Automatically generated C config: don't edit + * Linux kernel version: 2.6.32.7 + * Thu Feb 4 10:20:41 2010 + */ +#define AUTOCONF_INCLUDED +#define CONFIG_ACPI_AC 1 +#define CONFIG_ACPI_SYSFS_POWER 1 +#define CONFIG_DEBUG_SPINLOCK_SLEEP 1 +#define CONFIG_XENO_OPT_NATIVE_MPS 1 +#define CONFIG_FRAME_WARN 2048 +#define CONFIG_ARCH_DEFCONFIG "arch/x86/configs/x86_64_defconfig" +#define CONFIG_XENO_OPT_PRIOCPL 1 +#define CONFIG_SND_HDA_CODEC_CIRRUS 1 +#define CONFIG_TASK_XACCT 1 +#define CONFIG_POSIX_MQUEUE_SYSCTL 1 +#define CONFIG_ARCH_MAY_HAVE_PC_FDC 1 +#define CONFIG_CRYPTO_BLKCIPHER2 1 +#define CONFIG_CRYPTO_MD5 1 +#define CONFIG_X86_MCE_THRESHOLD 1 +#define CONFIG_PROC_KCORE 1 +#define CONFIG_XENO_OPT_TIMER_LIST 1 +#define CONFIG_CRYPTO_WORKQUEUE 1 +#define CONFIG_FTRACE_MCOUNT_RECORD 1 +#define CONFIG_CRYPTO_CBC 1 +#define CONFIG_TRACING 1 +#define CONFIG_DEVKMEM 1 +#define CONFIG_HAVE_FUNCTION_TRACER 1 +#define CONFIG_X86_PAT 1 +#define CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ 12 +#define CONFIG_XENO_OPT_NUCLEUS 1 +#define CONFIG_USE_GENERIC_SMP_HELPERS 1 +#define CONFIG_HAVE_KERNEL_BZIP2 1 +#define CONFIG_SND_SEQ_HRTIMER_DEFAULT 1 +#define CONFIG_SERIAL_8250_SHARE_IRQ 1 +#define CONFIG_PNPACPI 1 +#define CONFIG_TIMERFD 1 +#define CONFIG_ENABLE_MUST_CHECK 1 +#define CONFIG_EVENTFD 1 +#define CONFIG_X86_LOCAL_APIC 1 +#define CONFIG_NOP_TRACER 1 +#define CONFIG_BLK_DEV_SD 1 +#define CONFIG_HAS_DMA 1 +#define CONFIG_THERMAL 1 +#define CONFIG_X86_CPU 1 +#define CONFIG_EXT3_FS_XATTR 1 +#define CONFIG_NFS_COMMON 1 +#define CONFIG_X86_TSC 1 +#define CONFIG_HAVE_CPUMASK_OF_CPU_MAP 1 +#define CONFIG_SERIO_SERPORT 1 +#define CONFIG_SOUND_OSS_CORE_PRECLAIM 1 +#define CONFIG_HW_CONSOLE 1 +#define CONFIG_XENOMAI 1 +#define CONFIG_HPET_TIMER 1 +#define CONFIG_VGA_ARB 1 +#define CONFIG_CONSOLE_TRANSLATIONS 1 +#define CONFIG_ATA_SFF 1 +#define CONFIG_CHR_DEV_SG 1 +#define CONFIG_XENO_OPT_SYS_HEAPSZ 256 +#define CONFIG_KEYBOARD_ATKBD 1 +#define CONFIG_RCU_FANOUT 64 +#define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS 1 +#define CONFIG_HAS_IOPORT 1 +#define CONFIG_HAVE_ARCH_KGDB 1 +#define CONFIG_USB_EHCI_TT_NEWSCHED 1 +#define CONFIG_COMPAT_BINFMT_ELF 1 +#define CONFIG_PHYS_ADDR_T_64BIT 1 +#define CONFIG_XENO_OPT_PIPE 1 +#define CONFIG_MODULES 1 +#define CONFIG_HAVE_SETUP_PER_CPU_AREA 1 +#define CONFIG_OUTPUT_FORMAT "elf64-x86-64" +#define CONFIG_DYNAMIC_FTRACE 1 +#define CONFIG_USB_ARCH_HAS_EHCI 1 +#define CONFIG_CPU_SUP_INTEL 1 +#define CONFIG_PCI_QUIRKS 1 +#define CONFIG_INPUT_MISC 1 +#define CONFIG_MOUSE_PS2_TRACKPOINT 1 +#define CONFIG_BLK_DEV_SR 1 +#define CONFIG_EXT3_FS 1 +#define CONFIG_VT_CONSOLE 1 +#define CONFIG_CRYPTO_DES 1 +#define CONFIG_GENERIC_TRACER 1 +#define CONFIG_SND_MIXER_OSS 1 +#define CONFIG_PREEMPT 1 +#define CONFIG_ACPI 1 +#define CONFIG_USB_HID 1 +#define CONFIG_RTC_LIB 1 +#define CONFIG_SPARSEMEM_VMEMMAP 1 +#define CONFIG_GENERIC_IRQ_PROBE 1 +#define CONFIG_USER_STACKTRACE_SUPPORT 1 +#define CONFIG_CRYPTO_ALGAPI2 1 +#define CONFIG_SND_PCI 1 +#define CONFIG_DEFCONFIG_LIST "/lib/modules/$UNAME_RELEASE/.config" +#define CONFIG_X86_TRAMPOLINE 1 +#define CONFIG_DETECT_SOFTLOCKUP 1 +#define CONFIG_SLAB 1 +#define CONFIG_HAVE_LATENCYTOP_SUPPORT 1 +#define CONFIG_HOTPLUG 1 +#define CONFIG_USB_ARCH_HAS_OHCI 1 +#define CONFIG_SND_PCM 1 +#define CONFIG_DEVPORT 1 +#define CONFIG_SWIOTLB 1 +#define CONFIG_PCI_MSI 1 +#define CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK 1 +#define CONFIG_SPARSEMEM_VMEMMAP_ENABLE 1 +#define CONFIG_COMPAT_FOR_U64_ALIGNMENT 1 +#define CONFIG_SPARSEMEM_EXTREME 1 +#define CONFIG_SUNRPC_GSS 1 +#define CONFIG_PHYSICAL_ALIGN 0x1000000 +#define CONFIG_ARCH_PROC_KCORE_TEXT 1 +#define CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ 1024 +#define CONFIG_DEBUG_SPINLOCK 1 +#define CONFIG_VFAT_FS 1 +#define CONFIG_SERIAL_8250_PCI 1 +#define CONFIG_XENO_DRIVERS_SWITCHTEST 1 +#define CONFIG_IO_DELAY_TYPE_UDELAY 2 +#define CONFIG_R8169 1 +#define CONFIG_EMBEDDED 1 +#define CONFIG_GENERIC_ACL 1 +#define CONFIG_PROC_FS 1 +#define CONFIG_ARCH_HAS_CACHE_LINE_SIZE 1 +#define CONFIG_XENO_OPT_TIMING_VIRTICK 1000 +#define CONFIG_X86_CMPXCHG 1 +#define CONFIG_XENO_OPT_NATIVE_HEAP 1 +#define CONFIG_INET 1 +#define CONFIG_ARCH_SPARSEMEM_ENABLE 1 +#define CONFIG_RT_MUTEXES 1 +#define CONFIG_HPET 1 +#define CONFIG_SYSVIPC 1 +#define CONFIG_WLAN 1 +#define CONFIG_SCSI 1 +#define CONFIG_TCP_CONG_CUBIC 1 +#define CONFIG_X86_L1_CACHE_SHIFT 6 +#define CONFIG_SERIAL_8250_RSA 1 +#define CONFIG_ATA_VERBOSE_ERROR 1 +#define CONFIG_HAVE_IOREMAP_PROT 1 +#define CONFIG_SND_HDA_CODEC_REALTEK 1 +#define CONFIG_ACPI_PROCFS_POWER 1 +#define CONFIG_XENO_OPT_NATIVE_COND 1 +#define CONFIG_PROC_PAGE_MONITOR 1 +#define CONFIG_XENO_OPT_RTDM_FILDES 128 +#define CONFIG_IOSCHED_CFQ 1 +#define CONFIG_X86_IO_APIC 1 +#define CONFIG_PAGEFLAGS_EXTENDED 1 +#define CONFIG_NR_CPUS 8 +#define CONFIG_X86_EXTENDED_PLATFORM 1 +#define CONFIG_ARCH_HAS_DEFAULT_IDLE 1 +#define CONFIG_GENERIC_BUG 1 +#define CONFIG_SERIAL_8250_PNP 1 +#define CONFIG_CRYPTO_MANAGER 1 +#define CONFIG_X86_L1_CACHE_BYTES 64 +#define CONFIG_SWAP 1 +#define CONFIG_HAVE_MLOCKED_PAGE_BIT 1 +#define CONFIG_IPIPE_DEBUG 1 +#define CONFIG_EXT3_FS_POSIX_ACL 1 +#define CONFIG_AC97_BUS 1 +#define CONFIG_CRC32 1 +#define CONFIG_X86_MPPARSE 1 +#define CONFIG_DEFAULT_CFQ 1 +#define CONFIG_XENO_OPT_NATIVE_PIPE 1 +#define CONFIG_FAT_DEFAULT_IOCHARSET "iso8859-15" +#define CONFIG_INPUT_KEYBOARD 1 +#define CONFIG_EXTRA_FIRMWARE "" +#define CONFIG_PACKET_MMAP 1 +#define CONFIG_HAVE_DMA_ATTRS 1 +#define CONFIG_SND_SEQUENCER_OSS 1 +#define CONFIG_UNIX 1 +#define CONFIG_CPU_IDLE_GOV_MENU 1 +#define CONFIG_GENERIC_CPU 1 +#define CONFIG_XENO_OPT_RTDM_PERIOD 0 +#define CONFIG_ARCH_SELECT_MEMORY_MODEL 1 +#define CONFIG_MTRR 1 +#define CONFIG_PCI_DOMAINS 1 +#define CONFIG_ISA_DMA_API 1 +#define CONFIG_SERIAL_CORE 1 +#define CONFIG_SIGNALFD 1 +#define CONFIG_ACPI_PROCFS 1 +#define CONFIG_SCHED_OMIT_FRAME_POINTER 1 +#define CONFIG_RING_BUFFER 1 +#define CONFIG_UID16 1 +#define CONFIG_LOCK_KERNEL 1 +#define CONFIG_HAVE_SYSCALL_TRACEPOINTS 1 +#define CONFIG_64BIT 1 +#define CONFIG_USB_EHCI_ROOT_HUB_TT 1 +#define CONFIG_SCSI_WAIT_SCAN_MODULE 1 +#define CONFIG_PHYSICAL_START 0x200000 +#define CONFIG_IKCONFIG 1 +#define CONFIG_MOUSE_PS2_ALPS 1 +#define CONFIG_IP_FIB_HASH 1 +#define CONFIG_DEFAULT_MMAP_MIN_ADDR 4096 +#define CONFIG_XENO_FASTSYNCH 1 +#define CONFIG_TRACEPOINTS 1 +#define CONFIG_ARCH_USES_PG_UNCACHED 1 +#define CONFIG_ANON_INODES 1 +#define CONFIG_GENERIC_FIND_LAST_BIT 1 +#define CONFIG_SLABINFO 1 +#define CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST 1 +#define CONFIG_VGA_CONSOLE 1 +#define CONFIG_PCIEPORTBUS 1 +#define CONFIG_NLS_DEFAULT "iso8859-1" +#define CONFIG_SND_RAWMIDI 1 +#define CONFIG_ACPI_FAN 1 +#define CONFIG_ATA_ACPI 1 +#define CONFIG_SPLIT_PTLOCK_CPUS 4 +#define CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ 1 +#define CONFIG_SYSVIPC_COMPAT 1 +#define CONFIG_X86_SUPPORTS_MEMORY_FAILURE 1 +#define CONFIG_HAVE_DMA_API_DEBUG 1 +#define CONFIG_ARCH_WANT_FRAME_POINTERS 1 +#define CONFIG_ARCH_SUPPORTS_MSI 1 +#define CONFIG_HAVE_IDE 1 +#define CONFIG_IO_DELAY_TYPE_0X80 0 +#define CONFIG_LOCKD_V4 1 +#define CONFIG_SCHED_MC 1 +#define CONFIG_VM_EVENT_COUNTERS 1 +#define CONFIG_ACPI_BUTTON 1 +#define CONFIG_HAVE_FTRACE_NMI_ENTER 1 +#define CONFIG_CRYPTO_RNG2 1 +#define CONFIG_GENERIC_FIND_NEXT_BIT 1 +#define CONFIG_X86_CMOV 1 +#define CONFIG_XENO_OPT_TIMING_SCHEDLAT 0 +#define CONFIG_KERNEL_GZIP 1 +#define CONFIG_GENERIC_TIME_VSYSCALL 1 +#define CONFIG_UNUSED_SYMBOLS 1 +#define CONFIG_SYSCTL_SYSCALL 1 +#define CONFIG_MTRR_SANITIZER 1 +#define CONFIG_USB_STORAGE 1 +#define CONFIG_SND_HRTIMER 1 +#define CONFIG_SERIAL_8250_RUNTIME_UARTS 4 +#define CONFIG_COMPAT_VDSO 1 +#define CONFIG_ELF_CORE 1 +#define CONFIG_DEBUG_FS 1 +#define CONFIG_ARCH_SUSPEND_POSSIBLE 1 +#define CONFIG_SERIAL_8250_CONSOLE 1 +#define CONFIG_X86_DEBUGCTLMSR 1 +#define CONFIG_MAGIC_SYSRQ 1 +#define CONFIG_SND_DRIVERS 1 +#define CONFIG_SCSI_DMA 1 +#define CONFIG_DEFAULT_IOSCHED "cfq" +#define CONFIG_CRYPTO_HASH 1 +#define CONFIG_UNIX98_PTYS 1 +#define CONFIG_IP_PNP_DHCP 1 +#define CONFIG_ACPI_SBS 1 +#define CONFIG_X86_THERMAL_VECTOR 1 +#define CONFIG_XENO_SKIN_RTDM 1 +#define CONFIG_SOUND_OSS_CORE 1 +#define CONFIG_CPU_IDLE 1 +#define CONFIG_SND_MPU401_UART 1 +#define CONFIG_SND_VMASTER 1 +#define CONFIG_SYN_COOKIES 1 +#define CONFIG_X86_INTERNODE_CACHE_BYTES 64 +#define CONFIG_DMIID 1 +#define CONFIG_MOUSE_PS2_LIFEBOOK 1 +#define CONFIG_SCSI_MULTI_LUN 1 +#define CONFIG_GENERIC_ISA_DMA 1 +#define CONFIG_DEFAULT_IO_DELAY_TYPE 0 +#define CONFIG_PNP_DEBUG_MESSAGES 1 +#define CONFIG_BLOCK 1 +#define CONFIG_SND_OSSEMUL 1 +#define CONFIG_GENERIC_CLOCKEVENTS_BUILD 1 +#define CONFIG_GENERIC_HWEIGHT 1 +#define CONFIG_CRYPTO_MANAGER2 1 +#define CONFIG_LOCKD 1 +#define CONFIG_ZONE_DMA 1 +#define CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE 0 +#define CONFIG_ENABLE_WARN_DEPRECATED 1 +#define CONFIG_TIMER_STATS 1 +#define CONFIG_ATA_PIIX 1 +#define CONFIG_KALLSYMS 1 +#define CONFIG_ARCH_HIBERNATION_POSSIBLE 1 +#define CONFIG_HID_SUPPORT 1 +#define CONFIG_HAVE_KVM 1 +#define CONFIG_FSNOTIFY 1 +#define CONFIG_SYSVIPC_SYSCTL 1 +#define CONFIG_HAVE_UNSTABLE_SCHED_CLOCK 1 +#define CONFIG_X86_VERBOSE_BOOTUP 1 +#define CONFIG_NLS_UTF8 1 +#define CONFIG_SND_HDA_CODEC_CA0110 1 +#define CONFIG_FIRMWARE_IN_KERNEL 1 +#define CONFIG_TASK_IO_ACCOUNTING 1 +#define CONFIG_PROC_SYSCTL 1 +#define CONFIG_CONTEXT_SWITCH_TRACER 1 +#define CONFIG_IO_DELAY_TYPE_0XED 1 +#define CONFIG_IPIPE 1 +#define CONFIG_SND_PCM_OSS 1 +#define CONFIG_INPUT_MOUSE 1 +#define CONFIG_FIRMWARE_MEMMAP 1 +#define CONFIG_SERIAL_8250_MANY_PORTS 1 +#define CONFIG_TREE_RCU 1 +#define CONFIG_XENO_OPT_STATS 1 +#define CONFIG_SHMEM 1 +#define CONFIG_TASK_DELAY_ACCT 1 +#define CONFIG_ARCH_HAS_CPU_RELAX 1 +#define CONFIG_INET_LRO 1 +#define CONFIG_EPOLL 1 +#define CONFIG_CRYPTO_AEAD2 1 +#define CONFIG_RELAY 1 +#define CONFIG_XENO_OPT_NATIVE_ALARM 1 +#define CONFIG_RPCSEC_GSS_KRB5 1 +#define CONFIG_NLS_CODEPAGE_850 1 +#define CONFIG_NLS_CODEPAGE_437 1 +#define CONFIG_STACKTRACE_SUPPORT 1 +#define CONFIG_SERIO 1 +#define CONFIG_UEVENT_HELPER_PATH "/sbin/hotplug" +#define CONFIG_XENO_HW_FPU 1 +#define CONFIG_HAVE_KRETPROBES 1 +#define CONFIG_FUNCTION_TRACER 1 +#define CONFIG_FILE_LOCKING 1 +#define CONFIG_USB_SUPPORT 1 +#define CONFIG_SND_VERBOSE_PROCFS 1 +#define CONFIG_HUGETLB_PAGE 1 +#define CONFIG_IA32_AOUT 1 +#define CONFIG_DEBUG_KERNEL 1 +#define CONFIG_HAVE_ARCH_KMEMCHECK 1 +#define CONFIG_SOUND 1 +#define CONFIG_ROOT_NFS 1 +#define CONFIG_TMPFS 1 +#define CONFIG_RTC_INTF_DEV_UIE_EMUL 1 +#define CONFIG_SCHED_HRTICK 1 +#define CONFIG_GENERIC_TIME 1 +#define CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST 1 +#define CONFIG_HAVE_FTRACE_MCOUNT_RECORD 1 +#define CONFIG_FUTEX 1 +#define CONFIG_SERIO_LIBPS2 1 +#define CONFIG_ARCH_SPARSEMEM_DEFAULT 1 +#define CONFIG_BLOCK_COMPAT 1 +#define CONFIG_EXPERIMENTAL 1 +#define CONFIG_GENERIC_CLOCKEVENTS_BROADCAST 1 +#define CONFIG_X86_64 1 +#define CONFIG_USB_EHCI_HCD 1 +#define CONFIG_INPUT 1 +#define CONFIG_VIRT_TO_BUS 1 +#define CONFIG_ACPI_PROC_EVENT 1 +#define CONFIG_LOCALVERSION "" +#define CONFIG_USB_DEVICEFS 1 +#define CONFIG_SERIO_PCIPS2 1 +#define CONFIG_ZLIB_INFLATE 1 +#define CONFIG_XENO_OPT_NATIVE_EVENT 1 +#define CONFIG_INIT_ENV_ARG_LIMIT 32 +#define CONFIG_IPIPE_DELAYED_ATOMICSW 1 +#define CONFIG_CRYPTO_HASH2 1 +#define CONFIG_IO_DELAY_TYPE_NONE 3 +#define CONFIG_ARCH_PHYS_ADDR_T_64BIT 1 +#define CONFIG_XENO_OPT_NATIVE_QUEUE 1 +#define CONFIG_ACPI_BATTERY 1 +#define CONFIG_PCSPKR_PLATFORM 1 +#define CONFIG_HZ_250 1 +#define CONFIG_XENO_HW_SMI_DETECT 1 +#define CONFIG_SYSFS 1 +#define CONFIG_SND_RAWMIDI_SEQ 1 +#define CONFIG_X86_CMPXCHG64 1 +#define CONFIG_HAVE_KPROBES 1 +#define CONFIG_IPIPE_COMPAT 1 +#define CONFIG_IOSCHED_NOOP 1 +#define CONFIG_STOP_MACHINE 1 +#define CONFIG_HIGH_RES_TIMERS 1 +#define CONFIG_HAVE_DYNAMIC_FTRACE 1 +#define CONFIG_ACPI_DOCK 1 +#define CONFIG_PM 1 +#define CONFIG_GENERIC_CLOCKEVENTS 1 +#define CONFIG_SERIAL_8250_EXTENDED 1 +#define CONFIG_NO_HZ 1 +#define CONFIG_LOCKDEP_SUPPORT 1 +#define CONFIG_MSDOS_PARTITION 1 +#define CONFIG_USB_UHCI_HCD 1 +#define CONFIG_XENO_OPT_POSIX_PERIOD 0 +#define CONFIG_XENO_OPT_PERVASIVE 1 +#define CONFIG_HAVE_ARCH_TRACEHOOK 1 +#define CONFIG_X86_PM_TIMER 1 +#define CONFIG_HZ 250 +#define CONFIG_GENERIC_FIND_FIRST_BIT 1 +#define CONFIG_HUGETLBFS 1 +#define CONFIG_XENO_OPT_NATIVE_MUTEX 1 +#define CONFIG_TRACING_SUPPORT 1 +#define CONFIG_ACPI_BLACKLIST_YEAR 0 +#define CONFIG_XENO_GENERIC_STACKPOOL 1 +#define CONFIG_IOMMU_HELPER 1 +#define CONFIG_XENO_OPT_SYS_STACKPOOLSZ 128 +#define CONFIG_SSB_POSSIBLE 1 +#define CONFIG_X86_MINIMUM_CPU_FAMILY 64 +#define CONFIG_USB_LIBUSUAL 1 +#define CONFIG_VT 1 +#define CONFIG_TICK_ONESHOT 1 +#define CONFIG_TMPFS_POSIX_ACL 1 +#define CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE 1 +#define CONFIG_GENERIC_PENDING_IRQ 1 +#define CONFIG_RWSEM_GENERIC_SPINLOCK 1 +#define CONFIG_BRANCH_PROFILE_NONE 1 +#define CONFIG_SPARSEMEM 1 +#define CONFIG_BASE_FULL 1 +#define CONFIG_PREVENT_FIRMWARE_BUILD 1 +#define CONFIG_SYSFS_DEPRECATED_V2 1 +#define CONFIG_CPU_SUP_CENTAUR 1 +#define CONFIG_GENERIC_CMOS_UPDATE 1 +#define CONFIG_GENERIC_CALIBRATE_DELAY 1 +#define CONFIG_ARCH_HAS_CPU_IDLE_WAIT 1 +#define CONFIG_NFS_ACL_SUPPORT 1 +#define CONFIG_HAS_IOMEM 1 +#define CONFIG_XENO_OPT_PIPELINE_HEAD 1 +#define CONFIG_FW_LOADER 1 +#define CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE 0 +#define CONFIG_IO_DELAY_0X80 1 +#define CONFIG_FTRACE 1 +#define CONFIG_MODULE_FORCE_UNLOAD 1 +#define CONFIG_RTC_INTF_DEV 1 +#define CONFIG_PACKET 1 +#define CONFIG_BSD_PROCESS_ACCT_V3 1 +#define CONFIG_CONSTRUCTORS 1 +#define CONFIG_DEBUG_BUGVERBOSE 1 +#define CONFIG_FS_POSIX_ACL 1 +#define CONFIG_SERIAL_CORE_CONSOLE 1 +#define CONFIG_X86_WP_WORKS_OK 1 +#define CONFIG_GENERIC_HARDIRQS 1 +#define CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC 1 +#define CONFIG_HPET_MMAP 1 +#define CONFIG_CPU_IDLE_GOV_LADDER 1 +#define CONFIG_CRAMFS 1 +#define CONFIG_MOUSE_PS2_LOGIPS2PP 1 +#define CONFIG_BOUNCE 1 +#define CONFIG_SERIO_CT82C710 1 +#define CONFIG_MOUSE_PS2 1 +#define CONFIG_SCSI_PROC_FS 1 +#define CONFIG_RTC_CLASS 1 +#define CONFIG_CRYPTO_PCOMP 1 +#define CONFIG_HAVE_MMIOTRACE_SUPPORT 1 +#define CONFIG_INOTIFY_USER 1 +#define CONFIG_XENO_SKIN_POSIX 1 +#define CONFIG_IPIPE_DEBUG_CONTEXT 1 +#define CONFIG_X86_HT 1 +#define CONFIG_INPUT_PCSPKR 1 +#define CONFIG_TASKSTATS 1 +#define CONFIG_TRACE_IRQFLAGS_SUPPORT 1 +#define CONFIG_COMPAT 1 +#define CONFIG_POSIX_MQUEUE 1 +#define CONFIG_SND_TIMER 1 +#define CONFIG_RTC_DRV_CMOS 1 +#define CONFIG_X86 1 +#define CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT 0 +#define CONFIG_SELECT_MEMORY_MODEL 1 +#define CONFIG_SYSFS_DEPRECATED 1 +#define CONFIG_GENERIC_BUG_RELATIVE_POINTERS 1 +#define CONFIG_SPARSEMEM_MANUAL 1 +#define CONFIG_SND_MPU401 1 +#define CONFIG_SERIO_RAW 1 +#define CONFIG_XENO_OPT_NATIVE_BUFFER 1 +#define CONFIG_EDD 1 +#define CONFIG_NFS_V3 1 +#define CONFIG_NFS_V4 1 +#define CONFIG_JBD 1 +#define CONFIG_USB_ARCH_HAS_HCD 1 +#define CONFIG_DEFAULT_TCP_CONG "cubic" +#define CONFIG_BSD_PROCESS_ACCT 1 +#define CONFIG_ZONE_DMA32 1 +#define CONFIG_GENERIC_IOMAP 1 +#define CONFIG_FAT_FS 1 +#define CONFIG_CRYPTO_BLKCIPHER 1 +#define CONFIG_XENO_OPT_NATIVE_PERIOD 0 +#define CONFIG_FTRACE_NMI_ENTER 1 +#define CONFIG_HID 1 +#define CONFIG_NLATTR 1 +#define CONFIG_X86_MCE_INTEL 1 +#define CONFIG_FAT_DEFAULT_CODEPAGE 437 +#define CONFIG_ATA 1 +#define CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT 1 +#define CONFIG_HAVE_KERNEL_LZMA 1 +#define CONFIG_PRINTK 1 +#define CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING 1 +#define CONFIG_POWER_SUPPLY 1 +#define CONFIG_AIO 1 +#define CONFIG_NLS_ISO8859_15 1 +#define CONFIG_MOUSE_PS2_SYNAPTICS 1 +#define CONFIG_DMI 1 +#define CONFIG_HAVE_FUNCTION_GRAPH_TRACER 1 +#define CONFIG_SUNRPC 1 +#define CONFIG_FS_MBCACHE 1 +#define CONFIG_SERIAL_8250_NR_UARTS 32 +#define CONFIG_DETECT_HUNG_TASK 1 +#define CONFIG_HAVE_MEMORY_PRESENT 1 +#define CONFIG_XENO_TESTING_MODULE_MODULE 1 +#define CONFIG_PCI 1 +#define CONFIG_IKCONFIG_PROC 1 +#define CONFIG_HAVE_KERNEL_GZIP 1 +#define CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG 1 +#define CONFIG_X86_64_SMP 1 +#define CONFIG_BUG 1 +#define CONFIG_NFS_FS 1 +#define CONFIG_MII 1 +#define CONFIG_PROCESSOR_SELECT 1 +#define CONFIG_NETWORK_FILESYSTEMS 1 +#define CONFIG_FIX_EARLYCON_MEM 1 +#define CONFIG_CRYPTO 1 +#define CONFIG_IPIPE_DOMAINS 4 +#define CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB 1 +#define CONFIG_SYSCTL 1 +#define CONFIG_DEBUG_PREEMPT 1 +#define CONFIG_MISC_FILESYSTEMS 1 +#define CONFIG_HAVE_OPROFILE 1 +#define CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK 1 +#define CONFIG_HAVE_PERF_EVENTS 1 +#define CONFIG_STACKTRACE 1 +#define CONFIG_SND 1 +#define CONFIG_HPET_EMULATE_RTC 1 +#define CONFIG_NETDEV_1000 1 +#define CONFIG_IP_PNP 1 +#define CONFIG_ZONE_DMA_FLAG 1 +#define CONFIG_NET 1 +#define CONFIG_MODULE_UNLOAD 1 +#define CONFIG_STANDALONE 1 +#define CONFIG_EVENT_TRACING 1 +#define CONFIG_NETDEVICES 1 +#define CONFIG_USB 1 +#define CONFIG_FRAME_POINTER 1 +#define CONFIG_SND_HDA_INTEL 1 +#define CONFIG_SND_DMA_SGBUF 1 +#define CONFIG_NLS 1 +#define CONFIG_CLOCKSOURCE_WATCHDOG 1 +#define CONFIG_CRYPTO_ALGAPI 1 +#define CONFIG_SERIAL_8250 1 +#define CONFIG_PNP 1 +#define CONFIG_SND_SEQUENCER 1 +#define CONFIG_MMU 1 +#define CONFIG_PCI_LEGACY 1 +#define CONFIG_SMP 1 +#define CONFIG_DEBUG_MUTEXES 1 +#define CONFIG_DEBUG_INFO 1 +#define CONFIG_EXT3_FS_SECURITY 1 +#define CONFIG_BASE_SMALL 0 +#define CONFIG_XENO_SKIN_NATIVE 1 +#define CONFIG_AUDIT_ARCH 1 +#define CONFIG_SND_AC97_CODEC 1 +#define CONFIG_SND_ENS1371 1 +#define CONFIG_BINFMT_ELF 1 +#define CONFIG_PCI_DIRECT 1 +#define CONFIG_XENO_OPT_NATIVE_SEM 1 +#define CONFIG_BINARY_PRINTF 1 +#define CONFIG_XENO_OPT_PIPE_NRDEV 32 +#define CONFIG_SND_SUPPORT_OLD_API 1 +#define CONFIG_SERIO_I8042 1 +#define CONFIG_XENO_DRIVERS_TIMERBENCH 1 +#define CONFIG_XENO_OPT_REGISTRY_NRSLOTS 512 +#define CONFIG_IPIPE_DEBUG_INTERNAL 1 +#define CONFIG_SND_PCM_OSS_PLUGINS 1 +#define CONFIG_PCIEAER 1 +#define CONFIG_HAVE_MLOCK 1 +#define CONFIG_X86_MCE 1 +#define CONFIG_XENO_OPT_SEM_HEAPSZ 12 +#define CONFIG_BITREVERSE 1 +#define CONFIG_LOG_BUF_SHIFT 16 +#define CONFIG_IA32_EMULATION 1 +#define CONFIG_ARCH_POPULATES_NODE_MAP 1 +#define CONFIG_NFS_V3_ACL 1 +#define CONFIG_DUMMY_CONSOLE 1 diff --git a/include/linux/bounds.h b/include/linux/bounds.h new file mode 100644 index 0000000..2ca0fee --- /dev/null +++ b/include/linux/bounds.h @@ -0,0 +1,13 @@ +#ifndef __LINUX_BOUNDS_H__ +#define __LINUX_BOUNDS_H__ +/* + * DO NOT MODIFY. + * + * This file was generated by Kbuild + * + */ + +#define NR_PAGEFLAGS 24 /* __NR_PAGEFLAGS # */ +#define MAX_NR_ZONES 4 /* __MAX_NR_ZONES # */ + +#endif diff --git a/include/linux/compile.h b/include/linux/compile.h new file mode 100644 index 0000000..70f55aa --- /dev/null +++ b/include/linux/compile.h @@ -0,0 +1,9 @@ +/* This file is auto generated, version 92 */ +/* SMP PREEMPT */ +#define UTS_MACHINE "x86_64" +#define UTS_VERSION "#92 SMP PREEMPT Thu Feb 4 10:29:12 CET 2010" +#define LINUX_COMPILE_TIME "10:29:12" +#define LINUX_COMPILE_BY "rpm" +#define LINUX_COMPILE_HOST "redshift" +#define LINUX_COMPILE_DOMAIN "xenomai.org" +#define LINUX_COMPILER "gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9) " diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index d5b3876..010aa8b 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -207,24 +207,28 @@ extern void irq_enter(void); */ extern void irq_exit(void); -#define nmi_enter() \ - do { \ - ftrace_nmi_enter(); \ - BUG_ON(in_nmi()); \ - add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ - lockdep_off(); \ - rcu_nmi_enter(); \ - trace_hardirq_enter(); \ +#define nmi_enter() \ + do { \ + if (likely(!ipipe_test_foreign_stack())) { \ + ftrace_nmi_enter(); \ + BUG_ON(in_nmi()); \ + add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ + lockdep_off(); \ + rcu_nmi_enter(); \ + trace_hardirq_enter(); \ + } \ } while (0) -#define nmi_exit() \ - do { \ - trace_hardirq_exit(); \ - rcu_nmi_exit(); \ - lockdep_on(); \ - BUG_ON(!in_nmi()); \ - sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ - ftrace_nmi_exit(); \ +#define nmi_exit() \ + do { \ + if (likely(!ipipe_test_foreign_stack())) { \ + trace_hardirq_exit(); \ + rcu_nmi_exit(); \ + lockdep_on(); \ + BUG_ON(!in_nmi()); \ + sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ + ftrace_nmi_exit(); \ + } \ } while (0) #endif /* LINUX_HARDIRQ_H */ diff --git a/include/linux/ipipe.h b/include/linux/ipipe.h new file mode 100644 index 0000000..59da296 --- /dev/null +++ b/include/linux/ipipe.h @@ -0,0 +1,684 @@ +/* -*- linux-c -*- + * include/linux/ipipe.h + * + * Copyright (C) 2002-2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_H +#define __LINUX_IPIPE_H + +#include <linux/spinlock.h> +#include <linux/cache.h> +#include <linux/percpu.h> +#include <linux/mutex.h> +#include <linux/linkage.h> +#include <linux/ipipe_base.h> +#include <asm/ipipe.h> + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + +#include <linux/cpumask.h> +#include <asm/system.h> + +static inline int ipipe_disable_context_check(int cpu) +{ + return xchg(&per_cpu(ipipe_percpu_context_check, cpu), 0); +} + +static inline void ipipe_restore_context_check(int cpu, int old_state) +{ + per_cpu(ipipe_percpu_context_check, cpu) = old_state; +} + +static inline void ipipe_context_check_off(void) +{ + int cpu; + for_each_online_cpu(cpu) + per_cpu(ipipe_percpu_context_check, cpu) = 0; +} + +#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +static inline int ipipe_disable_context_check(int cpu) +{ + return 0; +} + +static inline void ipipe_restore_context_check(int cpu, int old_state) { } + +static inline void ipipe_context_check_off(void) { } + +#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +#ifdef CONFIG_IPIPE + +#define IPIPE_VERSION_STRING IPIPE_ARCH_STRING +#define IPIPE_RELEASE_NUMBER ((IPIPE_MAJOR_NUMBER << 16) | \ + (IPIPE_MINOR_NUMBER << 8) | \ + (IPIPE_PATCH_NUMBER)) + +#ifndef BROKEN_BUILTIN_RETURN_ADDRESS +#define __BUILTIN_RETURN_ADDRESS0 ((unsigned long)__builtin_return_address(0)) +#define __BUILTIN_RETURN_ADDRESS1 ((unsigned long)__builtin_return_address(1)) +#endif /* !BUILTIN_RETURN_ADDRESS */ + +#define IPIPE_ROOT_PRIO 100 +#define IPIPE_ROOT_ID 0 +#define IPIPE_ROOT_NPTDKEYS 4 /* Must be <= BITS_PER_LONG */ + +#define IPIPE_RESET_TIMER 0x1 +#define IPIPE_GRAB_TIMER 0x2 + +/* Global domain flags */ +#define IPIPE_SPRINTK_FLAG 0 /* Synchronous printk() allowed */ +#define IPIPE_AHEAD_FLAG 1 /* Domain always heads the pipeline */ + +/* Interrupt control bits */ +#define IPIPE_HANDLE_FLAG 0 +#define IPIPE_PASS_FLAG 1 +#define IPIPE_ENABLE_FLAG 2 +#define IPIPE_DYNAMIC_FLAG IPIPE_HANDLE_FLAG +#define IPIPE_STICKY_FLAG 3 +#define IPIPE_SYSTEM_FLAG 4 +#define IPIPE_LOCK_FLAG 5 +#define IPIPE_WIRED_FLAG 6 +#define IPIPE_EXCLUSIVE_FLAG 7 + +#define IPIPE_HANDLE_MASK (1 << IPIPE_HANDLE_FLAG) +#define IPIPE_PASS_MASK (1 << IPIPE_PASS_FLAG) +#define IPIPE_ENABLE_MASK (1 << IPIPE_ENABLE_FLAG) +#define IPIPE_DYNAMIC_MASK IPIPE_HANDLE_MASK +#define IPIPE_STICKY_MASK (1 << IPIPE_STICKY_FLAG) +#define IPIPE_SYSTEM_MASK (1 << IPIPE_SYSTEM_FLAG) +#define IPIPE_LOCK_MASK (1 << IPIPE_LOCK_FLAG) +#define IPIPE_WIRED_MASK (1 << IPIPE_WIRED_FLAG) +#define IPIPE_EXCLUSIVE_MASK (1 << IPIPE_EXCLUSIVE_FLAG) + +#define IPIPE_DEFAULT_MASK (IPIPE_HANDLE_MASK|IPIPE_PASS_MASK) +#define IPIPE_STDROOT_MASK (IPIPE_HANDLE_MASK|IPIPE_PASS_MASK|IPIPE_SYSTEM_MASK) + +#define IPIPE_EVENT_SELF 0x80000000 + +#define IPIPE_NR_CPUS NR_CPUS + +/* This accessor assumes hw IRQs are off on SMP; allows assignment. */ +#define __ipipe_current_domain __ipipe_get_cpu_var(ipipe_percpu_domain) +/* This read-only accessor makes sure that hw IRQs are off on SMP. */ +#define ipipe_current_domain \ + ({ \ + struct ipipe_domain *__ipd__; \ + unsigned long __flags__; \ + local_irq_save_hw_smp(__flags__); \ + __ipd__ = __ipipe_current_domain; \ + local_irq_restore_hw_smp(__flags__); \ + __ipd__; \ + }) + +#define ipipe_virtual_irq_p(irq) ((irq) >= IPIPE_VIRQ_BASE && \ + (irq) < IPIPE_NR_IRQS) + +#define IPIPE_SAME_HANDLER ((ipipe_irq_handler_t)(-1)) + +struct irq_desc; + +typedef void (*ipipe_irq_ackfn_t)(unsigned irq, struct irq_desc *desc); + +typedef int (*ipipe_event_handler_t)(unsigned event, + struct ipipe_domain *from, + void *data); +struct ipipe_domain { + + int slot; /* Slot number in percpu domain data array. */ + struct list_head p_link; /* Link in pipeline */ + ipipe_event_handler_t evhand[IPIPE_NR_EVENTS]; /* Event handlers. */ + unsigned long long evself; /* Self-monitored event bits. */ + + struct ipipe_irqdesc { + unsigned long control; + ipipe_irq_ackfn_t acknowledge; + ipipe_irq_handler_t handler; + void *cookie; + } ____cacheline_aligned irqs[IPIPE_NR_IRQS]; + + int priority; + void *pdd; + unsigned long flags; + unsigned domid; + const char *name; + struct mutex mutex; +}; + +#define IPIPE_HEAD_PRIORITY (-1) /* For domains always heading the pipeline */ + +struct ipipe_domain_attr { + + unsigned domid; /* Domain identifier -- Magic value set by caller */ + const char *name; /* Domain name -- Warning: won't be dup'ed! */ + int priority; /* Priority in interrupt pipeline */ + void (*entry) (void); /* Domain entry point */ + void *pdd; /* Per-domain (opaque) data pointer */ +}; + +#define __ipipe_irq_cookie(ipd, irq) (ipd)->irqs[irq].cookie +#define __ipipe_irq_handler(ipd, irq) (ipd)->irqs[irq].handler +#define __ipipe_cpudata_irq_hits(ipd, cpu, irq) ipipe_percpudom(ipd, irqall, cpu)[irq] + +extern unsigned __ipipe_printk_virq; + +extern unsigned long __ipipe_virtual_irq_map; + +extern struct list_head __ipipe_pipeline; + +extern int __ipipe_event_monitors[]; + +/* Private interface */ + +void ipipe_init(void); + +#ifdef CONFIG_PROC_FS +void ipipe_init_proc(void); + +#ifdef CONFIG_IPIPE_TRACE +void __ipipe_init_tracer(void); +#else /* !CONFIG_IPIPE_TRACE */ +#define __ipipe_init_tracer() do { } while(0) +#endif /* CONFIG_IPIPE_TRACE */ + +#else /* !CONFIG_PROC_FS */ +#define ipipe_init_proc() do { } while(0) +#endif /* CONFIG_PROC_FS */ + +void __ipipe_init_stage(struct ipipe_domain *ipd); + +void __ipipe_cleanup_domain(struct ipipe_domain *ipd); + +void __ipipe_add_domain_proc(struct ipipe_domain *ipd); + +void __ipipe_remove_domain_proc(struct ipipe_domain *ipd); + +void __ipipe_flush_printk(unsigned irq, void *cookie); + +void __ipipe_walk_pipeline(struct list_head *pos); + +void __ipipe_pend_irq(unsigned irq, struct list_head *head); + +int __ipipe_dispatch_event(unsigned event, void *data); + +void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq); + +void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq); + +void __ipipe_sync_stage(int dovirt); + +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq); + +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq); + +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq); + +void __ipipe_pin_range_globally(unsigned long start, unsigned long end); + +/* Must be called hw IRQs off. */ +static inline void ipipe_irq_lock(unsigned irq) +{ + __ipipe_lock_irq(__ipipe_current_domain, ipipe_processor_id(), irq); +} + +/* Must be called hw IRQs off. */ +static inline void ipipe_irq_unlock(unsigned irq) +{ + __ipipe_unlock_irq(__ipipe_current_domain, irq); +} + +#ifndef __ipipe_sync_pipeline +#define __ipipe_sync_pipeline(dovirt) __ipipe_sync_stage(dovirt) +#endif + +#ifndef __ipipe_run_irqtail +#define __ipipe_run_irqtail() do { } while(0) +#endif + +#define __ipipe_pipeline_head_p(ipd) (&(ipd)->p_link == __ipipe_pipeline.next) + +#define __ipipe_ipending_p(p) ((p)->irqpend_himap != 0) + +/* + * Keep the following as a macro, so that client code could check for + * the support of the invariant pipeline head optimization. + */ +#define __ipipe_pipeline_head() \ + list_entry(__ipipe_pipeline.next, struct ipipe_domain, p_link) + +#define local_irq_enable_hw_cond() local_irq_enable_hw() +#define local_irq_disable_hw_cond() local_irq_disable_hw() +#define local_irq_save_hw_cond(flags) local_irq_save_hw(flags) +#define local_irq_restore_hw_cond(flags) local_irq_restore_hw(flags) + +#ifdef CONFIG_SMP +cpumask_t __ipipe_set_irq_affinity(unsigned irq, cpumask_t cpumask); +int __ipipe_send_ipi(unsigned ipi, cpumask_t cpumask); +#define local_irq_save_hw_smp(flags) local_irq_save_hw(flags) +#define local_irq_restore_hw_smp(flags) local_irq_restore_hw(flags) +#else /* !CONFIG_SMP */ +#define local_irq_save_hw_smp(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_smp(flags) do { } while(0) +#endif /* CONFIG_SMP */ + +#define local_irq_save_full(vflags, rflags) \ + do { \ + local_irq_save(vflags); \ + local_irq_save_hw(rflags); \ + } while(0) + +#define local_irq_restore_full(vflags, rflags) \ + do { \ + local_irq_restore_hw(rflags); \ + local_irq_restore(vflags); \ + } while(0) + +static inline void __local_irq_restore_nosync(unsigned long x) +{ + struct ipipe_percpu_domain_data *p = ipipe_root_cpudom_ptr(); + + if (raw_irqs_disabled_flags(x)) { + set_bit(IPIPE_STALL_FLAG, &p->status); + trace_hardirqs_off(); + } else { + trace_hardirqs_on(); + clear_bit(IPIPE_STALL_FLAG, &p->status); + } +} + +static inline void local_irq_restore_nosync(unsigned long x) +{ + unsigned long flags; + local_irq_save_hw_smp(flags); + __local_irq_restore_nosync(x); + local_irq_restore_hw_smp(flags); +} + +#define __ipipe_root_domain_p (__ipipe_current_domain == ipipe_root_domain) +#define ipipe_root_domain_p (ipipe_current_domain == ipipe_root_domain) + +static inline int __ipipe_event_monitored_p(int ev) +{ + if (__ipipe_event_monitors[ev] > 0) + return 1; + + return (ipipe_current_domain->evself & (1LL << ev)) != 0; +} + +#define ipipe_sigwake_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SIGWAKE)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SIGWAKE, p); \ +} while(0) + +#define ipipe_exit_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_EXIT)) \ + __ipipe_dispatch_event(IPIPE_EVENT_EXIT, p); \ +} while(0) + +#define ipipe_setsched_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SETSCHED)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SETSCHED, p); \ +} while(0) + +#define ipipe_schedule_notify(prev, next) \ +do { \ + if ((((prev)->flags|(next)->flags) & PF_EVNOTIFY) && \ + __ipipe_event_monitored_p(IPIPE_EVENT_SCHEDULE)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SCHEDULE,next); \ +} while(0) + +#define ipipe_trap_notify(ex, regs) \ +({ \ + unsigned long __flags__; \ + int __ret__ = 0; \ + local_irq_save_hw_smp(__flags__); \ + if ((test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)) || \ + ((current)->flags & PF_EVNOTIFY)) && \ + __ipipe_event_monitored_p(ex)) { \ + local_irq_restore_hw_smp(__flags__); \ + __ret__ = __ipipe_dispatch_event(ex, regs); \ + } else \ + local_irq_restore_hw_smp(__flags__); \ + __ret__; \ +}) + +static inline void ipipe_init_notify(struct task_struct *p) +{ + if (__ipipe_event_monitored_p(IPIPE_EVENT_INIT)) + __ipipe_dispatch_event(IPIPE_EVENT_INIT, p); +} + +struct mm_struct; + +static inline void ipipe_cleanup_notify(struct mm_struct *mm) +{ + if (__ipipe_event_monitored_p(IPIPE_EVENT_CLEANUP)) + __ipipe_dispatch_event(IPIPE_EVENT_CLEANUP, mm); +} + +/* Public interface */ + +int ipipe_register_domain(struct ipipe_domain *ipd, + struct ipipe_domain_attr *attr); + +int ipipe_unregister_domain(struct ipipe_domain *ipd); + +void ipipe_suspend_domain(void); + +int ipipe_virtualize_irq(struct ipipe_domain *ipd, + unsigned irq, + ipipe_irq_handler_t handler, + void *cookie, + ipipe_irq_ackfn_t acknowledge, + unsigned modemask); + +int ipipe_control_irq(unsigned irq, + unsigned clrmask, + unsigned setmask); + +unsigned ipipe_alloc_virq(void); + +int ipipe_free_virq(unsigned virq); + +int ipipe_trigger_irq(unsigned irq); + +static inline void __ipipe_propagate_irq(unsigned irq) +{ + struct list_head *next = __ipipe_current_domain->p_link.next; + if (next == &ipipe_root.p_link) { + /* Fast path: root must handle all interrupts. */ + __ipipe_set_irq_pending(&ipipe_root, irq); + return; + } + __ipipe_pend_irq(irq, next); +} + +static inline void __ipipe_schedule_irq(unsigned irq) +{ + __ipipe_pend_irq(irq, &__ipipe_current_domain->p_link); +} + +static inline void __ipipe_schedule_irq_head(unsigned irq) +{ + __ipipe_set_irq_pending(__ipipe_pipeline_head(), irq); +} + +static inline void __ipipe_schedule_irq_root(unsigned irq) +{ + __ipipe_set_irq_pending(&ipipe_root, irq); +} + +static inline void ipipe_propagate_irq(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_propagate_irq(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq_head(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq_head(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq_root(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq_root(irq); + local_irq_restore_hw(flags); +} + +void ipipe_stall_pipeline_from(struct ipipe_domain *ipd); + +unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd); + +unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd); + +static inline void ipipe_unstall_pipeline_from(struct ipipe_domain *ipd) +{ + ipipe_test_and_unstall_pipeline_from(ipd); +} + +void ipipe_restore_pipeline_from(struct ipipe_domain *ipd, + unsigned long x); + +static inline unsigned long ipipe_test_pipeline_from(struct ipipe_domain *ipd) +{ + return test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline void ipipe_stall_pipeline_head(void) +{ + local_irq_disable_hw(); + __set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status)); +} + +static inline unsigned long ipipe_test_and_stall_pipeline_head(void) +{ + local_irq_disable_hw(); + return __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status)); +} + +void ipipe_unstall_pipeline_head(void); + +void __ipipe_restore_pipeline_head(unsigned long x); + +static inline void ipipe_restore_pipeline_head(unsigned long x) +{ + /* On some archs, __test_and_set_bit() might return different + * truth value than test_bit(), so we test the exclusive OR of + * both statuses, assuming that the lowest bit is always set in + * the truth value (if this is wrong, the failed optimization will + * be caught in __ipipe_restore_pipeline_head() if + * CONFIG_DEBUG_KERNEL is set). */ + if ((x ^ test_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status))) & 1) + __ipipe_restore_pipeline_head(x); +} + +#define ipipe_unstall_pipeline() \ + ipipe_unstall_pipeline_from(ipipe_current_domain) + +#define ipipe_test_and_unstall_pipeline() \ + ipipe_test_and_unstall_pipeline_from(ipipe_current_domain) + +#define ipipe_test_pipeline() \ + ipipe_test_pipeline_from(ipipe_current_domain) + +#define ipipe_test_and_stall_pipeline() \ + ipipe_test_and_stall_pipeline_from(ipipe_current_domain) + +#define ipipe_stall_pipeline() \ + ipipe_stall_pipeline_from(ipipe_current_domain) + +#define ipipe_restore_pipeline(x) \ + ipipe_restore_pipeline_from(ipipe_current_domain, (x)) + +void ipipe_init_attr(struct ipipe_domain_attr *attr); + +int ipipe_get_sysinfo(struct ipipe_sysinfo *sysinfo); + +unsigned long ipipe_critical_enter(void (*syncfn) (void)); + +void ipipe_critical_exit(unsigned long flags); + +static inline void ipipe_set_printk_sync(struct ipipe_domain *ipd) +{ + set_bit(IPIPE_SPRINTK_FLAG, &ipd->flags); +} + +static inline void ipipe_set_printk_async(struct ipipe_domain *ipd) +{ + clear_bit(IPIPE_SPRINTK_FLAG, &ipd->flags); +} + +static inline void ipipe_set_foreign_stack(struct ipipe_domain *ipd) +{ + /* Must be called hw interrupts off. */ + __set_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline void ipipe_clear_foreign_stack(struct ipipe_domain *ipd) +{ + /* Must be called hw interrupts off. */ + __clear_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline int ipipe_test_foreign_stack(void) +{ + /* Must be called hw interrupts off. */ + return test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)); +} + +#ifndef ipipe_safe_current +#define ipipe_safe_current() \ +({ \ + struct task_struct *p; \ + unsigned long flags; \ + local_irq_save_hw_smp(flags); \ + p = ipipe_test_foreign_stack() ? &init_task : current; \ + local_irq_restore_hw_smp(flags); \ + p; \ +}) +#endif + +ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd, + unsigned event, + ipipe_event_handler_t handler); + +cpumask_t ipipe_set_irq_affinity(unsigned irq, + cpumask_t cpumask); + +int ipipe_send_ipi(unsigned ipi, + cpumask_t cpumask); + +int ipipe_setscheduler_root(struct task_struct *p, + int policy, + int prio); + +int ipipe_reenter_root(struct task_struct *prev, + int policy, + int prio); + +int ipipe_alloc_ptdkey(void); + +int ipipe_free_ptdkey(int key); + +int ipipe_set_ptd(int key, + void *value); + +void *ipipe_get_ptd(int key); + +int ipipe_disable_ondemand_mappings(struct task_struct *tsk); + +static inline void ipipe_nmi_enter(void) +{ + int cpu = ipipe_processor_id(); + + per_cpu(ipipe_nmi_saved_root, cpu) = ipipe_root_cpudom_var(status); + __set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + per_cpu(ipipe_saved_context_check_state, cpu) = + ipipe_disable_context_check(cpu); +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ +} + +static inline void ipipe_nmi_exit(void) +{ + int cpu = ipipe_processor_id(); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + ipipe_restore_context_check + (cpu, per_cpu(ipipe_saved_context_check_state, cpu)); +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ + + if (!test_bit(IPIPE_STALL_FLAG, &per_cpu(ipipe_nmi_saved_root, cpu))) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); +} + +#else /* !CONFIG_IPIPE */ + +#define ipipe_init() do { } while(0) +#define ipipe_suspend_domain() do { } while(0) +#define ipipe_sigwake_notify(p) do { } while(0) +#define ipipe_setsched_notify(p) do { } while(0) +#define ipipe_init_notify(p) do { } while(0) +#define ipipe_exit_notify(p) do { } while(0) +#define ipipe_cleanup_notify(mm) do { } while(0) +#define ipipe_trap_notify(t,r) 0 +#define ipipe_init_proc() do { } while(0) + +static inline void __ipipe_pin_range_globally(unsigned long start, + unsigned long end) +{ +} + +static inline int ipipe_test_foreign_stack(void) +{ + return 0; +} + +#define local_irq_enable_hw_cond() do { } while(0) +#define local_irq_disable_hw_cond() do { } while(0) +#define local_irq_save_hw_cond(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_cond(flags) do { } while(0) +#define local_irq_save_hw_smp(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_smp(flags) do { } while(0) + +#define ipipe_irq_lock(irq) do { } while(0) +#define ipipe_irq_unlock(irq) do { } while(0) + +#define __ipipe_root_domain_p 1 +#define ipipe_root_domain_p 1 +#define ipipe_safe_current current +#define ipipe_processor_id() smp_processor_id() + +#define ipipe_nmi_enter() do { } while (0) +#define ipipe_nmi_exit() do { } while (0) + +#define local_irq_disable_head() local_irq_disable() + +#define local_irq_save_full(vflags, rflags) do { (void)(vflags); local_irq_save(rflags); } while(0) +#define local_irq_restore_full(vflags, rflags) do { (void)(vflags); local_irq_restore(rflags); } while(0) +#define local_irq_restore_nosync(vflags) local_irq_restore(vflags) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_H */ diff --git a/include/linux/ipipe_base.h b/include/linux/ipipe_base.h new file mode 100644 index 0000000..2b0705d --- /dev/null +++ b/include/linux/ipipe_base.h @@ -0,0 +1,133 @@ +/* -*- linux-c -*- + * include/linux/ipipe_base.h + * + * Copyright (C) 2002-2007 Philippe Gerum. + * 2007 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_BASE_H +#define __LINUX_IPIPE_BASE_H + +#ifdef CONFIG_IPIPE + +#include <asm/ipipe_base.h> +#include <asm/bitsperlong.h> + +#define __bpl_up(x) (((x)+(BITS_PER_LONG-1)) & ~(BITS_PER_LONG-1)) +/* Number of virtual IRQs (must be a multiple of BITS_PER_LONG) */ +#define IPIPE_NR_VIRQS BITS_PER_LONG +/* First virtual IRQ # (must be aligned on BITS_PER_LONG) */ +#define IPIPE_VIRQ_BASE __bpl_up(IPIPE_NR_XIRQS) +/* Total number of IRQ slots */ +#define IPIPE_NR_IRQS (IPIPE_VIRQ_BASE+IPIPE_NR_VIRQS) + +#define IPIPE_IRQ_LOMAPSZ (IPIPE_NR_IRQS / BITS_PER_LONG) +#if IPIPE_IRQ_LOMAPSZ > BITS_PER_LONG +/* + * We need a 3-level mapping. This allows us to handle up to 32k IRQ + * vectors on 32bit machines, 256k on 64bit ones. + */ +#define __IPIPE_3LEVEL_IRQMAP 1 +#define IPIPE_IRQ_MDMAPSZ (__bpl_up(IPIPE_IRQ_LOMAPSZ) / BITS_PER_LONG) +#else +/* + * 2-level mapping is enough. This allows us to handle up to 1024 IRQ + * vectors on 32bit machines, 4096 on 64bit ones. + */ +#define __IPIPE_2LEVEL_IRQMAP 1 +#endif + +#define IPIPE_IRQ_DOALL 0 +#define IPIPE_IRQ_DOVIRT 1 + +/* Per-cpu pipeline status */ +#define IPIPE_STALL_FLAG 0 /* Stalls a pipeline stage -- guaranteed at bit #0 */ +#define IPIPE_SYNC_FLAG 1 /* The interrupt syncer is running for the domain */ +#define IPIPE_NOSTACK_FLAG 2 /* Domain currently runs on a foreign stack */ + +#define IPIPE_STALL_MASK (1L << IPIPE_STALL_FLAG) +#define IPIPE_SYNC_MASK (1L << IPIPE_SYNC_FLAG) +#define IPIPE_NOSTACK_MASK (1L << IPIPE_NOSTACK_FLAG) + +typedef void (*ipipe_irq_handler_t)(unsigned int irq, + void *cookie); + +extern struct ipipe_domain ipipe_root; + +#define ipipe_root_domain (&ipipe_root) + +void __ipipe_unstall_root(void); + +void __ipipe_restore_root(unsigned long x); + +#define ipipe_preempt_disable(flags) \ + do { \ + local_irq_save_hw(flags); \ + if (__ipipe_root_domain_p) \ + preempt_disable(); \ + } while (0) + +#define ipipe_preempt_enable(flags) \ + do { \ + if (__ipipe_root_domain_p) { \ + preempt_enable_no_resched(); \ + local_irq_restore_hw(flags); \ + preempt_check_resched(); \ + } else \ + local_irq_restore_hw(flags); \ + } while (0) + +#define ipipe_get_cpu(flags) ({ ipipe_preempt_disable(flags); ipipe_processor_id(); }) +#define ipipe_put_cpu(flags) ipipe_preempt_enable(flags) + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT +void ipipe_check_context(struct ipipe_domain *border_ipd); +#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */ +static inline void ipipe_check_context(struct ipipe_domain *border_ipd) { } +#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +/* Generic features */ + +#ifdef CONFIG_GENERIC_CLOCKEVENTS +#define __IPIPE_FEATURE_REQUEST_TICKDEV 1 +#endif +#define __IPIPE_FEATURE_DELAYED_ATOMICSW 1 +#define __IPIPE_FEATURE_FASTPEND_IRQ 1 +#define __IPIPE_FEATURE_TRACE_EVENT 1 + +#else /* !CONFIG_IPIPE */ + +#define ipipe_preempt_disable(flags) \ + do { \ + preempt_disable(); \ + (void)(flags); \ + } while (0) +#define ipipe_preempt_enable(flags) preempt_enable() + +#define ipipe_get_cpu(flags) ({ (void)(flags); get_cpu(); }) +#define ipipe_put_cpu(flags) \ + do { \ + (void)(flags); \ + put_cpu(); \ + } while (0) + +#define ipipe_check_context(ipd) do { } while(0) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_BASE_H */ diff --git a/include/linux/ipipe_lock.h b/include/linux/ipipe_lock.h new file mode 100644 index 0000000..b4d0e4b --- /dev/null +++ b/include/linux/ipipe_lock.h @@ -0,0 +1,197 @@ +/* -*- linux-c -*- + * include/linux/ipipe_lock.h + * + * Copyright (C) 2009 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_LOCK_H +#define __LINUX_IPIPE_LOCK_H + +typedef struct { + arch_spinlock_t arch_lock; +} __ipipe_spinlock_t; + +#define ipipe_spinlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), __ipipe_spinlock_t *) + +#define std_spinlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), spinlock_t *) + +#define ipipe_spinlock(lock) ((__ipipe_spinlock_t *)(lock)) +#define std_spinlock(lock) ((spinlock_t *)(lock)) + +#define PICK_SPINLOCK_IRQSAVE(lock, flags) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + (flags) = __ipipe_spin_lock_irqsave(ipipe_spinlock(lock)); \ + else if (std_spinlock_p(lock)) \ + raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \ + else __bad_lock_type(); \ + } while (0) + +#define PICK_SPINTRYLOCK_IRQSAVE(lock, flags) \ + ({ \ + int __ret__; \ + if (ipipe_spinlock_p(lock)) \ + __ret__ = __ipipe_spin_trylock_irqsave(ipipe_spinlock(lock), &(flags)); \ + else if (std_spinlock_p(lock)) \ + __ret__ = raw_spin_trylock_irqsave(&std_spinlock(lock)->rlock, flags); \ + else __bad_lock_type(); \ + __ret__; \ + }) + +#define PICK_SPINUNLOCK_IRQRESTORE(lock, flags) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + __ipipe_spin_unlock_irqrestore(ipipe_spinlock(lock), flags); \ + else if (std_spinlock_p(lock)) \ + raw_spin_unlock_irqrestore(&std_spinlock(lock)->rlock, flags); \ + } while (0) + +#define PICK_SPINOP(op, lock) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + arch_spin##op(&ipipe_spinlock(lock)->arch_lock); \ + else if (std_spinlock_p(lock)) \ + raw_spin##op(&std_spinlock(lock)->rlock); \ + else __bad_lock_type(); \ + } while (0) + +#define PICK_SPINOP_RET(op, lock, type) \ + ({ \ + type __ret__; \ + if (ipipe_spinlock_p(lock)) \ + __ret__ = arch_spin##op(&ipipe_spinlock(lock)->arch_lock); \ + else if (std_spinlock_p(lock)) \ + __ret__ = raw_spin##op(&std_spinlock(lock)->rlock); \ + else { __ret__ = -1; __bad_lock_type(); } \ + __ret__; \ + }) + +#define arch_spin_lock_init(lock) \ + do { \ + IPIPE_DEFINE_SPINLOCK(__lock__); \ + *((ipipe_spinlock_t *)lock) = __lock__; \ + } while (0) + +#define arch_spin_lock_irq(lock) \ + do { \ + local_irq_disable_hw(); \ + arch_spin_lock(lock); \ + } while (0) + +#define arch_spin_unlock_irq(lock) \ + do { \ + arch_spin_unlock(lock); \ + local_irq_enable_hw(); \ + } while (0) + +typedef struct { + arch_rwlock_t arch_lock; +} __ipipe_rwlock_t; + +#define ipipe_rwlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), __ipipe_rwlock_t *) + +#define std_rwlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), rwlock_t *) + +#define ipipe_rwlock(lock) ((__ipipe_rwlock_t *)(lock)) +#define std_rwlock(lock) ((rwlock_t *)(lock)) + +#define PICK_RWOP(op, lock) \ + do { \ + if (ipipe_rwlock_p(lock)) \ + arch##op(&ipipe_rwlock(lock)->arch_lock); \ + else if (std_rwlock_p(lock)) \ + _raw##op(std_rwlock(lock)); \ + else __bad_lock_type(); \ + } while (0) + +extern int __bad_lock_type(void); + +#ifdef CONFIG_IPIPE + +#define ipipe_spinlock_t __ipipe_spinlock_t +#define IPIPE_DEFINE_SPINLOCK(x) ipipe_spinlock_t x = IPIPE_SPIN_LOCK_UNLOCKED +#define IPIPE_DECLARE_SPINLOCK(x) extern ipipe_spinlock_t x + +#define IPIPE_SPIN_LOCK_UNLOCKED \ + (__ipipe_spinlock_t) { .arch_lock = __ARCH_SPIN_LOCK_UNLOCKED } + +#define spin_lock_irqsave_cond(lock, flags) \ + spin_lock_irqsave(lock, flags) + +#define spin_unlock_irqrestore_cond(lock, flags) \ + spin_unlock_irqrestore(lock, flags) + +void __ipipe_spin_lock_irq(ipipe_spinlock_t *lock); + +void __ipipe_spin_unlock_irq(ipipe_spinlock_t *lock); + +unsigned long __ipipe_spin_lock_irqsave(ipipe_spinlock_t *lock); + +int __ipipe_spin_trylock_irqsave(ipipe_spinlock_t *lock, + unsigned long *x); + +void __ipipe_spin_unlock_irqrestore(ipipe_spinlock_t *lock, + unsigned long x); + +void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock); + +void __ipipe_spin_unlock_irqcomplete(unsigned long x); + +#define ipipe_rwlock_t __ipipe_rwlock_t +#define IPIPE_DEFINE_RWLOCK(x) ipipe_rwlock_t x = IPIPE_RW_LOCK_UNLOCKED +#define IPIPE_DECLARE_RWLOCK(x) extern ipipe_rwlock_t x + +#define IPIPE_RW_LOCK_UNLOCKED \ + (__ipipe_rwlock_t) { .arch_lock = __ARCH_RW_LOCK_UNLOCKED } + +#else /* !CONFIG_IPIPE */ + +#define ipipe_spinlock_t spinlock_t +#define IPIPE_DEFINE_SPINLOCK(x) DEFINE_SPINLOCK(x) +#define IPIPE_DECLARE_SPINLOCK(x) extern spinlock_t x +#define IPIPE_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED + +#define spin_lock_irqsave_cond(lock, flags) \ + do { \ + (void)(flags); \ + spin_lock(lock); \ + } while(0) + +#define spin_unlock_irqrestore_cond(lock, flags) \ + spin_unlock(lock) + +#define __ipipe_spin_lock_irq(lock) do { } while (0) +#define __ipipe_spin_unlock_irq(lock) do { } while (0) +#define __ipipe_spin_lock_irqsave(lock) 0 +#define __ipipe_spin_trylock_irqsave(lock, x) ({ (void)(x); 1; }) +#define __ipipe_spin_unlock_irqrestore(lock, x) do { (void)(x); } while (0) +#define __ipipe_spin_unlock_irqbegin(lock) do { } while (0) +#define __ipipe_spin_unlock_irqcomplete(x) do { (void)(x); } while (0) + +#define ipipe_rwlock_t rwlock_t +#define IPIPE_DEFINE_RWLOCK(x) DEFINE_RWLOCK(x) +#define IPIPE_DECLARE_RWLOCK(x) extern rwlock_t x +#define IPIPE_RW_LOCK_UNLOCKED RW_LOCK_UNLOCKED + +#endif /* !CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_LOCK_H */ diff --git a/include/linux/ipipe_percpu.h b/include/linux/ipipe_percpu.h new file mode 100644 index 0000000..f6727e3 --- /dev/null +++ b/include/linux/ipipe_percpu.h @@ -0,0 +1,89 @@ +/* -*- linux-c -*- + * include/linux/ipipe_percpu.h + * + * Copyright (C) 2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_PERCPU_H +#define __LINUX_IPIPE_PERCPU_H + +#include <asm/percpu.h> +#include <asm/ptrace.h> + +struct ipipe_domain; + +struct ipipe_percpu_domain_data { + unsigned long status; /* <= Must be first in struct. */ + unsigned long irqpend_himap; +#ifdef __IPIPE_3LEVEL_IRQMAP + unsigned long irqpend_mdmap[IPIPE_IRQ_MDMAPSZ]; +#endif + unsigned long irqpend_lomap[IPIPE_IRQ_LOMAPSZ]; + unsigned long irqheld_map[IPIPE_IRQ_LOMAPSZ]; + unsigned long irqall[IPIPE_NR_IRQS]; + u64 evsync; +}; + +/* + * CAREFUL: all accessors based on __raw_get_cpu_var() you may find in + * this file should be used only while hw interrupts are off, to + * prevent from CPU migration regardless of the running domain. + */ +#ifdef CONFIG_SMP +#define ipipe_percpudom_ptr(ipd, cpu) \ + (&per_cpu(ipipe_percpu_darray, cpu)[(ipd)->slot]) +#define ipipe_cpudom_ptr(ipd) \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[(ipd)->slot]) +#else +DECLARE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]); +#define ipipe_percpudom_ptr(ipd, cpu) \ + (per_cpu(ipipe_percpu_daddr, cpu)[(ipd)->slot]) +#define ipipe_cpudom_ptr(ipd) \ + (__ipipe_get_cpu_var(ipipe_percpu_daddr)[(ipd)->slot]) +#endif +#define ipipe_percpudom(ipd, var, cpu) (ipipe_percpudom_ptr(ipd, cpu)->var) +#define ipipe_cpudom_var(ipd, var) (ipipe_cpudom_ptr(ipd)->var) + +#define IPIPE_ROOT_SLOT 0 +#define IPIPE_HEAD_SLOT (CONFIG_IPIPE_DOMAINS - 1) + +DECLARE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]); + +DECLARE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain); + +DECLARE_PER_CPU(unsigned long, ipipe_nmi_saved_root); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT +DECLARE_PER_CPU(int, ipipe_percpu_context_check); +DECLARE_PER_CPU(int, ipipe_saved_context_check_state); +#endif + +#define ipipe_root_cpudom_ptr(var) \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT]) + +#define ipipe_root_cpudom_var(var) ipipe_root_cpudom_ptr()->var + +#define ipipe_this_cpudom_var(var) \ + ipipe_cpudom_var(__ipipe_current_domain, var) + +#define ipipe_head_cpudom_ptr() \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_HEAD_SLOT]) + +#define ipipe_head_cpudom_var(var) ipipe_head_cpudom_ptr()->var + +#endif /* !__LINUX_IPIPE_PERCPU_H */ diff --git a/include/linux/ipipe_tickdev.h b/include/linux/ipipe_tickdev.h new file mode 100644 index 0000000..4a1cb1b --- /dev/null +++ b/include/linux/ipipe_tickdev.h @@ -0,0 +1,58 @@ +/* -*- linux-c -*- + * include/linux/ipipe_tickdev.h + * + * Copyright (C) 2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_TICKDEV_H +#define __LINUX_IPIPE_TICKDEV_H + +#if defined(CONFIG_IPIPE) && defined(CONFIG_GENERIC_CLOCKEVENTS) + +#include <linux/clockchips.h> + +struct tick_device; + +struct ipipe_tick_device { + + void (*emul_set_mode)(enum clock_event_mode, + struct clock_event_device *cdev); + int (*emul_set_tick)(unsigned long delta, + struct clock_event_device *cdev); + void (*real_set_mode)(enum clock_event_mode mode, + struct clock_event_device *cdev); + int (*real_set_tick)(unsigned long delta, + struct clock_event_device *cdev); + struct tick_device *slave; + unsigned long real_max_delta_ns; + unsigned long real_mult; + int real_shift; +}; + +int ipipe_request_tickdev(const char *devname, + void (*emumode)(enum clock_event_mode mode, + struct clock_event_device *cdev), + int (*emutick)(unsigned long evt, + struct clock_event_device *cdev), + int cpu, unsigned long *tmfreq); + +void ipipe_release_tickdev(int cpu); + +#endif /* CONFIG_IPIPE && CONFIG_GENERIC_CLOCKEVENTS */ + +#endif /* !__LINUX_IPIPE_TICKDEV_H */ diff --git a/include/linux/ipipe_trace.h b/include/linux/ipipe_trace.h new file mode 100644 index 0000000..627b354 --- /dev/null +++ b/include/linux/ipipe_trace.h @@ -0,0 +1,72 @@ +/* -*- linux-c -*- + * include/linux/ipipe_trace.h + * + * Copyright (C) 2005 Luotao Fu. + * 2005-2007 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef _LINUX_IPIPE_TRACE_H +#define _LINUX_IPIPE_TRACE_H + +#ifdef CONFIG_IPIPE_TRACE + +#include <linux/types.h> + +void ipipe_trace_begin(unsigned long v); +void ipipe_trace_end(unsigned long v); +void ipipe_trace_freeze(unsigned long v); +void ipipe_trace_special(unsigned char special_id, unsigned long v); +void ipipe_trace_pid(pid_t pid, short prio); +void ipipe_trace_event(unsigned char id, unsigned long delay_tsc); +int ipipe_trace_max_reset(void); +int ipipe_trace_frozen_reset(void); + +#else /* !CONFIG_IPIPE_TRACE */ + +#define ipipe_trace_begin(v) do { (void)(v); } while(0) +#define ipipe_trace_end(v) do { (void)(v); } while(0) +#define ipipe_trace_freeze(v) do { (void)(v); } while(0) +#define ipipe_trace_special(id, v) do { (void)(id); (void)(v); } while(0) +#define ipipe_trace_pid(pid, prio) do { (void)(pid); (void)(prio); } while(0) +#define ipipe_trace_event(id, delay_tsc) do { (void)(id); (void)(delay_tsc); } while(0) +#define ipipe_trace_max_reset() do { } while(0) +#define ipipe_trace_froze_reset() do { } while(0) + +#endif /* !CONFIG_IPIPE_TRACE */ + +#ifdef CONFIG_IPIPE_TRACE_PANIC +void ipipe_trace_panic_freeze(void); +void ipipe_trace_panic_dump(void); +#else +static inline void ipipe_trace_panic_freeze(void) { } +static inline void ipipe_trace_panic_dump(void) { } +#endif + +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF +#define ipipe_trace_irq_entry(irq) ipipe_trace_begin(irq) +#define ipipe_trace_irq_exit(irq) ipipe_trace_end(irq) +#define ipipe_trace_irqsoff() ipipe_trace_begin(0x80000000UL) +#define ipipe_trace_irqson() ipipe_trace_end(0x80000000UL) +#else +#define ipipe_trace_irq_entry(irq) do { (void)(irq);} while(0) +#define ipipe_trace_irq_exit(irq) do { (void)(irq);} while(0) +#define ipipe_trace_irqsoff() do { } while(0) +#define ipipe_trace_irqson() do { } while(0) +#endif + +#endif /* !__LINUX_IPIPE_TRACE_H */ diff --git a/include/linux/irq.h b/include/linux/irq.h index 451481c..3dbfa17 100644 --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -124,6 +124,9 @@ struct irq_chip { void (*end)(unsigned int irq); int (*set_affinity)(unsigned int irq, const struct cpumask *dest); +#ifdef CONFIG_IPIPE + void (*move)(unsigned int irq); +#endif /* CONFIG_IPIPE */ int (*retrigger)(unsigned int irq); int (*set_type)(unsigned int irq, unsigned int flow_type); int (*set_wake)(unsigned int irq, unsigned int on); @@ -173,6 +176,12 @@ struct irq_2_iommu; * @name: flow handler name for /proc/interrupts output */ struct irq_desc { +#ifdef CONFIG_IPIPE + void (*ipipe_ack)(unsigned int irq, + struct irq_desc *desc); + void (*ipipe_end)(unsigned int irq, + struct irq_desc *desc); +#endif /* CONFIG_IPIPE */ unsigned int irq; struct timer_rand_state *timer_rand_state; unsigned int *kstat_irqs; @@ -346,6 +355,10 @@ extern void set_irq_chip_and_handler_name(unsigned int irq, struct irq_chip *chip, irq_flow_handler_t handle, const char *name); +extern irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, + int is_chained); + extern void __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, const char *name); @@ -357,6 +370,7 @@ static inline void __set_irq_handler_unlocked(int irq, struct irq_desc *desc; desc = irq_to_desc(irq); + handler = __fixup_irq_handler(desc, handler, 0); desc->handle_irq = handler; } diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 328bca6..4ed869d 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -14,6 +14,7 @@ #include <linux/compiler.h> #include <linux/bitops.h> #include <linux/log2.h> +#include <linux/ipipe_base.h> #include <linux/typecheck.h> #include <linux/dynamic_debug.h> #include <asm/byteorder.h> @@ -118,9 +119,12 @@ struct user; #ifdef CONFIG_PREEMPT_VOLUNTARY extern int _cond_resched(void); -# define might_resched() _cond_resched() +# define might_resched() do { \ + ipipe_check_context(ipipe_root_domain); \ + _cond_resched(); \ + } while (0) #else -# define might_resched() do { } while (0) +# define might_resched() ipipe_check_context(ipipe_root_domain) #endif #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 9ccf0e2..f500f1e 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -395,7 +395,7 @@ do { \ #endif /* CONFIG_LOCK_STAT */ -#ifdef CONFIG_LOCKDEP +#if defined (CONFIG_LOCKDEP) || defined(CONFIG_IPIPE) /* * On lockdep we dont want the hand-coded irq-enable of diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 2e681d9..130b7d5 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -9,13 +9,20 @@ #include <linux/thread_info.h> #include <linux/linkage.h> #include <linux/list.h> +#include <linux/ipipe_base.h> #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER) extern void add_preempt_count(int val); extern void sub_preempt_count(int val); #else -# define add_preempt_count(val) do { preempt_count() += (val); } while (0) -# define sub_preempt_count(val) do { preempt_count() -= (val); } while (0) +# define add_preempt_count(val) do { \ + ipipe_check_context(ipipe_root_domain); \ + preempt_count() += (val); \ + } while (0) +# define sub_preempt_count(val) do { \ + ipipe_check_context(ipipe_root_domain); \ + preempt_count() -= (val); \ + } while (0) #endif #define inc_preempt_count() add_preempt_count(1) diff --git a/include/linux/resource.h b/include/linux/resource.h index f1e914e..bc3dfa0 100644 --- a/include/linux/resource.h +++ b/include/linux/resource.h @@ -51,12 +51,6 @@ struct rlimit { #define PRIO_USER 2 /* - * Limit the stack by to some sane default: root can always - * increase this limit if needed.. 8MB seems reasonable. - */ -#define _STK_LIM (8*1024*1024) - -/* * GPG2 wants 64kB of mlocked memory, to make sure pass phrases * and other sensitive information are never written to disk. */ diff --git a/include/linux/rwlock.h b/include/linux/rwlock.h index 71e0b00..8119f0c 100644 --- a/include/linux/rwlock.h +++ b/include/linux/rwlock.h @@ -61,8 +61,8 @@ do { \ #define read_trylock(lock) __cond_lock(lock, _raw_read_trylock(lock)) #define write_trylock(lock) __cond_lock(lock, _raw_write_trylock(lock)) -#define write_lock(lock) _raw_write_lock(lock) -#define read_lock(lock) _raw_read_lock(lock) +#define write_lock(lock) PICK_RWOP(_write_lock, lock) +#define read_lock(lock) PICK_RWOP(_read_lock, lock) #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) @@ -96,8 +96,8 @@ do { \ #define read_lock_bh(lock) _raw_read_lock_bh(lock) #define write_lock_irq(lock) _raw_write_lock_irq(lock) #define write_lock_bh(lock) _raw_write_lock_bh(lock) -#define read_unlock(lock) _raw_read_unlock(lock) -#define write_unlock(lock) _raw_write_unlock(lock) +#define read_unlock(lock) PICK_RWOP(_read_unlock, lock) +#define write_unlock(lock) PICK_RWOP(_write_unlock, lock) #define read_unlock_irq(lock) _raw_read_unlock_irq(lock) #define write_unlock_irq(lock) _raw_write_unlock_irq(lock) diff --git a/include/linux/rwlock_api_smp.h b/include/linux/rwlock_api_smp.h index 9c9f049..62c8941 100644 --- a/include/linux/rwlock_api_smp.h +++ b/include/linux/rwlock_api_smp.h @@ -141,7 +141,9 @@ static inline int __raw_write_trylock(rwlock_t *lock) * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) static inline void __raw_read_lock(rwlock_t *lock) { diff --git a/include/linux/sched.h b/include/linux/sched.h index 78efe7c..c2eb9a4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -61,6 +61,7 @@ struct sched_param { #include <linux/errno.h> #include <linux/nodemask.h> #include <linux/mm_types.h> +#include <linux/ipipe.h> #include <asm/system.h> #include <asm/page.h> @@ -192,9 +193,17 @@ print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) #define TASK_DEAD 64 #define TASK_WAKEKILL 128 #define TASK_WAKING 256 +#ifdef CONFIG_IPIPE +#define TASK_ATOMICSWITCH 512 +#define TASK_NOWAKEUP 1024 +#define TASK_STATE_MAX 2048 +#define TASK_STATE_TO_CHAR_STR "RSDTtZXxKWAN" +#else /* !CONFIG_IPIPE */ +#define TASK_ATOMICSWITCH 0 +#define TASK_NOWAKEUP 0 #define TASK_STATE_MAX 512 - #define TASK_STATE_TO_CHAR_STR "RSDTtZXxKW" +#endif /* CONFIG_IPIPE */ extern char ___assert_task_state[1 - 2*!!( sizeof(TASK_STATE_TO_CHAR_STR)-1 != ilog2(TASK_STATE_MAX)+1)]; @@ -305,6 +314,15 @@ extern void trap_init(void); extern void update_process_times(int user); extern void scheduler_tick(void); +#ifdef CONFIG_IPIPE +void update_root_process_times(struct pt_regs *regs); +#else /* !CONFIG_IPIPE */ +static inline void update_root_process_times(struct pt_regs *regs) +{ + update_process_times(user_mode(regs)); +} +#endif /* CONFIG_IPIPE */ + extern void sched_show_task(struct task_struct *p); #ifdef CONFIG_DETECT_SOFTLOCKUP @@ -356,7 +374,7 @@ extern signed long schedule_timeout(signed long timeout); extern signed long schedule_timeout_interruptible(signed long timeout); extern signed long schedule_timeout_killable(signed long timeout); extern signed long schedule_timeout_uninterruptible(signed long timeout); -asmlinkage void schedule(void); +asmlinkage int schedule(void); extern int mutex_spin_on_owner(struct mutex *lock, struct thread_info *owner); struct nsproxy; @@ -486,6 +504,9 @@ extern int get_dumpable(struct mm_struct *mm); #endif /* leave room for more dump flags */ #define MMF_VM_MERGEABLE 16 /* KSM may merge identical pages */ +#ifdef CONFIG_IPIPE +#define MMF_VM_PINNED 31 /* ondemand load up and COW disabled */ +#endif #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK) @@ -1512,6 +1533,9 @@ struct task_struct { #endif atomic_t fs_excl; /* holding fs exclusive resources */ struct rcu_head rcu; +#ifdef CONFIG_IPIPE + void *ptd[IPIPE_ROOT_NPTDKEYS]; +#endif /* * cache last used pipe for splice @@ -1759,6 +1783,11 @@ extern void thread_group_times(struct task_struct *p, cputime_t *ut, cputime_t * #define PF_EXITING 0x00000004 /* getting shut down */ #define PF_EXITPIDONE 0x00000008 /* pi exit done on shut down */ #define PF_VCPU 0x00000010 /* I'm a virtual CPU */ +#ifdef CONFIG_IPIPE +#define PF_EVNOTIFY 0x00000020 /* Notify other domains about internal events */ +#else +#define PF_EVNOTIFY 0 +#endif /* CONFIG_IPIPE */ #define PF_FORKNOEXEC 0x00000040 /* forked but didn't exec */ #define PF_MCE_PROCESS 0x00000080 /* process policy on mce errors */ #define PF_SUPERPRIV 0x00000100 /* used super-user privileges */ diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 8608821..c93ec16 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -88,6 +88,8 @@ # include <linux/spinlock_up.h> #endif +#include <linux/ipipe_lock.h> + #ifdef CONFIG_DEBUG_SPINLOCK extern void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name, struct lock_class_key *key); @@ -270,16 +272,9 @@ static inline raw_spinlock_t *spinlock_check(spinlock_t *lock) return &lock->rlock; } -#define spin_lock_init(_lock) \ -do { \ - spinlock_check(_lock); \ - raw_spin_lock_init(&(_lock)->rlock); \ -} while (0) +#define spin_lock_init(_lock) PICK_SPINOP(_lock_init, _lock) -static inline void spin_lock(spinlock_t *lock) -{ - raw_spin_lock(&lock->rlock); -} +#define spin_lock(lock) PICK_SPINOP(_lock, lock) static inline void spin_lock_bh(spinlock_t *lock) { @@ -288,7 +283,7 @@ static inline void spin_lock_bh(spinlock_t *lock) static inline int spin_trylock(spinlock_t *lock) { - return raw_spin_trylock(&lock->rlock); + return PICK_SPINOP_RET(_trylock, lock, int); } #define spin_lock_nested(lock, subclass) \ @@ -301,40 +296,27 @@ do { \ raw_spin_lock_nest_lock(spinlock_check(lock), nest_lock); \ } while (0) -static inline void spin_lock_irq(spinlock_t *lock) -{ - raw_spin_lock_irq(&lock->rlock); -} +#define spin_lock_irq(lock) PICK_SPINOP(_lock_irq, lock) -#define spin_lock_irqsave(lock, flags) \ -do { \ - raw_spin_lock_irqsave(spinlock_check(lock), flags); \ -} while (0) +#define spin_lock_irqsave(lock, flags) \ + PICK_SPINLOCK_IRQSAVE(lock, flags) #define spin_lock_irqsave_nested(lock, flags, subclass) \ do { \ raw_spin_lock_irqsave_nested(spinlock_check(lock), flags, subclass); \ } while (0) -static inline void spin_unlock(spinlock_t *lock) -{ - raw_spin_unlock(&lock->rlock); -} +#define spin_unlock(lock) PICK_SPINOP(_unlock, lock) static inline void spin_unlock_bh(spinlock_t *lock) { raw_spin_unlock_bh(&lock->rlock); } -static inline void spin_unlock_irq(spinlock_t *lock) -{ - raw_spin_unlock_irq(&lock->rlock); -} +#define spin_unlock_irq(lock) PICK_SPINOP(_unlock_irq, lock) -static inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) -{ - raw_spin_unlock_irqrestore(&lock->rlock, flags); -} +#define spin_unlock_irqrestore(lock, flags) \ + PICK_SPINUNLOCK_IRQRESTORE(lock, flags) static inline int spin_trylock_bh(spinlock_t *lock) { @@ -347,9 +329,7 @@ static inline int spin_trylock_irq(spinlock_t *lock) } #define spin_trylock_irqsave(lock, flags) \ -({ \ - raw_spin_trylock_irqsave(spinlock_check(lock), flags); \ -}) + PICK_SPINTRYLOCK_IRQSAVE(lock, flags) static inline void spin_unlock_wait(spinlock_t *lock) { diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_smp.h index e253ccd..378e01e 100644 --- a/include/linux/spinlock_api_smp.h +++ b/include/linux/spinlock_api_smp.h @@ -99,7 +99,9 @@ static inline int __raw_spin_trylock(raw_spinlock_t *lock) * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock) { @@ -113,7 +115,7 @@ static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock) * do_raw_spin_lock_flags() code, because lockdep assumes * that interrupts are not re-enabled during lock-acquire: */ -#ifdef CONFIG_LOCKDEP +#if defined(CONFIG_LOCKDEP) || defined(CONFIG_IPIPE) LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); #else do_raw_spin_lock_flags(lock, &flags); diff --git a/include/linux/spinlock_up.h b/include/linux/spinlock_up.h index b14f6a9..e400972 100644 --- a/include/linux/spinlock_up.h +++ b/include/linux/spinlock_up.h @@ -49,13 +49,6 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock) /* * Read-write spinlocks. No debug version. */ -#define arch_read_lock(lock) do { (void)(lock); } while (0) -#define arch_write_lock(lock) do { (void)(lock); } while (0) -#define arch_read_trylock(lock) ({ (void)(lock); 1; }) -#define arch_write_trylock(lock) ({ (void)(lock); 1; }) -#define arch_read_unlock(lock) do { (void)(lock); } while (0) -#define arch_write_unlock(lock) do { (void)(lock); } while (0) - #else /* DEBUG_SPINLOCK */ #define arch_spin_is_locked(lock) ((void)(lock), 0) /* for sched.c and kernel_lock.c: */ @@ -65,6 +58,13 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock) # define arch_spin_trylock(lock) ({ (void)(lock); 1; }) #endif /* DEBUG_SPINLOCK */ +#define arch_read_lock(lock) do { (void)(lock); } while (0) +#define arch_write_lock(lock) do { (void)(lock); } while (0) +#define arch_read_trylock(lock) ({ (void)(lock); 1; }) +#define arch_write_trylock(lock) ({ (void)(lock); 1; }) +#define arch_read_unlock(lock) do { (void)(lock); } while (0) +#define arch_write_unlock(lock) do { (void)(lock); } while (0) + #define arch_spin_is_contended(lock) (((void)(lock), 0)) #define arch_read_can_lock(lock) (((void)(lock), 1)) diff --git a/include/linux/utsrelease.h b/include/linux/utsrelease.h new file mode 100644 index 0000000..a7e4640 --- /dev/null +++ b/include/linux/utsrelease.h @@ -0,0 +1 @@ +#define UTS_RELEASE "2.6.32.7" diff --git a/init/Kconfig b/init/Kconfig index d95ca7c..1c6c3bf 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -78,6 +78,7 @@ config INIT_ENV_ARG_LIMIT config LOCALVERSION string "Local version - append to kernel release" + default "-ipipe" help Append an extra string to the end of your kernel version. This will show up when you type uname, for example. diff --git a/init/main.c b/init/main.c index 4cb47a1..e714997 100644 --- a/init/main.c +++ b/init/main.c @@ -530,7 +530,7 @@ asmlinkage void __init start_kernel(void) cgroup_init_early(); - local_irq_disable(); + local_irq_disable_hw(); early_boot_irqs_off(); early_init_irq_lock_class(); @@ -593,6 +593,11 @@ asmlinkage void __init start_kernel(void) softirq_init(); timekeeping_init(); time_init(); + /* + * We need to wait for the interrupt and time subsystems to be + * initialized before enabling the pipeline. + */ + ipipe_init(); profile_init(); if (!irqs_disabled()) printk(KERN_CRIT "start_kernel(): bug: interrupts were " @@ -774,6 +779,7 @@ static void __init do_basic_setup(void) init_tmpfs(); driver_init(); init_irq_proc(); + ipipe_init_proc(); do_ctors(); do_initcalls(); } diff --git a/kernel/Makefile b/kernel/Makefile index 864ff75..864aa99 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -85,6 +85,7 @@ obj-$(CONFIG_TREE_PREEMPT_RCU) += rcutree.o obj-$(CONFIG_TREE_RCU_TRACE) += rcutree_trace.o obj-$(CONFIG_TINY_RCU) += rcutiny.o obj-$(CONFIG_RELAY) += relay.o +obj-$(CONFIG_IPIPE) += ipipe/ obj-$(CONFIG_SYSCTL) += utsname_sysctl.o obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o diff --git a/kernel/exit.c b/kernel/exit.c index 546774a..1c3531a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -967,6 +967,7 @@ NORET_TYPE void do_exit(long code) acct_process(); trace_sched_process_exit(tsk); + ipipe_exit_notify(tsk); exit_sem(tsk); exit_files(tsk); exit_fs(tsk); diff --git a/kernel/fork.c b/kernel/fork.c index f88bd98..545fd6e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -513,6 +513,7 @@ void mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); exit_mmap(mm); + ipipe_cleanup_notify(mm); set_mm_exe_file(mm, NULL); if (!list_empty(&mm->mmlist)) { spin_lock(&mmlist_lock); @@ -923,7 +924,7 @@ static void copy_flags(unsigned long clone_flags, struct task_struct *p) { unsigned long new_flags = p->flags; - new_flags &= ~PF_SUPERPRIV; + new_flags &= ~(PF_SUPERPRIV | PF_EVNOTIFY); new_flags |= PF_FORKNOEXEC; new_flags |= PF_STARTING; p->flags = new_flags; @@ -1300,6 +1301,9 @@ static struct task_struct *copy_process(unsigned long clone_flags, write_unlock_irq(&tasklist_lock); proc_fork_connector(p); cgroup_post_fork(p); +#ifdef CONFIG_IPIPE + memset(p->ptd, 0, sizeof(p->ptd)); +#endif /* CONFIG_IPIPE */ perf_event_fork(p); return p; @@ -1698,11 +1702,14 @@ SYSCALL_DEFINE1(unshare, unsigned long, unshare_flags) } if (new_mm) { + unsigned long flags; mm = current->mm; active_mm = current->active_mm; current->mm = new_mm; + ipipe_mm_switch_protect(flags); current->active_mm = new_mm; activate_mm(active_mm, new_mm); + ipipe_mm_switch_unprotect(flags); new_mm = mm; } diff --git a/kernel/ipipe/Kconfig b/kernel/ipipe/Kconfig new file mode 100644 index 0000000..8ff9d37 --- /dev/null +++ b/kernel/ipipe/Kconfig @@ -0,0 +1,23 @@ +config IPIPE + bool "Interrupt pipeline" + default y + ---help--- + Activate this option if you want the interrupt pipeline to be + compiled in. + +config IPIPE_DOMAINS + int "Max domains" + depends on IPIPE + default 4 + ---help--- + The maximum number of I-pipe domains to run concurrently. + +config IPIPE_DELAYED_ATOMICSW + bool + depends on IPIPE + default n + +config IPIPE_UNMASKED_CONTEXT_SWITCH + bool + depends on IPIPE + default n diff --git a/kernel/ipipe/Kconfig.debug b/kernel/ipipe/Kconfig.debug new file mode 100644 index 0000000..629c894 --- /dev/null +++ b/kernel/ipipe/Kconfig.debug @@ -0,0 +1,97 @@ +config IPIPE_DEBUG + bool "I-pipe debugging" + depends on IPIPE + +config IPIPE_DEBUG_CONTEXT + bool "Check for illicit cross-domain calls" + depends on IPIPE_DEBUG + default y + ---help--- + Enable this feature to arm checkpoints in the kernel that + verify the correct invocation context. On entry of critical + Linux services a warning is issued if the caller is not + running over the root domain. + +config IPIPE_DEBUG_INTERNAL + bool "Enable internal debug checks" + depends on IPIPE_DEBUG + default y + ---help--- + When this feature is enabled, I-pipe will perform internal + consistency checks of its subsystems, e.g. on per-cpu variable + access. + +config IPIPE_TRACE + bool "Latency tracing" + depends on IPIPE_DEBUG + select FRAME_POINTER + select KALLSYMS + select PROC_FS + ---help--- + Activate this option if you want to use per-function tracing of + the kernel. The tracer will collect data via instrumentation + features like the one below or with the help of explicite calls + of ipipe_trace_xxx(). See include/linux/ipipe_trace.h for the + in-kernel tracing API. The collected data and runtime control + is available via /proc/ipipe/trace/*. + +if IPIPE_TRACE + +config IPIPE_TRACE_ENABLE + bool "Enable tracing on boot" + default y + ---help--- + Disable this option if you want to arm the tracer after booting + manually ("echo 1 > /proc/ipipe/tracer/enable"). This can reduce + boot time on slow embedded devices due to the tracer overhead. + +config IPIPE_TRACE_MCOUNT + bool "Instrument function entries" + default y + select FUNCTION_TRACER + select TRACING + select CONTEXT_SWITCH_TRACER + select FTRACE_MCOUNT_RECORD + select DYNAMIC_FTRACE + ---help--- + When enabled, records every kernel function entry in the tracer + log. While this slows down the system noticeably, it provides + the highest level of information about the flow of events. + However, it can be switch off in order to record only explicit + I-pipe trace points. + +config IPIPE_TRACE_IRQSOFF + bool "Trace IRQs-off times" + default y + ---help--- + Activate this option if I-pipe shall trace the longest path + with hard-IRQs switched off. + +config IPIPE_TRACE_SHIFT + int "Depth of trace log (14 => 16Kpoints, 15 => 32Kpoints)" + range 10 18 + default 14 + ---help--- + The number of trace points to hold tracing data for each + trace path, as a power of 2. + +config IPIPE_TRACE_VMALLOC + bool "Use vmalloc'ed trace buffer" + default y if EMBEDDED + ---help--- + Instead of reserving static kernel data, the required buffer + is allocated via vmalloc during boot-up when this option is + enabled. This can help to start systems that are low on memory, + but it slightly degrades overall performance. Try this option + when a traced kernel hangs unexpectedly at boot time. + +config IPIPE_TRACE_PANIC + bool "Enable panic back traces" + default y + ---help--- + Provides services to freeze and dump a back trace on panic + situations. This is used on IPIPE_DEBUG_CONTEXT exceptions + as well as ordinary kernel oopses. You can control the number + of printed back trace points via /proc/ipipe/trace. + +endif diff --git a/kernel/ipipe/Makefile b/kernel/ipipe/Makefile new file mode 100644 index 0000000..6257dfa --- /dev/null +++ b/kernel/ipipe/Makefile @@ -0,0 +1,3 @@ + +obj-$(CONFIG_IPIPE) += core.o +obj-$(CONFIG_IPIPE_TRACE) += tracer.o diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c new file mode 100644 index 0000000..af1f7e2 --- /dev/null +++ b/kernel/ipipe/core.c @@ -0,0 +1,1970 @@ +/* -*- linux-c -*- + * linux/kernel/ipipe/core.c + * + * Copyright (C) 2002-2005 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Architecture-independent I-PIPE core support. + */ + +#include <linux/version.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/kernel.h> +#include <linux/sched.h> +#include <linux/sched.h> +#include <linux/kallsyms.h> +#include <linux/interrupt.h> +#include <linux/bitops.h> +#include <linux/tick.h> +#include <linux/prefetch.h> +#ifdef CONFIG_PROC_FS +#include <linux/proc_fs.h> +#include <linux/seq_file.h> +#endif /* CONFIG_PROC_FS */ +#include <linux/ipipe_trace.h> +#include <linux/ipipe_tickdev.h> +#include <linux/irq.h> + +static int __ipipe_ptd_key_count; + +static unsigned long __ipipe_ptd_key_map; + +static unsigned long __ipipe_domain_slot_map; + +struct ipipe_domain ipipe_root; + +#ifndef CONFIG_SMP +/* + * Create an alias to the unique root status, so that arch-dep code + * may get simple and easy access to this percpu variable. We also + * create an array of pointers to the percpu domain data; this tends + * to produce a better code when reaching non-root domains. We make + * sure that the early boot code would be able to dereference the + * pointer to the root domain data safely by statically initializing + * its value (local_irq*() routines depend on this). + */ +#if __GNUC__ >= 4 +extern unsigned long __ipipe_root_status +__attribute__((alias(__stringify(__raw_get_cpu_var(ipipe_percpu_darray))))); +EXPORT_SYMBOL(__ipipe_root_status); +#else /* __GNUC__ < 4 */ +/* + * Work around a GCC 3.x issue making alias symbols unusable as + * constant initializers. + */ +unsigned long *const __ipipe_root_status_addr = + &__raw_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT].status; +EXPORT_SYMBOL(__ipipe_root_status_addr); +#endif /* __GNUC__ < 4 */ + +DEFINE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]) = +{ [IPIPE_ROOT_SLOT] = (struct ipipe_percpu_domain_data *)&__raw_get_cpu_var(ipipe_percpu_darray) }; +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_daddr); +#endif /* !CONFIG_SMP */ + +DEFINE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]) = +{ [IPIPE_ROOT_SLOT] = { .status = IPIPE_STALL_MASK } }; /* Root domain stalled on each CPU at startup. */ + +DEFINE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain) = { &ipipe_root }; + +DEFINE_PER_CPU(unsigned long, ipipe_nmi_saved_root); /* Copy of root status during NMI */ + +static IPIPE_DEFINE_SPINLOCK(__ipipe_pipelock); + +LIST_HEAD(__ipipe_pipeline); + +unsigned long __ipipe_virtual_irq_map; + +#ifdef CONFIG_PRINTK +unsigned __ipipe_printk_virq; +#endif /* CONFIG_PRINTK */ + +int __ipipe_event_monitors[IPIPE_NR_EVENTS]; + +#ifdef CONFIG_GENERIC_CLOCKEVENTS + +DECLARE_PER_CPU(struct tick_device, tick_cpu_device); + +static DEFINE_PER_CPU(struct ipipe_tick_device, ipipe_tick_cpu_device); + +int ipipe_request_tickdev(const char *devname, + void (*emumode)(enum clock_event_mode mode, + struct clock_event_device *cdev), + int (*emutick)(unsigned long delta, + struct clock_event_device *cdev), + int cpu, unsigned long *tmfreq) +{ + struct ipipe_tick_device *itd; + struct tick_device *slave; + struct clock_event_device *evtdev; + unsigned long long freq; + unsigned long flags; + int status; + + flags = ipipe_critical_enter(NULL); + + itd = &per_cpu(ipipe_tick_cpu_device, cpu); + + if (itd->slave != NULL) { + status = -EBUSY; + goto out; + } + + slave = &per_cpu(tick_cpu_device, cpu); + + if (strcmp(slave->evtdev->name, devname)) { + /* + * No conflict so far with the current tick device, + * check whether the requested device is sane and has + * been blessed by the kernel. + */ + status = __ipipe_check_tickdev(devname) ? + CLOCK_EVT_MODE_UNUSED : CLOCK_EVT_MODE_SHUTDOWN; + goto out; + } + + /* + * Our caller asks for using the same clock event device for + * ticking than we do, let's create a tick emulation device to + * interpose on the set_next_event() method, so that we may + * both manage the device in oneshot mode. Only the tick + * emulation code will actually program the clockchip hardware + * for the next shot, though. + * + * CAUTION: we still have to grab the tick device even when it + * current runs in periodic mode, since the kernel may switch + * to oneshot dynamically (highres/no_hz tick mode). + */ + + evtdev = slave->evtdev; + status = evtdev->mode; + + if (status == CLOCK_EVT_MODE_SHUTDOWN) + goto out; + + itd->slave = slave; + itd->emul_set_mode = emumode; + itd->emul_set_tick = emutick; + itd->real_set_mode = evtdev->set_mode; + itd->real_set_tick = evtdev->set_next_event; + itd->real_max_delta_ns = evtdev->max_delta_ns; + itd->real_mult = evtdev->mult; + itd->real_shift = evtdev->shift; + freq = (1000000000ULL * evtdev->mult) >> evtdev->shift; + *tmfreq = (unsigned long)freq; + evtdev->set_mode = emumode; + evtdev->set_next_event = emutick; + evtdev->max_delta_ns = ULONG_MAX; + evtdev->mult = 1; + evtdev->shift = 0; +out: + ipipe_critical_exit(flags); + + return status; +} + +void ipipe_release_tickdev(int cpu) +{ + struct ipipe_tick_device *itd; + struct tick_device *slave; + struct clock_event_device *evtdev; + unsigned long flags; + + flags = ipipe_critical_enter(NULL); + + itd = &per_cpu(ipipe_tick_cpu_device, cpu); + + if (itd->slave != NULL) { + slave = &per_cpu(tick_cpu_device, cpu); + evtdev = slave->evtdev; + evtdev->set_mode = itd->real_set_mode; + evtdev->set_next_event = itd->real_set_tick; + evtdev->max_delta_ns = itd->real_max_delta_ns; + evtdev->mult = itd->real_mult; + evtdev->shift = itd->real_shift; + itd->slave = NULL; + } + + ipipe_critical_exit(flags); +} + +#endif /* CONFIG_GENERIC_CLOCKEVENTS */ + +/* + * ipipe_init() -- Initialization routine of the IPIPE layer. Called + * by the host kernel early during the boot procedure. + */ +void __init ipipe_init(void) +{ + struct ipipe_domain *ipd = &ipipe_root; + + __ipipe_check_platform(); /* Do platform dependent checks first. */ + + /* + * A lightweight registration code for the root domain. We are + * running on the boot CPU, hw interrupts are off, and + * secondary CPUs are still lost in space. + */ + + /* Reserve percpu data slot #0 for the root domain. */ + ipd->slot = 0; + set_bit(0, &__ipipe_domain_slot_map); + + ipd->name = "Linux"; + ipd->domid = IPIPE_ROOT_ID; + ipd->priority = IPIPE_ROOT_PRIO; + + __ipipe_init_stage(ipd); + + INIT_LIST_HEAD(&ipd->p_link); + list_add_tail(&ipd->p_link, &__ipipe_pipeline); + + __ipipe_init_platform(); + +#ifdef CONFIG_PRINTK + __ipipe_printk_virq = ipipe_alloc_virq(); /* Cannot fail here. */ + ipd->irqs[__ipipe_printk_virq].handler = &__ipipe_flush_printk; + ipd->irqs[__ipipe_printk_virq].cookie = NULL; + ipd->irqs[__ipipe_printk_virq].acknowledge = NULL; + ipd->irqs[__ipipe_printk_virq].control = IPIPE_HANDLE_MASK; +#endif /* CONFIG_PRINTK */ + + __ipipe_enable_pipeline(); + + printk(KERN_INFO "I-pipe %s: pipeline enabled.\n", + IPIPE_VERSION_STRING); +} + +void __ipipe_init_stage(struct ipipe_domain *ipd) +{ + struct ipipe_percpu_domain_data *p; + unsigned long status; + int cpu, n; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + status = p->status; + memset(p, 0, sizeof(*p)); + p->status = status; + } + + for (n = 0; n < IPIPE_NR_IRQS; n++) { + ipd->irqs[n].acknowledge = NULL; + ipd->irqs[n].handler = NULL; + ipd->irqs[n].control = IPIPE_PASS_MASK; /* Pass but don't handle */ + } + + for (n = 0; n < IPIPE_NR_EVENTS; n++) + ipd->evhand[n] = NULL; + + ipd->evself = 0LL; + mutex_init(&ipd->mutex); + + __ipipe_hook_critical_ipi(ipd); +} + +void __ipipe_cleanup_domain(struct ipipe_domain *ipd) +{ + ipipe_unstall_pipeline_from(ipd); + +#ifdef CONFIG_SMP + { + struct ipipe_percpu_domain_data *p; + int cpu; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + while (__ipipe_ipending_p(p)) + cpu_relax(); + } + } +#else + __raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = NULL; +#endif + + clear_bit(ipd->slot, &__ipipe_domain_slot_map); +} + +void __ipipe_unstall_root(void) +{ + struct ipipe_percpu_domain_data *p; + + local_irq_disable_hw(); + +#ifdef CONFIG_IPIPE_DEBUG_INTERNAL + /* This helps catching bad usage from assembly call sites. */ + BUG_ON(!__ipipe_root_domain_p); +#endif + + p = ipipe_root_cpudom_ptr(); + + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (unlikely(__ipipe_ipending_p(p))) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + local_irq_enable_hw(); +} + +void __ipipe_restore_root(unsigned long x) +{ +#ifdef CONFIG_IPIPE_DEBUG_INTERNAL + BUG_ON(!ipipe_root_domain_p); +#endif + + if (x) + __ipipe_stall_root(); + else + __ipipe_unstall_root(); +} + +void ipipe_stall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags; + /* + * We have to prevent against race on updating the status + * variable _and_ CPU migration at the same time, so disable + * hw IRQs here. + */ + local_irq_save_hw(flags); + + __set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (!__ipipe_pipeline_head_p(ipd)) + local_irq_restore_hw(flags); +} + +unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags, x; + + /* See ipipe_stall_pipeline_from() */ + local_irq_save_hw(flags); + + x = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (!__ipipe_pipeline_head_p(ipd)) + local_irq_restore_hw(flags); + + return x; +} + +unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags, x; + struct list_head *pos; + + local_irq_save_hw(flags); + + x = __test_and_clear_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (ipd == __ipipe_current_domain) + pos = &ipd->p_link; + else + pos = __ipipe_pipeline.next; + + __ipipe_walk_pipeline(pos); + + if (likely(__ipipe_pipeline_head_p(ipd))) + local_irq_enable_hw(); + else + local_irq_restore_hw(flags); + + return x; +} + +void ipipe_restore_pipeline_from(struct ipipe_domain *ipd, + unsigned long x) +{ + if (x) + ipipe_stall_pipeline_from(ipd); + else + ipipe_unstall_pipeline_from(ipd); +} + +void ipipe_unstall_pipeline_head(void) +{ + struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr(); + struct ipipe_domain *head_domain; + + local_irq_disable_hw(); + + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (unlikely(__ipipe_ipending_p(p))) { + head_domain = __ipipe_pipeline_head(); + if (likely(head_domain == __ipipe_current_domain)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else + __ipipe_walk_pipeline(&head_domain->p_link); + } + + local_irq_enable_hw(); +} + +void __ipipe_restore_pipeline_head(unsigned long x) +{ + struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr(); + struct ipipe_domain *head_domain; + + local_irq_disable_hw(); + + if (x) { +#ifdef CONFIG_DEBUG_KERNEL + static int warned; + if (!warned && test_and_set_bit(IPIPE_STALL_FLAG, &p->status)) { + /* + * Already stalled albeit ipipe_restore_pipeline_head() + * should have detected it? Send a warning once. + */ + warned = 1; + printk(KERN_WARNING + "I-pipe: ipipe_restore_pipeline_head() optimization failed.\n"); + dump_stack(); + } +#else /* !CONFIG_DEBUG_KERNEL */ + set_bit(IPIPE_STALL_FLAG, &p->status); +#endif /* CONFIG_DEBUG_KERNEL */ + } + else { + __clear_bit(IPIPE_STALL_FLAG, &p->status); + if (unlikely(__ipipe_ipending_p(p))) { + head_domain = __ipipe_pipeline_head(); + if (likely(head_domain == __ipipe_current_domain)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else + __ipipe_walk_pipeline(&head_domain->p_link); + } + local_irq_enable_hw(); + } +} + +void __ipipe_spin_lock_irq(ipipe_spinlock_t *lock) +{ + local_irq_disable_hw(); + arch_spin_lock(&lock->arch_lock); + __set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); +} + +void __ipipe_spin_unlock_irq(ipipe_spinlock_t *lock) +{ + arch_spin_unlock(&lock->arch_lock); + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_enable_hw(); +} + +unsigned long __ipipe_spin_lock_irqsave(ipipe_spinlock_t *lock) +{ + unsigned long flags; + int s; + + local_irq_save_hw(flags); + arch_spin_lock(&lock->arch_lock); + s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + + return raw_mangle_irq_bits(s, flags); +} + +int __ipipe_spin_trylock_irqsave(ipipe_spinlock_t *lock, + unsigned long *x) +{ + unsigned long flags; + int s; + + local_irq_save_hw(flags); + if (!arch_spin_trylock(&lock->arch_lock)) { + local_irq_restore_hw(flags); + return 0; + } + s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + *x = raw_mangle_irq_bits(s, flags); + + return 1; +} + +void __ipipe_spin_unlock_irqrestore(ipipe_spinlock_t *lock, + unsigned long x) +{ + arch_spin_unlock(&lock->arch_lock); + if (!raw_demangle_irq_bits(&x)) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_restore_hw(x); +} + +void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock) +{ + arch_spin_unlock(&lock->arch_lock); +} + +void __ipipe_spin_unlock_irqcomplete(unsigned long x) +{ + if (!raw_demangle_irq_bits(&x)) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_restore_hw(x); +} + +#ifdef __IPIPE_3LEVEL_IRQMAP + +/* Must be called hw IRQs off. */ +static inline void __ipipe_set_irq_held(struct ipipe_percpu_domain_data *p, + unsigned int irq) +{ + __set_bit(irq, p->irqheld_map); + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd); + int l0b, l1b; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + prefetchw(p); + + if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) { + __set_bit(irq, p->irqpend_lomap); + __set_bit(l1b, p->irqpend_mdmap); + __set_bit(l0b, &p->irqpend_himap); + } else + __set_bit(irq, p->irqheld_map); + + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b, l1b; + + if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + + p = ipipe_percpudom_ptr(ipd, cpu); + if (__test_and_clear_bit(irq, p->irqpend_lomap)) { + __set_bit(irq, p->irqheld_map); + if (p->irqpend_lomap[l1b] == 0) { + __clear_bit(l1b, p->irqpend_mdmap); + if (p->irqpend_mdmap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } + } +} + +/* Must be called hw IRQs off. */ +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b, l1b, cpu; + + if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + if (test_and_clear_bit(irq, p->irqheld_map)) { + /* We need atomic ops here: */ + set_bit(irq, p->irqpend_lomap); + set_bit(l1b, p->irqpend_mdmap); + set_bit(l0b, &p->irqpend_himap); + } + } +} + +static inline int __ipipe_next_irq(struct ipipe_percpu_domain_data *p, + int dovirt) +{ + unsigned long l0m, l1m, l2m, himask, mdmask; + int l0b, l1b, l2b, vl0b, vl1b; + unsigned int irq; + + if (dovirt) { + /* + * All virtual IRQs are mapped by a single long word. + * There is exactly BITS_PER_LONG virqs, and they are + * always last in the interrupt map, starting at + * IPIPE_VIRQ_BASE. Therefore, we only need to test a + * single bit within the high and middle maps to check + * whether a virtual IRQ is pending (the computations + * below are constant). + */ + vl0b = IPIPE_VIRQ_BASE / (BITS_PER_LONG * BITS_PER_LONG); + himask = (1L << vl0b); + vl1b = IPIPE_VIRQ_BASE / BITS_PER_LONG; + mdmask = (1L << (vl1b & (BITS_PER_LONG-1))); + } else + himask = mdmask = ~0L; + + l0m = p->irqpend_himap & himask; + if (unlikely(l0m == 0)) + return -1; + + l0b = __ipipe_ffnz(l0m); + l1m = p->irqpend_mdmap[l0b] & mdmask; + if (unlikely(l1m == 0)) + return -1; + + l1b = __ipipe_ffnz(l1m) + l0b * BITS_PER_LONG; + l2m = p->irqpend_lomap[l1b]; + if (unlikely(l2m == 0)) + return -1; + + l2b = __ipipe_ffnz(l2m); + irq = l1b * BITS_PER_LONG + l2b; + + __clear_bit(irq, p->irqpend_lomap); + if (p->irqpend_lomap[l1b] == 0) { + __clear_bit(l1b, p->irqpend_mdmap); + if (p->irqpend_mdmap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } + + return irq; +} + +#else /* __IPIPE_2LEVEL_IRQMAP */ + +/* Must be called hw IRQs off. */ +static inline void __ipipe_set_irq_held(struct ipipe_percpu_domain_data *p, + unsigned int irq) +{ + __set_bit(irq, p->irqheld_map); + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd); + int l0b = irq / BITS_PER_LONG; + + prefetchw(p); + + if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) { + __set_bit(irq, p->irqpend_lomap); + __set_bit(l0b, &p->irqpend_himap); + } else + __set_bit(irq, p->irqheld_map); + + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b = irq / BITS_PER_LONG; + + if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + p = ipipe_percpudom_ptr(ipd, cpu); + if (__test_and_clear_bit(irq, p->irqpend_lomap)) { + __set_bit(irq, p->irqheld_map); + if (p->irqpend_lomap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } +} + +/* Must be called hw IRQs off. */ +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b = irq / BITS_PER_LONG, cpu; + + if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + if (test_and_clear_bit(irq, p->irqheld_map)) { + /* We need atomic ops here: */ + set_bit(irq, p->irqpend_lomap); + set_bit(l0b, &p->irqpend_himap); + } + } +} + +static inline int __ipipe_next_irq(struct ipipe_percpu_domain_data *p, + int dovirt) +{ + unsigned long l0m, l1m, himask = ~0L; + int l0b, l1b; + + himask <<= dovirt ? IPIPE_VIRQ_BASE/BITS_PER_LONG : 0; + + l0m = p->irqpend_himap & himask; + if (unlikely(l0m == 0)) + return -1; + + l0b = __ipipe_ffnz(l0m); + l1m = p->irqpend_lomap[l0b]; + if (unlikely(l1m == 0)) + return -1; + + l1b = __ipipe_ffnz(l1m); + __clear_bit(l1b, &p->irqpend_lomap[l0b]); + if (p->irqpend_lomap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + + return l0b * BITS_PER_LONG + l1b; +} + +#endif /* __IPIPE_2LEVEL_IRQMAP */ + +/* + * __ipipe_walk_pipeline(): Plays interrupts pending in the log. Must + * be called with local hw interrupts disabled. + */ +void __ipipe_walk_pipeline(struct list_head *pos) +{ + struct ipipe_domain *this_domain = __ipipe_current_domain, *next_domain; + struct ipipe_percpu_domain_data *p, *np; + + p = ipipe_cpudom_ptr(this_domain); + + while (pos != &__ipipe_pipeline) { + + next_domain = list_entry(pos, struct ipipe_domain, p_link); + np = ipipe_cpudom_ptr(next_domain); + + if (test_bit(IPIPE_STALL_FLAG, &np->status)) + break; /* Stalled stage -- do not go further. */ + + if (__ipipe_ipending_p(np)) { + if (next_domain == this_domain) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else { + + p->evsync = 0; + __ipipe_current_domain = next_domain; + ipipe_suspend_domain(); /* Sync stage and propagate interrupts. */ + + if (__ipipe_current_domain == next_domain) + __ipipe_current_domain = this_domain; + /* + * Otherwise, something changed the current domain under our + * feet recycling the register set; do not override the new + * domain. + */ + + if (__ipipe_ipending_p(p) && + !test_bit(IPIPE_STALL_FLAG, &p->status)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + } + break; + } else if (next_domain == this_domain) + break; + + pos = next_domain->p_link.next; + } +} + +/* + * ipipe_suspend_domain() -- Suspend the current domain, switching to + * the next one which has pending work down the pipeline. + */ +void ipipe_suspend_domain(void) +{ + struct ipipe_domain *this_domain, *next_domain; + struct ipipe_percpu_domain_data *p; + struct list_head *ln; + unsigned long flags; + + local_irq_save_hw(flags); + + this_domain = next_domain = __ipipe_current_domain; + p = ipipe_cpudom_ptr(this_domain); + p->status &= ~(IPIPE_STALL_MASK|IPIPE_SYNC_MASK); + + if (__ipipe_ipending_p(p)) + goto sync_stage; + + for (;;) { + ln = next_domain->p_link.next; + + if (ln == &__ipipe_pipeline) + break; + + next_domain = list_entry(ln, struct ipipe_domain, p_link); + p = ipipe_cpudom_ptr(next_domain); + + if (p->status & IPIPE_STALL_MASK) + break; + + if (!__ipipe_ipending_p(p)) + continue; + + __ipipe_current_domain = next_domain; +sync_stage: + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + if (__ipipe_current_domain != next_domain) + /* + * Something has changed the current domain under our + * feet, recycling the register set; take note. + */ + this_domain = __ipipe_current_domain; + } + + __ipipe_current_domain = this_domain; + + local_irq_restore_hw(flags); +} + + +/* ipipe_alloc_virq() -- Allocate a pipelined virtual/soft interrupt. + * Virtual interrupts are handled in exactly the same way than their + * hw-generated counterparts wrt pipelining. + */ +unsigned ipipe_alloc_virq(void) +{ + unsigned long flags, irq = 0; + int ipos; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + if (__ipipe_virtual_irq_map != ~0) { + ipos = ffz(__ipipe_virtual_irq_map); + set_bit(ipos, &__ipipe_virtual_irq_map); + irq = ipos + IPIPE_VIRQ_BASE; + } + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return irq; +} + +/* + * ipipe_control_irq() -- Change modes of a pipelined interrupt for + * the current domain. + */ +int ipipe_virtualize_irq(struct ipipe_domain *ipd, + unsigned irq, + ipipe_irq_handler_t handler, + void *cookie, + ipipe_irq_ackfn_t acknowledge, + unsigned modemask) +{ + ipipe_irq_handler_t old_handler; + struct irq_desc *desc; + unsigned long flags; + int err; + + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + + if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) + return -EPERM; + + if (!test_bit(IPIPE_AHEAD_FLAG, &ipd->flags)) + /* Silently unwire interrupts for non-heading domains. */ + modemask &= ~IPIPE_WIRED_MASK; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + old_handler = ipd->irqs[irq].handler; + + if (handler != NULL) { + if (handler == IPIPE_SAME_HANDLER) { + handler = old_handler; + cookie = ipd->irqs[irq].cookie; + + if (handler == NULL) { + err = -EINVAL; + goto unlock_and_exit; + } + } else if ((modemask & IPIPE_EXCLUSIVE_MASK) != 0 && + old_handler != NULL) { + err = -EBUSY; + goto unlock_and_exit; + } + + /* Wired interrupts can only be delivered to domains + * always heading the pipeline, and using dynamic + * propagation. */ + + if ((modemask & IPIPE_WIRED_MASK) != 0) { + if ((modemask & (IPIPE_PASS_MASK | IPIPE_STICKY_MASK)) != 0) { + err = -EINVAL; + goto unlock_and_exit; + } + modemask |= (IPIPE_HANDLE_MASK); + } + + if ((modemask & IPIPE_STICKY_MASK) != 0) + modemask |= IPIPE_HANDLE_MASK; + } else + modemask &= + ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK | + IPIPE_EXCLUSIVE_MASK | IPIPE_WIRED_MASK); + + if (acknowledge == NULL && !ipipe_virtual_irq_p(irq)) + /* + * Acknowledge handler unspecified for a hw interrupt: + * use the Linux-defined handler instead. + */ + acknowledge = ipipe_root_domain->irqs[irq].acknowledge; + + ipd->irqs[irq].handler = handler; + ipd->irqs[irq].cookie = cookie; + ipd->irqs[irq].acknowledge = acknowledge; + ipd->irqs[irq].control = modemask; + + if (irq < NR_IRQS && !ipipe_virtual_irq_p(irq)) { + desc = irq_to_desc(irq); + if (handler != NULL) { + if (desc) + __ipipe_enable_irqdesc(ipd, irq); + + if ((modemask & IPIPE_ENABLE_MASK) != 0) { + if (ipd != __ipipe_current_domain) { + /* + * IRQ enable/disable state is domain-sensitive, so we + * may not change it for another domain. What is + * allowed however is forcing some domain to handle an + * interrupt source, by passing the proper 'ipd' + * descriptor which thus may be different from + * __ipipe_current_domain. + */ + err = -EPERM; + goto unlock_and_exit; + } + if (desc) + __ipipe_enable_irq(irq); + } + } else if (old_handler != NULL && desc) + __ipipe_disable_irqdesc(ipd, irq); + } + + err = 0; + + unlock_and_exit: + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return err; +} + +/* ipipe_control_irq() -- Change modes of a pipelined interrupt for + * the current domain. */ + +int ipipe_control_irq(unsigned irq, unsigned clrmask, unsigned setmask) +{ + struct ipipe_domain *ipd; + unsigned long flags; + + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + ipd = __ipipe_current_domain; + + if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) { + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + return -EPERM; + } + + if (ipd->irqs[irq].handler == NULL) + setmask &= ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK); + + if ((setmask & IPIPE_STICKY_MASK) != 0) + setmask |= IPIPE_HANDLE_MASK; + + if ((clrmask & (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK)) != 0) /* If one goes, both go. */ + clrmask |= (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK); + + ipd->irqs[irq].control &= ~clrmask; + ipd->irqs[irq].control |= setmask; + + if ((setmask & IPIPE_ENABLE_MASK) != 0) + __ipipe_enable_irq(irq); + else if ((clrmask & IPIPE_ENABLE_MASK) != 0) + __ipipe_disable_irq(irq); + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return 0; +} + +/* __ipipe_dispatch_event() -- Low-level event dispatcher. */ + +int __ipipe_dispatch_event (unsigned event, void *data) +{ + struct ipipe_domain *start_domain, *this_domain, *next_domain; + struct ipipe_percpu_domain_data *np; + ipipe_event_handler_t evhand; + struct list_head *pos, *npos; + unsigned long flags; + int propagate = 1; + + local_irq_save_hw(flags); + + start_domain = this_domain = __ipipe_current_domain; + + list_for_each_safe(pos, npos, &__ipipe_pipeline) { + /* + * Note: Domain migration may occur while running + * event or interrupt handlers, in which case the + * current register set is going to be recycled for a + * different domain than the initiating one. We do + * care for that, always tracking the current domain + * descriptor upon return from those handlers. + */ + next_domain = list_entry(pos, struct ipipe_domain, p_link); + np = ipipe_cpudom_ptr(next_domain); + + /* + * Keep a cached copy of the handler's address since + * ipipe_catch_event() may clear it under our feet. + */ + evhand = next_domain->evhand[event]; + + if (evhand != NULL) { + __ipipe_current_domain = next_domain; + np->evsync |= (1LL << event); + local_irq_restore_hw(flags); + propagate = !evhand(event, start_domain, data); + local_irq_save_hw(flags); + /* + * We may have a migration issue here, if the + * current task is migrated to another CPU on + * behalf of the invoked handler, usually when + * a syscall event is processed. However, + * ipipe_catch_event() will make sure that a + * CPU that clears a handler for any given + * event will not attempt to wait for itself + * to clear the evsync bit for that event, + * which practically plugs the hole, without + * resorting to a much more complex strategy. + */ + np->evsync &= ~(1LL << event); + if (__ipipe_current_domain != next_domain) + this_domain = __ipipe_current_domain; + } + + /* NEVER sync the root stage here. */ + if (next_domain != ipipe_root_domain && + __ipipe_ipending_p(np) && + !test_bit(IPIPE_STALL_FLAG, &np->status)) { + __ipipe_current_domain = next_domain; + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + if (__ipipe_current_domain != next_domain) + this_domain = __ipipe_current_domain; + } + + __ipipe_current_domain = this_domain; + + if (next_domain == this_domain || !propagate) + break; + } + + local_irq_restore_hw(flags); + + return !propagate; +} + +/* + * __ipipe_dispatch_wired -- Wired interrupt dispatcher. Wired + * interrupts are immediately and unconditionally delivered to the + * domain heading the pipeline upon receipt, and such domain must have + * been registered as an invariant head for the system (priority == + * IPIPE_HEAD_PRIORITY). The motivation for using wired interrupts is + * to get an extra-fast dispatching path for those IRQs, by relying on + * a straightforward logic based on assumptions that must always be + * true for invariant head domains. The following assumptions are + * made when dealing with such interrupts: + * + * 1- Wired interrupts are purely dynamic, i.e. the decision to + * propagate them down the pipeline must be done from the head domain + * ISR. + * 2- Wired interrupts cannot be shared or sticky. + * 3- The root domain cannot be an invariant pipeline head, in + * consequence of what the root domain cannot handle wired + * interrupts. + * 4- Wired interrupts must have a valid acknowledge handler for the + * head domain (if needed, see __ipipe_handle_irq). + * + * Called with hw interrupts off. + */ + +void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head); + + prefetchw(p); + + if (unlikely(test_bit(IPIPE_LOCK_FLAG, &head->irqs[irq].control))) { + /* + * If we can't process this IRQ right now, we must + * mark it as held, so that it will get played during + * normal log sync when the corresponding interrupt + * source is eventually unlocked. + */ + __ipipe_set_irq_held(p, irq); + return; + } + + if (test_bit(IPIPE_STALL_FLAG, &p->status)) { + __ipipe_set_irq_pending(head, irq); + return; + } + + __ipipe_dispatch_wired_nocheck(head, irq); +} + +void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq) /* hw interrupts off */ +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head); + struct ipipe_domain *old; + + prefetchw(p); + + old = __ipipe_current_domain; + __ipipe_current_domain = head; /* Switch to the head domain. */ + + p->irqall[irq]++; + __set_bit(IPIPE_STALL_FLAG, &p->status); + head->irqs[irq].handler(irq, head->irqs[irq].cookie); /* Call the ISR. */ + __ipipe_run_irqtail(); + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (__ipipe_current_domain == head) { + __ipipe_current_domain = old; + if (old == head) { + if (__ipipe_ipending_p(p)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + return; + } + } + + __ipipe_walk_pipeline(&head->p_link); +} + +/* + * __ipipe_sync_stage() -- Flush the pending IRQs for the current + * domain (and processor). This routine flushes the interrupt log + * (see "Optimistic interrupt protection" from D. Stodolsky et al. for + * more on the deferred interrupt scheme). Every interrupt that + * occurred while the pipeline was stalled gets played. WARNING: + * callers on SMP boxen should always check for CPU migration on + * return of this routine. + * + * This routine must be called with hw interrupts off. + */ +void __ipipe_sync_stage(int dovirt) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *ipd; + int cpu, irq; + + ipd = __ipipe_current_domain; + p = ipipe_cpudom_ptr(ipd); + + if (__test_and_set_bit(IPIPE_SYNC_FLAG, &p->status)) { + /* + * Some questionable code in the root domain may enter + * busy waits for IRQs over interrupt context, so we + * unfortunately have to allow piling up IRQs for + * them. Non-root domains are not allowed to do this. + */ + if (ipd != ipipe_root_domain) + return; + } + + cpu = ipipe_processor_id(); + + for (;;) { + irq = __ipipe_next_irq(p, dovirt); + if (irq < 0) + break; + /* + * Make sure the compiler does not reorder + * wrongly, so that all updates to maps are + * done before the handler gets called. + */ + barrier(); + + if (test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control)) + continue; + + __set_bit(IPIPE_STALL_FLAG, &p->status); + smp_wmb(); + + if (ipd == ipipe_root_domain) + trace_hardirqs_off(); + + __ipipe_run_isr(ipd, irq); + barrier(); + p = ipipe_cpudom_ptr(__ipipe_current_domain); +#ifdef CONFIG_SMP + { + int newcpu = ipipe_processor_id(); + + if (newcpu != cpu) { /* Handle CPU migration. */ + /* + * We expect any domain to clear the SYNC bit each + * time it switches in a new task, so that preemptions + * and/or CPU migrations (in the SMP case) over the + * ISR do not lock out the log syncer for some + * indefinite amount of time. In the Linux case, + * schedule() handles this (see kernel/sched.c). For + * this reason, we don't bother clearing it here for + * the source CPU in the migration handling case, + * since it must have scheduled another task in by + * now. + */ + __set_bit(IPIPE_SYNC_FLAG, &p->status); + cpu = newcpu; + } + } +#endif /* CONFIG_SMP */ +#ifdef CONFIG_TRACE_IRQFLAGS + if (__ipipe_root_domain_p && + test_bit(IPIPE_STALL_FLAG, &p->status)) + trace_hardirqs_on(); +#endif + __clear_bit(IPIPE_STALL_FLAG, &p->status); + } + + __clear_bit(IPIPE_SYNC_FLAG, &p->status); +} + +/* ipipe_register_domain() -- Link a new domain to the pipeline. */ + +int ipipe_register_domain(struct ipipe_domain *ipd, + struct ipipe_domain_attr *attr) +{ + struct ipipe_percpu_domain_data *p; + struct list_head *pos = NULL; + struct ipipe_domain *_ipd; + unsigned long flags; + + if (!ipipe_root_domain_p) { + printk(KERN_WARNING + "I-pipe: Only the root domain may register a new domain.\n"); + return -EPERM; + } + + flags = ipipe_critical_enter(NULL); + + if (attr->priority == IPIPE_HEAD_PRIORITY) { + if (test_bit(IPIPE_HEAD_SLOT, &__ipipe_domain_slot_map)) { + ipipe_critical_exit(flags); + return -EAGAIN; /* Cannot override current head. */ + } + ipd->slot = IPIPE_HEAD_SLOT; + } else + ipd->slot = ffz(__ipipe_domain_slot_map); + + if (ipd->slot < CONFIG_IPIPE_DOMAINS) { + set_bit(ipd->slot, &__ipipe_domain_slot_map); + list_for_each(pos, &__ipipe_pipeline) { + _ipd = list_entry(pos, struct ipipe_domain, p_link); + if (_ipd->domid == attr->domid) + break; + } + } + + ipipe_critical_exit(flags); + + if (pos != &__ipipe_pipeline) { + if (ipd->slot < CONFIG_IPIPE_DOMAINS) + clear_bit(ipd->slot, &__ipipe_domain_slot_map); + return -EBUSY; + } + +#ifndef CONFIG_SMP + /* + * Set up the perdomain pointers for direct access to the + * percpu domain data. This saves a costly multiply each time + * we need to refer to the contents of the percpu domain data + * array. + */ + __raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = &__raw_get_cpu_var(ipipe_percpu_darray)[ipd->slot]; +#endif + + ipd->name = attr->name; + ipd->domid = attr->domid; + ipd->pdd = attr->pdd; + ipd->flags = 0; + + if (attr->priority == IPIPE_HEAD_PRIORITY) { + ipd->priority = INT_MAX; + __set_bit(IPIPE_AHEAD_FLAG,&ipd->flags); + } + else + ipd->priority = attr->priority; + + __ipipe_init_stage(ipd); + + INIT_LIST_HEAD(&ipd->p_link); + +#ifdef CONFIG_PROC_FS + __ipipe_add_domain_proc(ipd); +#endif /* CONFIG_PROC_FS */ + + flags = ipipe_critical_enter(NULL); + + list_for_each(pos, &__ipipe_pipeline) { + _ipd = list_entry(pos, struct ipipe_domain, p_link); + if (ipd->priority > _ipd->priority) + break; + } + + list_add_tail(&ipd->p_link, pos); + + ipipe_critical_exit(flags); + + printk(KERN_INFO "I-pipe: Domain %s registered.\n", ipd->name); + + if (attr->entry == NULL) + return 0; + + /* + * Finally, allow the new domain to perform its initialization + * duties. + */ + local_irq_save_hw_smp(flags); + __ipipe_current_domain = ipd; + local_irq_restore_hw_smp(flags); + attr->entry(); + local_irq_save_hw(flags); + __ipipe_current_domain = ipipe_root_domain; + p = ipipe_root_cpudom_ptr(); + + if (__ipipe_ipending_p(p) && + !test_bit(IPIPE_STALL_FLAG, &p->status)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + local_irq_restore_hw(flags); + + return 0; +} + +/* ipipe_unregister_domain() -- Remove a domain from the pipeline. */ + +int ipipe_unregister_domain(struct ipipe_domain *ipd) +{ + unsigned long flags; + + if (!ipipe_root_domain_p) { + printk(KERN_WARNING + "I-pipe: Only the root domain may unregister a domain.\n"); + return -EPERM; + } + + if (ipd == ipipe_root_domain) { + printk(KERN_WARNING + "I-pipe: Cannot unregister the root domain.\n"); + return -EPERM; + } +#ifdef CONFIG_SMP + { + struct ipipe_percpu_domain_data *p; + unsigned int irq; + int cpu; + + /* + * In the SMP case, wait for the logged events to drain on + * other processors before eventually removing the domain + * from the pipeline. + */ + + ipipe_unstall_pipeline_from(ipd); + + flags = ipipe_critical_enter(NULL); + + for (irq = 0; irq < IPIPE_NR_IRQS; irq++) { + clear_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control); + clear_bit(IPIPE_STICKY_FLAG, &ipd->irqs[irq].control); + set_bit(IPIPE_PASS_FLAG, &ipd->irqs[irq].control); + } + + ipipe_critical_exit(flags); + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + while (__ipipe_ipending_p(p)) + cpu_relax(); + } + } +#endif /* CONFIG_SMP */ + + mutex_lock(&ipd->mutex); + +#ifdef CONFIG_PROC_FS + __ipipe_remove_domain_proc(ipd); +#endif /* CONFIG_PROC_FS */ + + /* + * Simply remove the domain from the pipeline and we are almost done. + */ + + flags = ipipe_critical_enter(NULL); + list_del_init(&ipd->p_link); + ipipe_critical_exit(flags); + + __ipipe_cleanup_domain(ipd); + + mutex_unlock(&ipd->mutex); + + printk(KERN_INFO "I-pipe: Domain %s unregistered.\n", ipd->name); + + return 0; +} + +/* + * ipipe_propagate_irq() -- Force a given IRQ propagation on behalf of + * a running interrupt handler to the next domain down the pipeline. + * ipipe_schedule_irq() -- Does almost the same as above, but attempts + * to pend the interrupt for the current domain first. + * Must be called hw IRQs off. + */ +void __ipipe_pend_irq(unsigned irq, struct list_head *head) +{ + struct ipipe_domain *ipd; + struct list_head *ln; + +#ifdef CONFIG_IPIPE_DEBUG + BUG_ON(irq >= IPIPE_NR_IRQS || + (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))); +#endif + for (ln = head; ln != &__ipipe_pipeline; ln = ipd->p_link.next) { + ipd = list_entry(ln, struct ipipe_domain, p_link); + if (test_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control)) { + __ipipe_set_irq_pending(ipd, irq); + return; + } + } +} + +/* ipipe_free_virq() -- Release a virtual/soft interrupt. */ + +int ipipe_free_virq(unsigned virq) +{ + if (!ipipe_virtual_irq_p(virq)) + return -EINVAL; + + clear_bit(virq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map); + + return 0; +} + +void ipipe_init_attr(struct ipipe_domain_attr *attr) +{ + attr->name = "anon"; + attr->domid = 1; + attr->entry = NULL; + attr->priority = IPIPE_ROOT_PRIO; + attr->pdd = NULL; +} + +/* + * ipipe_catch_event() -- Interpose or remove an event handler for a + * given domain. + */ +ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd, + unsigned event, + ipipe_event_handler_t handler) +{ + ipipe_event_handler_t old_handler; + unsigned long flags; + int self = 0, cpu; + + if (event & IPIPE_EVENT_SELF) { + event &= ~IPIPE_EVENT_SELF; + self = 1; + } + + if (event >= IPIPE_NR_EVENTS) + return NULL; + + flags = ipipe_critical_enter(NULL); + + if (!(old_handler = xchg(&ipd->evhand[event],handler))) { + if (handler) { + if (self) + ipd->evself |= (1LL << event); + else + __ipipe_event_monitors[event]++; + } + } + else if (!handler) { + if (ipd->evself & (1LL << event)) + ipd->evself &= ~(1LL << event); + else + __ipipe_event_monitors[event]--; + } else if ((ipd->evself & (1LL << event)) && !self) { + __ipipe_event_monitors[event]++; + ipd->evself &= ~(1LL << event); + } else if (!(ipd->evself & (1LL << event)) && self) { + __ipipe_event_monitors[event]--; + ipd->evself |= (1LL << event); + } + + ipipe_critical_exit(flags); + + if (!handler && ipipe_root_domain_p) { + /* + * If we cleared a handler on behalf of the root + * domain, we have to wait for any current invocation + * to drain, since our caller might subsequently unmap + * the target domain. To this aim, this code + * synchronizes with __ipipe_dispatch_event(), + * guaranteeing that either the dispatcher sees a null + * handler in which case it discards the invocation + * (which also prevents from entering a livelock), or + * finds a valid handler and calls it. Symmetrically, + * ipipe_catch_event() ensures that the called code + * won't be unmapped under our feet until the event + * synchronization flag is cleared for the given event + * on all CPUs. + */ + preempt_disable(); + cpu = smp_processor_id(); + /* + * Hack: this solves the potential migration issue + * raised in __ipipe_dispatch_event(). This is a + * work-around which makes the assumption that other + * CPUs will subsequently, either process at least one + * interrupt for the target domain, or call + * __ipipe_dispatch_event() without going through a + * migration while running the handler at least once; + * practically, this is safe on any normally running + * system. + */ + ipipe_percpudom(ipd, evsync, cpu) &= ~(1LL << event); + preempt_enable(); + + for_each_online_cpu(cpu) { + while (ipipe_percpudom(ipd, evsync, cpu) & (1LL << event)) + schedule_timeout_interruptible(HZ / 50); + } + } + + return old_handler; +} + +cpumask_t ipipe_set_irq_affinity (unsigned irq, cpumask_t cpumask) +{ +#ifdef CONFIG_SMP + if (irq >= IPIPE_NR_XIRQS) + /* Allow changing affinity of external IRQs only. */ + return CPU_MASK_NONE; + + if (num_online_cpus() > 1) + return __ipipe_set_irq_affinity(irq,cpumask); +#endif /* CONFIG_SMP */ + + return CPU_MASK_NONE; +} + +int ipipe_send_ipi (unsigned ipi, cpumask_t cpumask) + +{ +#ifdef CONFIG_SMP + return __ipipe_send_ipi(ipi,cpumask); +#else /* !CONFIG_SMP */ + return -EINVAL; +#endif /* CONFIG_SMP */ +} + +int ipipe_alloc_ptdkey (void) +{ + unsigned long flags; + int key = -1; + + spin_lock_irqsave(&__ipipe_pipelock,flags); + + if (__ipipe_ptd_key_count < IPIPE_ROOT_NPTDKEYS) { + key = ffz(__ipipe_ptd_key_map); + set_bit(key,&__ipipe_ptd_key_map); + __ipipe_ptd_key_count++; + } + + spin_unlock_irqrestore(&__ipipe_pipelock,flags); + + return key; +} + +int ipipe_free_ptdkey (int key) +{ + unsigned long flags; + + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return -EINVAL; + + spin_lock_irqsave(&__ipipe_pipelock,flags); + + if (test_and_clear_bit(key,&__ipipe_ptd_key_map)) + __ipipe_ptd_key_count--; + + spin_unlock_irqrestore(&__ipipe_pipelock,flags); + + return 0; +} + +int ipipe_set_ptd (int key, void *value) + +{ + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return -EINVAL; + + current->ptd[key] = value; + + return 0; +} + +void *ipipe_get_ptd (int key) + +{ + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return NULL; + + return current->ptd[key]; +} + +#ifdef CONFIG_PROC_FS + +struct proc_dir_entry *ipipe_proc_root; + +static int __ipipe_version_info_proc(char *page, + char **start, + off_t off, int count, int *eof, void *data) +{ + int len = sprintf(page, "%s\n", IPIPE_VERSION_STRING); + + len -= off; + + if (len <= off + count) + *eof = 1; + + *start = page + off; + + if(len > count) + len = count; + + if(len < 0) + len = 0; + + return len; +} + +static int __ipipe_common_info_show(struct seq_file *p, void *data) +{ + struct ipipe_domain *ipd = (struct ipipe_domain *)p->private; + char handling, stickiness, lockbit, exclusive, virtuality; + + unsigned long ctlbits; + unsigned irq; + + seq_printf(p, " +----- Handling ([A]ccepted, [G]rabbed, [W]ired, [D]iscarded)\n"); + seq_printf(p, " |+---- Sticky\n"); + seq_printf(p, " ||+--- Locked\n"); + seq_printf(p, " |||+-- Exclusive\n"); + seq_printf(p, " ||||+- Virtual\n"); + seq_printf(p, "[IRQ] |||||\n"); + + mutex_lock(&ipd->mutex); + + for (irq = 0; irq < IPIPE_NR_IRQS; irq++) { + /* Remember to protect against + * ipipe_virtual_irq/ipipe_control_irq if more fields + * get involved. */ + ctlbits = ipd->irqs[irq].control; + + if (irq >= IPIPE_NR_XIRQS && !ipipe_virtual_irq_p(irq)) + /* + * There might be a hole between the last external + * IRQ and the first virtual one; skip it. + */ + continue; + + if (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map)) + /* Non-allocated virtual IRQ; skip it. */ + continue; + + /* + * Statuses are as follows: + * o "accepted" means handled _and_ passed down the pipeline. + * o "grabbed" means handled, but the interrupt might be + * terminated _or_ passed down the pipeline depending on + * what the domain handler asks for to the I-pipe. + * o "wired" is basically the same as "grabbed", except that + * the interrupt is unconditionally delivered to an invariant + * pipeline head domain. + * o "passed" means unhandled by the domain but passed + * down the pipeline. + * o "discarded" means unhandled and _not_ passed down the + * pipeline. The interrupt merely disappears from the + * current domain down to the end of the pipeline. + */ + if (ctlbits & IPIPE_HANDLE_MASK) { + if (ctlbits & IPIPE_PASS_MASK) + handling = 'A'; + else if (ctlbits & IPIPE_WIRED_MASK) + handling = 'W'; + else + handling = 'G'; + } else if (ctlbits & IPIPE_PASS_MASK) + /* Do not output if no major action is taken. */ + continue; + else + handling = 'D'; + + if (ctlbits & IPIPE_STICKY_MASK) + stickiness = 'S'; + else + stickiness = '.'; + + if (ctlbits & IPIPE_LOCK_MASK) + lockbit = 'L'; + else + lockbit = '.'; + + if (ctlbits & IPIPE_EXCLUSIVE_MASK) + exclusive = 'X'; + else + exclusive = '.'; + + if (ipipe_virtual_irq_p(irq)) + virtuality = 'V'; + else + virtuality = '.'; + + seq_printf(p, " %3u: %c%c%c%c%c\n", + irq, handling, stickiness, lockbit, exclusive, virtuality); + } + + seq_printf(p, "[Domain info]\n"); + + seq_printf(p, "id=0x%.8x\n", ipd->domid); + + if (test_bit(IPIPE_AHEAD_FLAG,&ipd->flags)) + seq_printf(p, "priority=topmost\n"); + else + seq_printf(p, "priority=%d\n", ipd->priority); + + mutex_unlock(&ipd->mutex); + + return 0; +} + +static int __ipipe_common_info_open(struct inode *inode, struct file *file) +{ + return single_open(file, __ipipe_common_info_show, PROC_I(inode)->pde->data); +} + +static struct file_operations __ipipe_info_proc_ops = { + .owner = THIS_MODULE, + .open = __ipipe_common_info_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +void __ipipe_add_domain_proc(struct ipipe_domain *ipd) +{ + struct proc_dir_entry *e = create_proc_entry(ipd->name, 0444, ipipe_proc_root); + if (e) { + e->proc_fops = &__ipipe_info_proc_ops; + e->data = (void*) ipd; + } +} + +void __ipipe_remove_domain_proc(struct ipipe_domain *ipd) +{ + remove_proc_entry(ipd->name,ipipe_proc_root); +} + +void __init ipipe_init_proc(void) +{ + ipipe_proc_root = create_proc_entry("ipipe",S_IFDIR, 0); + create_proc_read_entry("version",0444,ipipe_proc_root,&__ipipe_version_info_proc,NULL); + __ipipe_add_domain_proc(ipipe_root_domain); + + __ipipe_init_tracer(); +} + +#endif /* CONFIG_PROC_FS */ + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + +DEFINE_PER_CPU(int, ipipe_percpu_context_check) = { 1 }; +DEFINE_PER_CPU(int, ipipe_saved_context_check_state); + +void ipipe_check_context(struct ipipe_domain *border_domain) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *this_domain; + unsigned long flags; + int cpu; + + local_irq_save_hw_smp(flags); + + this_domain = __ipipe_current_domain; + p = ipipe_head_cpudom_ptr(); + if (likely(this_domain->priority <= border_domain->priority && + !test_bit(IPIPE_STALL_FLAG, &p->status))) { + local_irq_restore_hw_smp(flags); + return; + } + + cpu = ipipe_processor_id(); + if (!per_cpu(ipipe_percpu_context_check, cpu)) { + local_irq_restore_hw_smp(flags); + return; + } + + local_irq_restore_hw_smp(flags); + + ipipe_context_check_off(); + ipipe_trace_panic_freeze(); + ipipe_set_printk_sync(__ipipe_current_domain); + + if (this_domain->priority > border_domain->priority) + printk(KERN_ERR "I-pipe: Detected illicit call from domain " + "'%s'\n" + KERN_ERR " into a service reserved for domain " + "'%s' and below.\n", + this_domain->name, border_domain->name); + else + printk(KERN_ERR "I-pipe: Detected stalled topmost domain, " + "probably caused by a bug.\n" + " A critical section may have been " + "left unterminated.\n"); + dump_stack(); + ipipe_trace_panic_dump(); +} + +EXPORT_SYMBOL(ipipe_check_context); + +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ + +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) + +int notrace __ipipe_check_percpu_access(void) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *this_domain; + unsigned long flags; + int ret = 0; + + local_irq_save_hw_notrace(flags); + + this_domain = __raw_get_cpu_var(ipipe_percpu_domain); + + /* + * Only the root domain may implement preemptive CPU migration + * of tasks, so anything above in the pipeline should be fine. + */ + if (this_domain->priority > IPIPE_ROOT_PRIO) + goto out; + + if (raw_irqs_disabled_flags(flags)) + goto out; + + /* + * Last chance: hw interrupts were enabled on entry while + * running over the root domain, but the root stage might be + * currently stalled, in which case preemption would be + * disabled, and no migration could occur. + */ + if (this_domain == ipipe_root_domain) { + p = ipipe_root_cpudom_ptr(); + if (test_bit(IPIPE_STALL_FLAG, &p->status)) + goto out; + } + /* + * Our caller may end up accessing the wrong per-cpu variable + * instance due to CPU migration; tell it to complain about + * this. + */ + ret = 1; +out: + local_irq_restore_hw_notrace(flags); + + return ret; +} + +#endif /* CONFIG_IPIPE_DEBUG_INTERNAL && CONFIG_SMP */ + +EXPORT_SYMBOL(ipipe_virtualize_irq); +EXPORT_SYMBOL(ipipe_control_irq); +EXPORT_SYMBOL(ipipe_suspend_domain); +EXPORT_SYMBOL(ipipe_alloc_virq); +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_domain); +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_darray); +EXPORT_SYMBOL(ipipe_root); +EXPORT_SYMBOL(ipipe_stall_pipeline_from); +EXPORT_SYMBOL(ipipe_test_and_stall_pipeline_from); +EXPORT_SYMBOL(ipipe_test_and_unstall_pipeline_from); +EXPORT_SYMBOL(ipipe_restore_pipeline_from); +EXPORT_SYMBOL(ipipe_unstall_pipeline_head); +EXPORT_SYMBOL(__ipipe_restore_pipeline_head); +EXPORT_SYMBOL(__ipipe_unstall_root); +EXPORT_SYMBOL(__ipipe_restore_root); +EXPORT_SYMBOL(__ipipe_spin_lock_irq); +EXPORT_SYMBOL(__ipipe_spin_unlock_irq); +EXPORT_SYMBOL(__ipipe_spin_lock_irqsave); +EXPORT_SYMBOL(__ipipe_spin_trylock_irqsave); +EXPORT_SYMBOL(__ipipe_spin_unlock_irqrestore); +EXPORT_SYMBOL(__ipipe_pipeline); +EXPORT_SYMBOL(__ipipe_lock_irq); +EXPORT_SYMBOL(__ipipe_unlock_irq); +EXPORT_SYMBOL(ipipe_register_domain); +EXPORT_SYMBOL(ipipe_unregister_domain); +EXPORT_SYMBOL(ipipe_free_virq); +EXPORT_SYMBOL(ipipe_init_attr); +EXPORT_SYMBOL(ipipe_catch_event); +EXPORT_SYMBOL(ipipe_alloc_ptdkey); +EXPORT_SYMBOL(ipipe_free_ptdkey); +EXPORT_SYMBOL(ipipe_set_ptd); +EXPORT_SYMBOL(ipipe_get_ptd); +EXPORT_SYMBOL(ipipe_set_irq_affinity); +EXPORT_SYMBOL(ipipe_send_ipi); +EXPORT_SYMBOL(__ipipe_pend_irq); +EXPORT_SYMBOL(__ipipe_set_irq_pending); +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) +EXPORT_SYMBOL(__ipipe_check_percpu_access); +#endif +#ifdef CONFIG_GENERIC_CLOCKEVENTS +EXPORT_SYMBOL(ipipe_request_tickdev); +EXPORT_SYMBOL(ipipe_release_tickdev); +#endif + +EXPORT_SYMBOL(ipipe_critical_enter); +EXPORT_SYMBOL(ipipe_critical_exit); +EXPORT_SYMBOL(ipipe_trigger_irq); +EXPORT_SYMBOL(ipipe_get_sysinfo); diff --git a/kernel/ipipe/tracer.c b/kernel/ipipe/tracer.c new file mode 100644 index 0000000..001a83e --- /dev/null +++ b/kernel/ipipe/tracer.c @@ -0,0 +1,1441 @@ +/* -*- linux-c -*- + * kernel/ipipe/tracer.c + * + * Copyright (C) 2005 Luotao Fu. + * 2005-2008 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/version.h> +#include <linux/kallsyms.h> +#include <linux/seq_file.h> +#include <linux/proc_fs.h> +#include <linux/ctype.h> +#include <linux/vmalloc.h> +#include <linux/pid.h> +#include <linux/vermagic.h> +#include <linux/sched.h> +#include <linux/ipipe.h> +#include <linux/ftrace.h> +#include <asm/uaccess.h> + +#define IPIPE_TRACE_PATHS 4 /* <!> Do not lower below 3 */ +#define IPIPE_DEFAULT_ACTIVE 0 +#define IPIPE_DEFAULT_MAX 1 +#define IPIPE_DEFAULT_FROZEN 2 + +#define IPIPE_TRACE_POINTS (1 << CONFIG_IPIPE_TRACE_SHIFT) +#define WRAP_POINT_NO(point) ((point) & (IPIPE_TRACE_POINTS-1)) + +#define IPIPE_DEFAULT_PRE_TRACE 10 +#define IPIPE_DEFAULT_POST_TRACE 10 +#define IPIPE_DEFAULT_BACK_TRACE 100 + +#define IPIPE_DELAY_NOTE 1000 /* in nanoseconds */ +#define IPIPE_DELAY_WARN 10000 /* in nanoseconds */ + +#define IPIPE_TFLG_NMI_LOCK 0x0001 +#define IPIPE_TFLG_NMI_HIT 0x0002 +#define IPIPE_TFLG_NMI_FREEZE_REQ 0x0004 + +#define IPIPE_TFLG_HWIRQ_OFF 0x0100 +#define IPIPE_TFLG_FREEZING 0x0200 +#define IPIPE_TFLG_CURRDOM_SHIFT 10 /* bits 10..11: current domain */ +#define IPIPE_TFLG_CURRDOM_MASK 0x0C00 +#define IPIPE_TFLG_DOMSTATE_SHIFT 12 /* bits 12..15: domain stalled? */ +#define IPIPE_TFLG_DOMSTATE_BITS 3 + +#define IPIPE_TFLG_DOMAIN_STALLED(point, n) \ + (point->flags & (1 << (n + IPIPE_TFLG_DOMSTATE_SHIFT))) +#define IPIPE_TFLG_CURRENT_DOMAIN(point) \ + ((point->flags & IPIPE_TFLG_CURRDOM_MASK) >> IPIPE_TFLG_CURRDOM_SHIFT) + +struct ipipe_trace_point { + short type; + short flags; + unsigned long eip; + unsigned long parent_eip; + unsigned long v; + unsigned long long timestamp; +}; + +struct ipipe_trace_path { + volatile int flags; + int dump_lock; /* separated from flags due to cross-cpu access */ + int trace_pos; /* next point to fill */ + int begin, end; /* finalised path begin and end */ + int post_trace; /* non-zero when in post-trace phase */ + unsigned long long length; /* max path length in cycles */ + unsigned long nmi_saved_eip; /* for deferred requests from NMIs */ + unsigned long nmi_saved_parent_eip; + unsigned long nmi_saved_v; + struct ipipe_trace_point point[IPIPE_TRACE_POINTS]; +} ____cacheline_aligned_in_smp; + +enum ipipe_trace_type +{ + IPIPE_TRACE_FUNC = 0, + IPIPE_TRACE_BEGIN, + IPIPE_TRACE_END, + IPIPE_TRACE_FREEZE, + IPIPE_TRACE_SPECIAL, + IPIPE_TRACE_PID, + IPIPE_TRACE_EVENT, +}; + +#define IPIPE_TYPE_MASK 0x0007 +#define IPIPE_TYPE_BITS 3 + +#ifdef CONFIG_IPIPE_TRACE_VMALLOC +static DEFINE_PER_CPU(struct ipipe_trace_path *, trace_path); +#else /* !CONFIG_IPIPE_TRACE_VMALLOC */ +static DEFINE_PER_CPU(struct ipipe_trace_path, trace_path[IPIPE_TRACE_PATHS]) = + { [0 ... IPIPE_TRACE_PATHS-1] = { .begin = -1, .end = -1 } }; +#endif /* CONFIG_IPIPE_TRACE_VMALLOC */ + +int ipipe_trace_enable = 0; + +static DEFINE_PER_CPU(int, active_path) = { IPIPE_DEFAULT_ACTIVE }; +static DEFINE_PER_CPU(int, max_path) = { IPIPE_DEFAULT_MAX }; +static DEFINE_PER_CPU(int, frozen_path) = { IPIPE_DEFAULT_FROZEN }; +static IPIPE_DEFINE_SPINLOCK(global_path_lock); +static int pre_trace = IPIPE_DEFAULT_PRE_TRACE; +static int post_trace = IPIPE_DEFAULT_POST_TRACE; +static int back_trace = IPIPE_DEFAULT_BACK_TRACE; +static int verbose_trace = 1; +static unsigned long trace_overhead; + +static unsigned long trigger_begin; +static unsigned long trigger_end; + +static DEFINE_MUTEX(out_mutex); +static struct ipipe_trace_path *print_path; +#ifdef CONFIG_IPIPE_TRACE_PANIC +static struct ipipe_trace_path *panic_path; +#endif /* CONFIG_IPIPE_TRACE_PANIC */ +static int print_pre_trace; +static int print_post_trace; + + +static long __ipipe_signed_tsc2us(long long tsc); +static void +__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point); +static void __ipipe_print_symname(struct seq_file *m, unsigned long eip); + + +static notrace void +__ipipe_store_domain_states(struct ipipe_trace_point *point) +{ + struct ipipe_domain *ipd; + struct list_head *pos; + int i = 0; + + list_for_each_prev(pos, &__ipipe_pipeline) { + ipd = list_entry(pos, struct ipipe_domain, p_link); + + if (test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status))) + point->flags |= 1 << (i + IPIPE_TFLG_DOMSTATE_SHIFT); + + if (ipd == __ipipe_current_domain) + point->flags |= i << IPIPE_TFLG_CURRDOM_SHIFT; + + if (++i > IPIPE_TFLG_DOMSTATE_BITS) + break; + } +} + +static notrace int __ipipe_get_free_trace_path(int old, int cpu) +{ + int new_active = old; + struct ipipe_trace_path *tp; + + do { + if (++new_active == IPIPE_TRACE_PATHS) + new_active = 0; + tp = &per_cpu(trace_path, cpu)[new_active]; + } while (new_active == per_cpu(max_path, cpu) || + new_active == per_cpu(frozen_path, cpu) || + tp->dump_lock); + + return new_active; +} + +static notrace void +__ipipe_migrate_pre_trace(struct ipipe_trace_path *new_tp, + struct ipipe_trace_path *old_tp, int old_pos) +{ + int i; + + new_tp->trace_pos = pre_trace+1; + + for (i = new_tp->trace_pos; i > 0; i--) + memcpy(&new_tp->point[WRAP_POINT_NO(new_tp->trace_pos-i)], + &old_tp->point[WRAP_POINT_NO(old_pos-i)], + sizeof(struct ipipe_trace_point)); + + /* mark the end (i.e. the point before point[0]) invalid */ + new_tp->point[IPIPE_TRACE_POINTS-1].eip = 0; +} + +static notrace struct ipipe_trace_path * +__ipipe_trace_end(int cpu, struct ipipe_trace_path *tp, int pos) +{ + struct ipipe_trace_path *old_tp = tp; + long active = per_cpu(active_path, cpu); + unsigned long long length; + + /* do we have a new worst case? */ + length = tp->point[tp->end].timestamp - + tp->point[tp->begin].timestamp; + if (length > per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)].length) { + /* we need protection here against other cpus trying + to start a proc dump */ + spin_lock(&global_path_lock); + + /* active path holds new worst case */ + tp->length = length; + per_cpu(max_path, cpu) = active; + + /* find next unused trace path */ + active = __ipipe_get_free_trace_path(active, cpu); + + spin_unlock(&global_path_lock); + + tp = &per_cpu(trace_path, cpu)[active]; + + /* migrate last entries for pre-tracing */ + __ipipe_migrate_pre_trace(tp, old_tp, pos); + } + + return tp; +} + +static notrace struct ipipe_trace_path * +__ipipe_trace_freeze(int cpu, struct ipipe_trace_path *tp, int pos) +{ + struct ipipe_trace_path *old_tp = tp; + long active = per_cpu(active_path, cpu); + int n; + + /* frozen paths have no core (begin=end) */ + tp->begin = tp->end; + + /* we need protection here against other cpus trying + * to set their frozen path or to start a proc dump */ + spin_lock(&global_path_lock); + + per_cpu(frozen_path, cpu) = active; + + /* find next unused trace path */ + active = __ipipe_get_free_trace_path(active, cpu); + + /* check if this is the first frozen path */ + for_each_possible_cpu(n) { + if (n != cpu && + per_cpu(trace_path, n)[per_cpu(frozen_path, n)].end >= 0) + tp->end = -1; + } + + spin_unlock(&global_path_lock); + + tp = &per_cpu(trace_path, cpu)[active]; + + /* migrate last entries for pre-tracing */ + __ipipe_migrate_pre_trace(tp, old_tp, pos); + + return tp; +} + +void notrace +__ipipe_trace(enum ipipe_trace_type type, unsigned long eip, + unsigned long parent_eip, unsigned long v) +{ + struct ipipe_trace_path *tp, *old_tp; + int pos, next_pos, begin; + struct ipipe_trace_point *point; + unsigned long flags; + int cpu; + + local_irq_save_hw_notrace(flags); + + cpu = ipipe_processor_id(); + restart: + tp = old_tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + /* here starts a race window with NMIs - catched below */ + + /* check for NMI recursion */ + if (unlikely(tp->flags & IPIPE_TFLG_NMI_LOCK)) { + tp->flags |= IPIPE_TFLG_NMI_HIT; + + /* first freeze request from NMI context? */ + if ((type == IPIPE_TRACE_FREEZE) && + !(tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ)) { + /* save arguments and mark deferred freezing */ + tp->flags |= IPIPE_TFLG_NMI_FREEZE_REQ; + tp->nmi_saved_eip = eip; + tp->nmi_saved_parent_eip = parent_eip; + tp->nmi_saved_v = v; + } + return; /* no need for restoring flags inside IRQ */ + } + + /* clear NMI events and set lock (atomically per cpu) */ + tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT | + IPIPE_TFLG_NMI_FREEZE_REQ)) + | IPIPE_TFLG_NMI_LOCK; + + /* check active_path again - some nasty NMI may have switched + * it meanwhile */ + if (unlikely(tp != + &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)])) { + /* release lock on wrong path and restart */ + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* there is no chance that the NMI got deferred + * => no need to check for pending freeze requests */ + goto restart; + } + + /* get the point buffer */ + pos = tp->trace_pos; + point = &tp->point[pos]; + + /* store all trace point data */ + point->type = type; + point->flags = raw_irqs_disabled_flags(flags) ? IPIPE_TFLG_HWIRQ_OFF : 0; + point->eip = eip; + point->parent_eip = parent_eip; + point->v = v; + ipipe_read_tsc(point->timestamp); + + __ipipe_store_domain_states(point); + + /* forward to next point buffer */ + next_pos = WRAP_POINT_NO(pos+1); + tp->trace_pos = next_pos; + + /* only mark beginning if we haven't started yet */ + begin = tp->begin; + if (unlikely(type == IPIPE_TRACE_BEGIN) && (begin < 0)) + tp->begin = pos; + + /* end of critical path, start post-trace if not already started */ + if (unlikely(type == IPIPE_TRACE_END) && + (begin >= 0) && !tp->post_trace) + tp->post_trace = post_trace + 1; + + /* freeze only if the slot is free and we are not already freezing */ + if ((unlikely(type == IPIPE_TRACE_FREEZE) || + (unlikely(eip >= trigger_begin && eip <= trigger_end) && + type == IPIPE_TRACE_FUNC)) && + per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)].begin < 0 && + !(tp->flags & IPIPE_TFLG_FREEZING)) { + tp->post_trace = post_trace + 1; + tp->flags |= IPIPE_TFLG_FREEZING; + } + + /* enforce end of trace in case of overflow */ + if (unlikely(WRAP_POINT_NO(next_pos + 1) == begin)) { + tp->end = pos; + goto enforce_end; + } + + /* stop tracing this path if we are in post-trace and + * a) that phase is over now or + * b) a new TRACE_BEGIN came in but we are not freezing this path */ + if (unlikely((tp->post_trace > 0) && ((--tp->post_trace == 0) || + ((type == IPIPE_TRACE_BEGIN) && + !(tp->flags & IPIPE_TFLG_FREEZING))))) { + /* store the path's end (i.e. excluding post-trace) */ + tp->end = WRAP_POINT_NO(pos - post_trace + tp->post_trace); + + enforce_end: + if (tp->flags & IPIPE_TFLG_FREEZING) + tp = __ipipe_trace_freeze(cpu, tp, pos); + else + tp = __ipipe_trace_end(cpu, tp, pos); + + /* reset the active path, maybe already start a new one */ + tp->begin = (type == IPIPE_TRACE_BEGIN) ? + WRAP_POINT_NO(tp->trace_pos - 1) : -1; + tp->end = -1; + tp->post_trace = 0; + tp->flags = 0; + + /* update active_path not earlier to avoid races with NMIs */ + per_cpu(active_path, cpu) = tp - per_cpu(trace_path, cpu); + } + + /* we still have old_tp and point, + * let's reset NMI lock and check for catches */ + old_tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + if (unlikely(old_tp->flags & IPIPE_TFLG_NMI_HIT)) { + /* well, this late tagging may not immediately be visible for + * other cpus already dumping this path - a minor issue */ + point->flags |= IPIPE_TFLG_NMI_HIT; + + /* handle deferred freezing from NMI context */ + if (old_tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ) + __ipipe_trace(IPIPE_TRACE_FREEZE, old_tp->nmi_saved_eip, + old_tp->nmi_saved_parent_eip, + old_tp->nmi_saved_v); + } + + local_irq_restore_hw_notrace(flags); +} + +static unsigned long __ipipe_global_path_lock(void) +{ + unsigned long flags; + int cpu; + struct ipipe_trace_path *tp; + + spin_lock_irqsave(&global_path_lock, flags); + + cpu = ipipe_processor_id(); + restart: + tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + /* here is small race window with NMIs - catched below */ + + /* clear NMI events and set lock (atomically per cpu) */ + tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT | + IPIPE_TFLG_NMI_FREEZE_REQ)) + | IPIPE_TFLG_NMI_LOCK; + + /* check active_path again - some nasty NMI may have switched + * it meanwhile */ + if (tp != &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]) { + /* release lock on wrong path and restart */ + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* there is no chance that the NMI got deferred + * => no need to check for pending freeze requests */ + goto restart; + } + + return flags; +} + +static void __ipipe_global_path_unlock(unsigned long flags) +{ + int cpu; + struct ipipe_trace_path *tp; + + /* release spinlock first - it's not involved in the NMI issue */ + __ipipe_spin_unlock_irqbegin(&global_path_lock); + + cpu = ipipe_processor_id(); + tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* handle deferred freezing from NMI context */ + if (tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ) + __ipipe_trace(IPIPE_TRACE_FREEZE, tp->nmi_saved_eip, + tp->nmi_saved_parent_eip, tp->nmi_saved_v); + + /* See __ipipe_spin_lock_irqsave() and friends. */ + __ipipe_spin_unlock_irqcomplete(flags); +} + +void notrace ipipe_trace_begin(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_BEGIN, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_begin); + +void notrace ipipe_trace_end(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_END, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_end); + +void notrace ipipe_trace_freeze(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_FREEZE, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_freeze); + +void notrace ipipe_trace_special(unsigned char id, unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_SPECIAL | (id << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_special); + +void notrace ipipe_trace_pid(pid_t pid, short prio) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_PID | (prio << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, pid); +} +EXPORT_SYMBOL(ipipe_trace_pid); + +void notrace ipipe_trace_event(unsigned char id, unsigned long delay_tsc) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_EVENT | (id << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, delay_tsc); +} +EXPORT_SYMBOL(ipipe_trace_event); + +int ipipe_trace_max_reset(void) +{ + int cpu; + unsigned long flags; + struct ipipe_trace_path *path; + int ret = 0; + + flags = __ipipe_global_path_lock(); + + for_each_possible_cpu(cpu) { + path = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)]; + + if (path->dump_lock) { + ret = -EBUSY; + break; + } + + path->begin = -1; + path->end = -1; + path->trace_pos = 0; + path->length = 0; + } + + __ipipe_global_path_unlock(flags); + + return ret; +} +EXPORT_SYMBOL(ipipe_trace_max_reset); + +int ipipe_trace_frozen_reset(void) +{ + int cpu; + unsigned long flags; + struct ipipe_trace_path *path; + int ret = 0; + + flags = __ipipe_global_path_lock(); + + for_each_online_cpu(cpu) { + path = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)]; + + if (path->dump_lock) { + ret = -EBUSY; + break; + } + + path->begin = -1; + path->end = -1; + path->trace_pos = 0; + path->length = 0; + } + + __ipipe_global_path_unlock(flags); + + return ret; +} +EXPORT_SYMBOL(ipipe_trace_frozen_reset); + +static void +__ipipe_get_task_info(char *task_info, struct ipipe_trace_point *point, + int trylock) +{ + struct task_struct *task = NULL; + char buf[8]; + int i; + int locked = 1; + + if (trylock) { + if (!read_trylock(&tasklist_lock)) + locked = 0; + } else + read_lock(&tasklist_lock); + + if (locked) + task = find_task_by_pid_ns((pid_t)point->v, &init_pid_ns); + + if (task) + strncpy(task_info, task->comm, 11); + else + strcpy(task_info, "-<?>-"); + + if (locked) + read_unlock(&tasklist_lock); + + for (i = strlen(task_info); i < 11; i++) + task_info[i] = ' '; + + sprintf(buf, " %d ", point->type >> IPIPE_TYPE_BITS); + strcpy(task_info + (11 - strlen(buf)), buf); +} + +static void +__ipipe_get_event_date(char *buf,struct ipipe_trace_path *path, + struct ipipe_trace_point *point) +{ + long time; + int type; + + time = __ipipe_signed_tsc2us(point->timestamp - + path->point[path->begin].timestamp + point->v); + type = point->type >> IPIPE_TYPE_BITS; + + if (type == 0) + /* + * Event type #0 is predefined, stands for the next + * timer tick. + */ + sprintf(buf, "tick@%-6ld", time); + else + sprintf(buf, "%3d@%-7ld", type, time); +} + +#ifdef CONFIG_IPIPE_TRACE_PANIC +void ipipe_trace_panic_freeze(void) +{ + unsigned long flags; + int cpu; + + if (!ipipe_trace_enable) + return; + + ipipe_trace_enable = 0; + local_irq_save_hw_notrace(flags); + + cpu = ipipe_processor_id(); + + panic_path = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + local_irq_restore_hw(flags); +} +EXPORT_SYMBOL(ipipe_trace_panic_freeze); + +void ipipe_trace_panic_dump(void) +{ + int cnt = back_trace; + int start, pos; + char buf[16]; + + if (!panic_path) + return; + + ipipe_context_check_off(); + + printk("I-pipe tracer log (%d points):\n", cnt); + + start = pos = WRAP_POINT_NO(panic_path->trace_pos-1); + + while (cnt-- > 0) { + struct ipipe_trace_point *point = &panic_path->point[pos]; + long time; + char info[16]; + int i; + + printk(" %c", + (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' '); + + for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--) + printk("%c", + (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ? + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '#' : '+') : + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '*' : ' ')); + + if (!point->eip) + printk("-<invalid>-\n"); + else { + __ipipe_trace_point_type(buf, point); + printk("%s", buf); + + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + printk(" "); + break; + + case IPIPE_TRACE_PID: + __ipipe_get_task_info(info, + point, 1); + printk("%s", info); + break; + + case IPIPE_TRACE_EVENT: + __ipipe_get_event_date(info, + panic_path, point); + printk("%s", info); + break; + + default: + printk("0x%08lx ", point->v); + } + + time = __ipipe_signed_tsc2us(point->timestamp - + panic_path->point[start].timestamp); + printk(" %5ld ", time); + + __ipipe_print_symname(NULL, point->eip); + printk(" ("); + __ipipe_print_symname(NULL, point->parent_eip); + printk(")\n"); + } + pos = WRAP_POINT_NO(pos - 1); + } + + panic_path = NULL; +} +EXPORT_SYMBOL(ipipe_trace_panic_dump); +#endif /* CONFIG_IPIPE_TRACE_PANIC */ + + +/* --- /proc output --- */ + +static notrace int __ipipe_in_critical_trpath(long point_no) +{ + return ((WRAP_POINT_NO(point_no-print_path->begin) < + WRAP_POINT_NO(print_path->end-print_path->begin)) || + ((print_path->end == print_path->begin) && + (WRAP_POINT_NO(point_no-print_path->end) > + print_post_trace))); +} + +static long __ipipe_signed_tsc2us(long long tsc) +{ + unsigned long long abs_tsc; + long us; + + /* ipipe_tsc2us works on unsigned => handle sign separately */ + abs_tsc = (tsc >= 0) ? tsc : -tsc; + us = ipipe_tsc2us(abs_tsc); + if (tsc < 0) + return -us; + else + return us; +} + +static void +__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point) +{ + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + strcpy(buf, "func "); + break; + + case IPIPE_TRACE_BEGIN: + strcpy(buf, "begin "); + break; + + case IPIPE_TRACE_END: + strcpy(buf, "end "); + break; + + case IPIPE_TRACE_FREEZE: + strcpy(buf, "freeze "); + break; + + case IPIPE_TRACE_SPECIAL: + sprintf(buf, "(0x%02x) ", + point->type >> IPIPE_TYPE_BITS); + break; + + case IPIPE_TRACE_PID: + sprintf(buf, "[%5d] ", (pid_t)point->v); + break; + + case IPIPE_TRACE_EVENT: + sprintf(buf, "event "); + break; + } +} + +static void +__ipipe_print_pathmark(struct seq_file *m, struct ipipe_trace_point *point) +{ + char mark = ' '; + int point_no = point - print_path->point; + int i; + + if (print_path->end == point_no) + mark = '<'; + else if (print_path->begin == point_no) + mark = '>'; + else if (__ipipe_in_critical_trpath(point_no)) + mark = ':'; + seq_printf(m, "%c%c", mark, + (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' '); + + if (!verbose_trace) + return; + + for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--) + seq_printf(m, "%c", + (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ? + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '#' : '+') : + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? '*' : ' ')); +} + +static void +__ipipe_print_delay(struct seq_file *m, struct ipipe_trace_point *point) +{ + unsigned long delay = 0; + int next; + char *mark = " "; + + next = WRAP_POINT_NO(point+1 - print_path->point); + + if (next != print_path->trace_pos) + delay = ipipe_tsc2ns(print_path->point[next].timestamp - + point->timestamp); + + if (__ipipe_in_critical_trpath(point - print_path->point)) { + if (delay > IPIPE_DELAY_WARN) + mark = "! "; + else if (delay > IPIPE_DELAY_NOTE) + mark = "+ "; + } + seq_puts(m, mark); + + if (verbose_trace) + seq_printf(m, "%3lu.%03lu%c ", delay/1000, delay%1000, + (point->flags & IPIPE_TFLG_NMI_HIT) ? 'N' : ' '); + else + seq_puts(m, " "); +} + +static void __ipipe_print_symname(struct seq_file *m, unsigned long eip) +{ + char namebuf[KSYM_NAME_LEN+1]; + unsigned long size, offset; + const char *sym_name; + char *modname; + + sym_name = kallsyms_lookup(eip, &size, &offset, &modname, namebuf); + +#ifdef CONFIG_IPIPE_TRACE_PANIC + if (!m) { + /* panic dump */ + if (sym_name) { + printk("%s+0x%lx", sym_name, offset); + if (modname) + printk(" [%s]", modname); + } + } else +#endif /* CONFIG_IPIPE_TRACE_PANIC */ + { + if (sym_name) { + if (verbose_trace) { + seq_printf(m, "%s+0x%lx", sym_name, offset); + if (modname) + seq_printf(m, " [%s]", modname); + } else + seq_puts(m, sym_name); + } else + seq_printf(m, "<%08lx>", eip); + } +} + +static void __ipipe_print_headline(struct seq_file *m) +{ + seq_printf(m, "Calibrated minimum trace-point overhead: %lu.%03lu " + "us\n\n", trace_overhead/1000, trace_overhead%1000); + + if (verbose_trace) { + const char *name[4] = { [0 ... 3] = "<unused>" }; + struct list_head *pos; + int i = 0; + + list_for_each_prev(pos, &__ipipe_pipeline) { + struct ipipe_domain *ipd = + list_entry(pos, struct ipipe_domain, p_link); + + name[i] = ipd->name; + if (++i > 3) + break; + } + + seq_printf(m, + " +----- Hard IRQs ('|': locked)\n" + " |+---- %s\n" + " ||+--- %s\n" + " |||+-- %s\n" + " ||||+- %s%s\n" + " ||||| +---------- " + "Delay flag ('+': > %d us, '!': > %d us)\n" + " ||||| | +- " + "NMI noise ('N')\n" + " ||||| | |\n" + " Type User Val. Time Delay Function " + "(Parent)\n", + name[3], name[2], name[1], name[0], + name[0] ? " ('*': domain stalled, '+': current, " + "'#': current+stalled)" : "", + IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000); + } else + seq_printf(m, + " +--------------- Hard IRQs ('|': locked)\n" + " | +- Delay flag " + "('+': > %d us, '!': > %d us)\n" + " | |\n" + " Type Time Function (Parent)\n", + IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000); +} + +static void *__ipipe_max_prtrace_start(struct seq_file *m, loff_t *pos) +{ + loff_t n = *pos; + + mutex_lock(&out_mutex); + + if (!n) { + struct ipipe_trace_path *tp; + unsigned long length_usecs; + int points, cpu; + unsigned long flags; + + /* protect against max_path/frozen_path updates while we + * haven't locked our target path, also avoid recursively + * taking global_path_lock from NMI context */ + flags = __ipipe_global_path_lock(); + + /* find the longest of all per-cpu paths */ + print_path = NULL; + for_each_online_cpu(cpu) { + tp = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)]; + if ((print_path == NULL) || + (tp->length > print_path->length)) { + print_path = tp; + break; + } + } + print_path->dump_lock = 1; + + __ipipe_global_path_unlock(flags); + + /* does this path actually contain data? */ + if (print_path->end == print_path->begin) + return NULL; + + /* number of points inside the critical path */ + points = WRAP_POINT_NO(print_path->end-print_path->begin+1); + + /* pre- and post-tracing length, post-trace length was frozen + in __ipipe_trace, pre-trace may have to be reduced due to + buffer overrun */ + print_pre_trace = pre_trace; + print_post_trace = WRAP_POINT_NO(print_path->trace_pos - + print_path->end - 1); + if (points+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1) + print_pre_trace = IPIPE_TRACE_POINTS - 1 - points - + print_post_trace; + + length_usecs = ipipe_tsc2us(print_path->length); + seq_printf(m, "I-pipe worst-case tracing service on %s/ipipe-%s\n" + "------------------------------------------------------------\n", + UTS_RELEASE, IPIPE_ARCH_STRING); + seq_printf(m, "CPU: %d, Begin: %lld cycles, Trace Points: " + "%d (-%d/+%d), Length: %lu us\n", + cpu, print_path->point[print_path->begin].timestamp, + points, print_pre_trace, print_post_trace, length_usecs); + __ipipe_print_headline(m); + } + + /* check if we are inside the trace range */ + if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 + + print_pre_trace + print_post_trace)) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin - + print_pre_trace + n)]; +} + +static void *__ipipe_prtrace_next(struct seq_file *m, void *p, loff_t *pos) +{ + loff_t n = ++*pos; + + /* check if we are inside the trace range with the next entry */ + if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 + + print_pre_trace + print_post_trace)) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin - + print_pre_trace + *pos)]; +} + +static void __ipipe_prtrace_stop(struct seq_file *m, void *p) +{ + if (print_path) + print_path->dump_lock = 0; + mutex_unlock(&out_mutex); +} + +static int __ipipe_prtrace_show(struct seq_file *m, void *p) +{ + long time; + struct ipipe_trace_point *point = p; + char buf[16]; + + if (!point->eip) { + seq_puts(m, "-<invalid>-\n"); + return 0; + } + + __ipipe_print_pathmark(m, point); + __ipipe_trace_point_type(buf, point); + seq_puts(m, buf); + if (verbose_trace) + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + seq_puts(m, " "); + break; + + case IPIPE_TRACE_PID: + __ipipe_get_task_info(buf, point, 0); + seq_puts(m, buf); + break; + + case IPIPE_TRACE_EVENT: + __ipipe_get_event_date(buf, print_path, point); + seq_puts(m, buf); + break; + + default: + seq_printf(m, "0x%08lx ", point->v); + } + + time = __ipipe_signed_tsc2us(point->timestamp - + print_path->point[print_path->begin].timestamp); + seq_printf(m, "%5ld", time); + + __ipipe_print_delay(m, point); + __ipipe_print_symname(m, point->eip); + seq_puts(m, " ("); + __ipipe_print_symname(m, point->parent_eip); + seq_puts(m, ")\n"); + + return 0; +} + +static struct seq_operations __ipipe_max_ptrace_ops = { + .start = __ipipe_max_prtrace_start, + .next = __ipipe_prtrace_next, + .stop = __ipipe_prtrace_stop, + .show = __ipipe_prtrace_show +}; + +static int __ipipe_max_prtrace_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &__ipipe_max_ptrace_ops); +} + +static ssize_t +__ipipe_max_reset(struct file *file, const char __user *pbuffer, + size_t count, loff_t *data) +{ + mutex_lock(&out_mutex); + ipipe_trace_max_reset(); + mutex_unlock(&out_mutex); + + return count; +} + +struct file_operations __ipipe_max_prtrace_fops = { + .open = __ipipe_max_prtrace_open, + .read = seq_read, + .write = __ipipe_max_reset, + .llseek = seq_lseek, + .release = seq_release, +}; + +static void *__ipipe_frozen_prtrace_start(struct seq_file *m, loff_t *pos) +{ + loff_t n = *pos; + + mutex_lock(&out_mutex); + + if (!n) { + struct ipipe_trace_path *tp; + int cpu; + unsigned long flags; + + /* protect against max_path/frozen_path updates while we + * haven't locked our target path, also avoid recursively + * taking global_path_lock from NMI context */ + flags = __ipipe_global_path_lock(); + + /* find the first of all per-cpu frozen paths */ + print_path = NULL; + for_each_online_cpu(cpu) { + tp = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)]; + if (tp->end >= 0) { + print_path = tp; + break; + } + } + if (print_path) + print_path->dump_lock = 1; + + __ipipe_global_path_unlock(flags); + + if (!print_path) + return NULL; + + /* back- and post-tracing length, post-trace length was frozen + in __ipipe_trace, back-trace may have to be reduced due to + buffer overrun */ + print_pre_trace = back_trace-1; /* substract freeze point */ + print_post_trace = WRAP_POINT_NO(print_path->trace_pos - + print_path->end - 1); + if (1+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1) + print_pre_trace = IPIPE_TRACE_POINTS - 2 - + print_post_trace; + + seq_printf(m, "I-pipe frozen back-tracing service on %s/ipipe-%s\n" + "------------------------------------------------------" + "------\n", + UTS_RELEASE, IPIPE_ARCH_STRING); + seq_printf(m, "CPU: %d, Freeze: %lld cycles, Trace Points: %d (+%d)\n", + cpu, print_path->point[print_path->begin].timestamp, + print_pre_trace+1, print_post_trace); + __ipipe_print_headline(m); + } + + /* check if we are inside the trace range */ + if (n >= print_pre_trace + 1 + print_post_trace) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin- + print_pre_trace+n)]; +} + +static struct seq_operations __ipipe_frozen_ptrace_ops = { + .start = __ipipe_frozen_prtrace_start, + .next = __ipipe_prtrace_next, + .stop = __ipipe_prtrace_stop, + .show = __ipipe_prtrace_show +}; + +static int __ipipe_frozen_prtrace_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &__ipipe_frozen_ptrace_ops); +} + +static ssize_t +__ipipe_frozen_ctrl(struct file *file, const char __user *pbuffer, + size_t count, loff_t *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, pbuffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + ipipe_trace_frozen_reset(); + if (val > 0) + ipipe_trace_freeze(-1); + mutex_unlock(&out_mutex); + + return count; +} + +struct file_operations __ipipe_frozen_prtrace_fops = { + .open = __ipipe_frozen_prtrace_open, + .read = seq_read, + .write = __ipipe_frozen_ctrl, + .llseek = seq_lseek, + .release = seq_release, +}; + +static int __ipipe_rd_proc_val(char *page, char **start, off_t off, + int count, int *eof, void *data) +{ + int len; + + len = sprintf(page, "%u\n", *(int *)data); + len -= off; + if (len <= off + count) + *eof = 1; + *start = page + off; + if (len > count) + len = count; + if (len < 0) + len = 0; + + return len; +} + +static int __ipipe_wr_proc_val(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, buffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + *(int *)data = val; + mutex_unlock(&out_mutex); + + return count; +} + +static int __ipipe_rd_trigger(char *page, char **start, off_t off, int count, + int *eof, void *data) +{ + int len; + + if (!trigger_begin) + return 0; + + len = sprint_symbol(page, trigger_begin); + page[len++] = '\n'; + + len -= off; + if (len <= off + count) + *eof = 1; + *start = page + off; + if (len > count) + len = count; + if (len < 0) + len = 0; + + return len; +} + +static int __ipipe_wr_trigger(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char buf[KSYM_SYMBOL_LEN]; + unsigned long begin, end; + + if (count > sizeof(buf) - 1) + count = sizeof(buf) - 1; + if (copy_from_user(buf, buffer, count)) + return -EFAULT; + buf[count] = 0; + if (buf[count-1] == '\n') + buf[count-1] = 0; + + begin = kallsyms_lookup_name(buf); + if (!begin || !kallsyms_lookup_size_offset(begin, &end, NULL)) + return -ENOENT; + end += begin - 1; + + mutex_lock(&out_mutex); + /* invalidate the current range before setting a new one */ + trigger_end = 0; + wmb(); + ipipe_trace_frozen_reset(); + + /* set new range */ + trigger_begin = begin; + wmb(); + trigger_end = end; + mutex_unlock(&out_mutex); + + return count; +} + +#ifdef CONFIG_IPIPE_TRACE_MCOUNT +static void notrace +ipipe_trace_function(unsigned long ip, unsigned long parent_ip) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_FUNC, ip, parent_ip, 0); +} + +static struct ftrace_ops ipipe_trace_ops = { + .func = ipipe_trace_function +}; + +static int __ipipe_wr_enable(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, buffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + + if (ipipe_trace_enable) { + if (!val) + unregister_ftrace_function(&ipipe_trace_ops); + } else if (val) + register_ftrace_function(&ipipe_trace_ops); + + ipipe_trace_enable = val; + + mutex_unlock(&out_mutex); + + return count; +} +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ + +extern struct proc_dir_entry *ipipe_proc_root; + +static struct proc_dir_entry * __init +__ipipe_create_trace_proc_val(struct proc_dir_entry *trace_dir, + const char *name, int *value_ptr) +{ + struct proc_dir_entry *entry; + + entry = create_proc_entry(name, 0644, trace_dir); + if (entry) { + entry->data = value_ptr; + entry->read_proc = __ipipe_rd_proc_val; + entry->write_proc = __ipipe_wr_proc_val; + } + return entry; +} + +void __init __ipipe_init_tracer(void) +{ + struct proc_dir_entry *trace_dir; + struct proc_dir_entry *entry; + unsigned long long start, end, min = ULLONG_MAX; + int i; +#ifdef CONFIG_IPIPE_TRACE_VMALLOC + int cpu, path; + + for_each_possible_cpu(cpu) { + struct ipipe_trace_path *tp_buf; + + tp_buf = vmalloc_node(sizeof(struct ipipe_trace_path) * + IPIPE_TRACE_PATHS, cpu_to_node(cpu)); + if (!tp_buf) { + printk(KERN_ERR "I-pipe: " + "insufficient memory for trace buffer.\n"); + return; + } + memset(tp_buf, 0, + sizeof(struct ipipe_trace_path) * IPIPE_TRACE_PATHS); + for (path = 0; path < IPIPE_TRACE_PATHS; path++) { + tp_buf[path].begin = -1; + tp_buf[path].end = -1; + } + per_cpu(trace_path, cpu) = tp_buf; + } +#endif /* CONFIG_IPIPE_TRACE_VMALLOC */ + + /* Calculate minimum overhead of __ipipe_trace() */ + local_irq_disable_hw(); + for (i = 0; i < 100; i++) { + ipipe_read_tsc(start); + __ipipe_trace(IPIPE_TRACE_FUNC, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, 0); + ipipe_read_tsc(end); + + end -= start; + if (end < min) + min = end; + } + local_irq_enable_hw(); + trace_overhead = ipipe_tsc2ns(min); + +#ifdef CONFIG_IPIPE_TRACE_ENABLE + ipipe_trace_enable = 1; +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + register_ftrace_function(&ipipe_trace_ops); +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ +#endif /* CONFIG_IPIPE_TRACE_ENABLE */ + + trace_dir = create_proc_entry("trace", S_IFDIR, ipipe_proc_root); + + entry = create_proc_entry("max", 0644, trace_dir); + if (entry) + entry->proc_fops = &__ipipe_max_prtrace_fops; + + entry = create_proc_entry("frozen", 0644, trace_dir); + if (entry) + entry->proc_fops = &__ipipe_frozen_prtrace_fops; + + entry = create_proc_entry("trigger", 0644, trace_dir); + if (entry) { + entry->read_proc = __ipipe_rd_trigger; + entry->write_proc = __ipipe_wr_trigger; + } + + __ipipe_create_trace_proc_val(trace_dir, "pre_trace_points", + &pre_trace); + __ipipe_create_trace_proc_val(trace_dir, "post_trace_points", + &post_trace); + __ipipe_create_trace_proc_val(trace_dir, "back_trace_points", + &back_trace); + __ipipe_create_trace_proc_val(trace_dir, "verbose", + &verbose_trace); + entry = __ipipe_create_trace_proc_val(trace_dir, "enable", + &ipipe_trace_enable); +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + if (entry) + entry->write_proc = __ipipe_wr_enable; +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ +} diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c index ecc3fa2..e3e4263 100644 --- a/kernel/irq/chip.c +++ b/kernel/irq/chip.c @@ -15,6 +15,7 @@ #include <linux/module.h> #include <linux/interrupt.h> #include <linux/kernel_stat.h> +#include <linux/ipipe.h> #include "internals.h" @@ -425,7 +426,9 @@ handle_level_irq(unsigned int irq, struct irq_desc *desc) irqreturn_t action_ret; raw_spin_lock(&desc->lock); +#ifndef CONFIG_IPIPE mask_ack_irq(desc, irq); +#endif if (unlikely(desc->status & IRQ_INPROGRESS)) goto out_unlock; @@ -505,8 +508,13 @@ handle_fasteoi_irq(unsigned int irq, struct irq_desc *desc) raw_spin_lock(&desc->lock); desc->status &= ~IRQ_INPROGRESS; +#ifdef CONFIG_IPIPE + desc->chip->unmask(irq); +out: +#else out: desc->chip->eoi(irq); +#endif raw_spin_unlock(&desc->lock); } @@ -548,8 +556,10 @@ handle_edge_irq(unsigned int irq, struct irq_desc *desc) kstat_incr_irqs_this_cpu(irq, desc); /* Start handling the irq */ +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* Mark the IRQ currently in progress.*/ desc->status |= IRQ_INPROGRESS; @@ -603,8 +613,10 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc) kstat_incr_irqs_this_cpu(irq, desc); +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* CONFIG_IPIPE */ action_ret = handle_IRQ_event(irq, desc->action); if (!noirqdebug) @@ -614,6 +626,134 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc) desc->chip->eoi(irq); } +#ifdef CONFIG_IPIPE + +void __ipipe_ack_simple_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_end_simple_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_ack_level_irq(unsigned irq, struct irq_desc *desc) +{ + mask_ack_irq(desc, irq); +} + +void __ipipe_end_level_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->unmask) + desc->chip->unmask(irq); +} + +void __ipipe_ack_fasteoi_irq(unsigned irq, struct irq_desc *desc) +{ + desc->chip->eoi(irq); +} + +void __ipipe_end_fasteoi_irq(unsigned irq, struct irq_desc *desc) +{ + /* + * Non-requestable IRQs should not be masked in EOI handler. + */ + if (!(desc->status & IRQ_NOREQUEST)) + desc->chip->unmask(irq); +} + +void __ipipe_ack_edge_irq(unsigned irq, struct irq_desc *desc) +{ + desc->chip->ack(irq); +} + +void __ipipe_ack_percpu_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->ack) + desc->chip->ack(irq); +} + +void __ipipe_end_percpu_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->eoi) + desc->chip->eoi(irq); +} + +void __ipipe_end_edge_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_ack_bad_irq(unsigned irq, struct irq_desc *desc) +{ + static int done; + + handle_bad_irq(irq, desc); + + if (!done) { + printk(KERN_WARNING "%s: unknown flow handler for IRQ %d\n", + __FUNCTION__, irq); + done = 1; + } +} + +void __ipipe_noack_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_noend_irq(unsigned irq, struct irq_desc *desc) +{ +} + +irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, int is_chained) +{ + if (unlikely(handle == NULL)) { + desc->ipipe_ack = &__ipipe_ack_bad_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } else { + if (is_chained) { + desc->ipipe_ack = handle; + desc->ipipe_end = &__ipipe_noend_irq; + handle = __ipipe_noack_irq; + } else if (handle == &handle_simple_irq) { + desc->ipipe_ack = &__ipipe_ack_simple_irq; + desc->ipipe_end = &__ipipe_end_simple_irq; + } else if (handle == &handle_level_irq) { + desc->ipipe_ack = &__ipipe_ack_level_irq; + desc->ipipe_end = &__ipipe_end_level_irq; + } else if (handle == &handle_edge_irq) { + desc->ipipe_ack = &__ipipe_ack_edge_irq; + desc->ipipe_end = &__ipipe_end_edge_irq; + } else if (handle == &handle_fasteoi_irq) { + desc->ipipe_ack = &__ipipe_ack_fasteoi_irq; + desc->ipipe_end = &__ipipe_end_fasteoi_irq; + } else if (handle == &handle_percpu_irq) { + desc->ipipe_ack = &__ipipe_ack_percpu_irq; + desc->ipipe_end = &__ipipe_end_percpu_irq; + } else if (desc->chip == &no_irq_chip) { + desc->ipipe_ack = &__ipipe_noack_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } else { + desc->ipipe_ack = &__ipipe_ack_bad_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } + } + + /* Suppress intermediate trampoline routine. */ + ipipe_root_domain->irqs[desc->irq].acknowledge = desc->ipipe_ack; + + return handle; +} + +#else /* !CONFIG_IPIPE */ + +irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, int is_chained) +{ + return handle; +} + +#endif /* !CONFIG_IPIPE */ + void __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, const char *name) @@ -645,6 +785,8 @@ __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, chip_bus_lock(irq, desc); raw_spin_lock_irqsave(&desc->lock, flags); + handle = __fixup_irq_handler(desc, handle, is_chained); + /* Uninstall? */ if (handle == handle_bad_irq) { if (desc->chip != &no_irq_chip) diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c index 814940e..4c2e41d 100644 --- a/kernel/irq/handle.c +++ b/kernel/irq/handle.c @@ -462,8 +462,10 @@ unsigned int __do_IRQ(unsigned int irq) /* * No locking required for CPU-local interrupts: */ +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif if (likely(!(desc->status & IRQ_DISABLED))) { action_ret = handle_IRQ_event(irq, desc->action); if (!noirqdebug) @@ -474,8 +476,10 @@ unsigned int __do_IRQ(unsigned int irq) } raw_spin_lock(&desc->lock); +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* * REPLAY is when Linux resends an IRQ that was dropped earlier * WAITING is used by probe to mark irqs that are being tested diff --git a/kernel/lockdep.c b/kernel/lockdep.c index c62ec14..6d41b73 100644 --- a/kernel/lockdep.c +++ b/kernel/lockdep.c @@ -2327,7 +2327,7 @@ void trace_hardirqs_on_caller(unsigned long ip) /* we'll do an OFF -> ON transition: */ curr->hardirqs_enabled = 1; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (DEBUG_LOCKS_WARN_ON(current->hardirq_context)) return; @@ -2370,7 +2370,7 @@ void trace_hardirqs_off_caller(unsigned long ip) if (unlikely(!debug_locks || current->lockdep_recursion)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->hardirqs_enabled) { @@ -2402,7 +2402,7 @@ void trace_softirqs_on(unsigned long ip) if (unlikely(!debug_locks)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->softirqs_enabled) { @@ -2436,7 +2436,7 @@ void trace_softirqs_off(unsigned long ip) if (unlikely(!debug_locks)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->softirqs_enabled) { diff --git a/kernel/panic.c b/kernel/panic.c index c787333..ba50967 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -23,6 +23,7 @@ #include <linux/init.h> #include <linux/nmi.h> #include <linux/dmi.h> +#include <linux/ipipe_trace.h> int panic_on_oops; static unsigned long tainted_mask; @@ -307,6 +308,8 @@ void oops_enter(void) { tracing_off(); /* can't trust the integrity of the kernel anymore: */ + ipipe_trace_panic_freeze(); + ipipe_disable_context_check(ipipe_processor_id()); debug_locks_off(); do_oops_enter_exit(); } diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c index bbfe472..be44776 100644 --- a/kernel/power/hibernate.c +++ b/kernel/power/hibernate.c @@ -268,6 +268,7 @@ static int create_image(int platform_mode) goto Enable_cpus; local_irq_disable(); + local_irq_disable_hw_cond(); error = sysdev_suspend(PMSG_FREEZE); if (error) { @@ -297,6 +298,7 @@ static int create_image(int platform_mode) */ Enable_irqs: + local_irq_enable_hw_cond(); local_irq_enable(); Enable_cpus: @@ -389,6 +391,7 @@ static int resume_target_kernel(bool platform_mode) goto Enable_cpus; local_irq_disable(); + local_irq_disable_hw_cond(); error = sysdev_suspend(PMSG_QUIESCE); if (error) @@ -420,6 +423,7 @@ static int resume_target_kernel(bool platform_mode) sysdev_resume(); Enable_irqs: + local_irq_enable_hw_cond(); local_irq_enable(); Enable_cpus: @@ -501,6 +505,7 @@ int hibernation_platform_enter(void) goto Platform_finish; local_irq_disable(); + local_irq_disable_hw_cond(); sysdev_suspend(PMSG_HIBERNATE); hibernation_ops->enter(); /* We should never get here */ diff --git a/kernel/printk.c b/kernel/printk.c index 1751c45..a1e027a 100644 --- a/kernel/printk.c +++ b/kernel/printk.c @@ -566,6 +566,41 @@ static int have_callable_console(void) return 0; } +#ifdef CONFIG_IPIPE + +static IPIPE_DEFINE_SPINLOCK(__ipipe_printk_lock); + +static int __ipipe_printk_fill; + +static char __ipipe_printk_buf[__LOG_BUF_LEN]; + +void __ipipe_flush_printk (unsigned virq, void *cookie) +{ + char *p = __ipipe_printk_buf; + int len, lmax, out = 0; + unsigned long flags; + + goto start; + + do { + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); + start: + lmax = __ipipe_printk_fill; + while (out < lmax) { + len = strlen(p) + 1; + printk("%s",p); + p += len; + out += len; + } + spin_lock_irqsave(&__ipipe_printk_lock, flags); + } + while (__ipipe_printk_fill != lmax); + + __ipipe_printk_fill = 0; + + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); +} + /** * printk - print a kernel message * @fmt: format string @@ -590,6 +625,65 @@ static int have_callable_console(void) asmlinkage int printk(const char *fmt, ...) { + int r, fbytes, oldcount; + unsigned long flags; + int sprintk = 1; + int cs = -1; + va_list args; + + va_start(args, fmt); + + local_irq_save_hw(flags); + + if (test_bit(IPIPE_SPRINTK_FLAG, &__ipipe_current_domain->flags) || + oops_in_progress) + cs = ipipe_disable_context_check(ipipe_processor_id()); + else if (__ipipe_current_domain == ipipe_root_domain) { + struct ipipe_domain *dom; + + list_for_each_entry(dom, &__ipipe_pipeline, p_link) { + if (dom == ipipe_root_domain) + break; + if (test_bit(IPIPE_STALL_FLAG, + &ipipe_cpudom_var(dom, status))) + sprintk = 0; + } + } else + sprintk = 0; + + local_irq_restore_hw(flags); + + if (sprintk) { + r = vprintk(fmt, args); + if (cs != -1) + ipipe_restore_context_check(ipipe_processor_id(), cs); + goto out; + } + + spin_lock_irqsave(&__ipipe_printk_lock, flags); + + oldcount = __ipipe_printk_fill; + fbytes = __LOG_BUF_LEN - oldcount; + + if (fbytes > 1) { + r = vscnprintf(__ipipe_printk_buf + __ipipe_printk_fill, + fbytes, fmt, args) + 1; /* account for the null byte */ + __ipipe_printk_fill += r; + } else + r = 0; + + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); + + if (oldcount == 0) + ipipe_trigger_irq(__ipipe_printk_virq); +out: + va_end(args); + + return r; +} +#else /* !CONFIG_IPIPE */ +asmlinkage int printk(const char *fmt, ...) +{ va_list args; int r; @@ -599,6 +693,7 @@ asmlinkage int printk(const char *fmt, ...) return r; } +#endif /* CONFIG_IPIPE */ /* cpu currently holding logbuf_lock */ static volatile unsigned int printk_cpu = UINT_MAX; diff --git a/kernel/sched.c b/kernel/sched.c index 3a8fb30..cdb5bfd 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -2368,6 +2368,7 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) { int cpu, orig_cpu, this_cpu, success = 0; + unsigned int old_state; unsigned long flags; struct rq *rq, *orig_rq; @@ -2379,7 +2380,9 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, smp_wmb(); rq = orig_rq = task_rq_lock(p, &flags); update_rq_clock(rq); - if (!(p->state & state)) + old_state = p->state; + if (!(old_state & state) || + (old_state & (TASK_NOWAKEUP|TASK_ATOMICSWITCH))) goto out; if (p->se.on_rq) @@ -2869,13 +2872,15 @@ asmlinkage void schedule_tail(struct task_struct *prev) #endif if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); + + ipipe_init_notify(current); } /* * context_switch - switch to the new MM and the new * thread's register state. */ -static inline void +static inline int context_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { @@ -2917,12 +2922,23 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_to(prev, next, prev); barrier(); + +#ifdef CONFIG_IPIPE_DELAYED_ATOMICSW + current->state &= ~TASK_ATOMICSWITCH; +#else + prev->state &= ~TASK_ATOMICSWITCH; +#endif + if (task_hijacked(prev)) + return 1; + /* * this_rq must be evaluated again because prev may have moved * CPUs since it called schedule(), thus the 'rq' on its stack * frame will be invalid. */ finish_task_switch(this_rq(), prev); + + return 0; } /* @@ -5332,6 +5348,7 @@ notrace unsigned long get_parent_ip(unsigned long addr) void __kprobes add_preempt_count(int val) { + ipipe_check_context(ipipe_root_domain); #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? @@ -5354,6 +5371,7 @@ EXPORT_SYMBOL(add_preempt_count); void __kprobes sub_preempt_count(int val) { + ipipe_check_context(ipipe_root_domain); #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? @@ -5402,6 +5420,7 @@ static noinline void __schedule_bug(struct task_struct *prev) */ static inline void schedule_debug(struct task_struct *prev) { + ipipe_check_context(ipipe_root_domain); /* * Test if we are atomic. Since do_exit() needs to call into * schedule() atomically, we ignore that path for now. @@ -5478,7 +5497,7 @@ pick_next_task(struct rq *rq) /* * schedule() is the main scheduler function. */ -asmlinkage void __sched schedule(void) +asmlinkage int __sched schedule(void) { struct task_struct *prev, *next; unsigned long *switch_count; @@ -5492,6 +5511,9 @@ need_resched: rcu_sched_qs(cpu); prev = rq->curr; switch_count = &prev->nivcsw; + if (unlikely(prev->state & TASK_ATOMICSWITCH)) + /* Pop one disable level -- one still remains. */ + preempt_enable(); release_kernel_lock(prev); need_resched_nonpreemptible: @@ -5529,15 +5551,18 @@ need_resched_nonpreemptible: rq->curr = next; ++*switch_count; - context_switch(rq, prev, next); /* unlocks the rq */ + if (context_switch(rq, prev, next)) /* unlocks the rq */ + return 1; /* task hijacked by higher domain */ /* * the context switch might have flipped the stack from under * us, hence refresh the local variables. */ cpu = smp_processor_id(); rq = cpu_rq(cpu); - } else + } else { + prev->state &= ~TASK_ATOMICSWITCH; raw_spin_unlock_irq(&rq->lock); + } post_schedule(rq); @@ -5550,6 +5575,8 @@ need_resched_nonpreemptible: preempt_enable_no_resched(); if (need_resched()) goto need_resched; + + return 0; } EXPORT_SYMBOL(schedule); @@ -5633,7 +5660,8 @@ asmlinkage void __sched preempt_schedule(void) do { add_preempt_count(PREEMPT_ACTIVE); - schedule(); + if (schedule()) + return; sub_preempt_count(PREEMPT_ACTIVE); /* @@ -6396,6 +6424,7 @@ recheck: oldprio = p->prio; __setscheduler(rq, p, policy, param->sched_priority); + ipipe_setsched_notify(p); if (running) p->sched_class->set_curr_task(rq); @@ -7048,6 +7077,7 @@ void __cpuinit init_idle(struct task_struct *idle, int cpu) #else task_thread_info(idle)->preempt_count = 0; #endif + ipipe_check_context(ipipe_root_domain); /* * The idle tasks have their own, simple scheduling class: */ @@ -11039,3 +11069,64 @@ void synchronize_sched_expedited(void) EXPORT_SYMBOL_GPL(synchronize_sched_expedited); #endif /* #else #ifndef CONFIG_SMP */ + +#ifdef CONFIG_IPIPE + +int ipipe_setscheduler_root(struct task_struct *p, int policy, int prio) +{ + const struct sched_class *prev_class = p->sched_class; + int oldprio, on_rq, running; + unsigned long flags; + struct rq *rq; + + raw_spin_lock_irqsave(&p->pi_lock, flags); + rq = __task_rq_lock(p); + update_rq_clock(rq); + on_rq = p->se.on_rq; + running = task_current(rq, p); + if (on_rq) + deactivate_task(rq, p, 0); + if (running) + p->sched_class->put_prev_task(rq, p); + + p->sched_reset_on_fork = 0; + + oldprio = p->prio; + __setscheduler(rq, p, policy, prio); + ipipe_setsched_notify(p); + + if (running) + p->sched_class->set_curr_task(rq); + if (on_rq) { + activate_task(rq, p, 0); + + check_class_changed(rq, p, prev_class, oldprio, running); + } + __task_rq_unlock(rq); + raw_spin_unlock_irqrestore(&p->pi_lock, flags); + + rt_mutex_adjust_pi(p); + + return 0; +} +EXPORT_SYMBOL_GPL(ipipe_setscheduler_root); + +int ipipe_reenter_root(struct task_struct *prev, int policy, int prio) +{ + struct rq *rq = this_rq(); + + finish_task_switch(rq, prev); + + post_schedule(rq); + + (void)reacquire_kernel_lock(current); + preempt_enable_no_resched(); + + if (current->policy != policy || current->rt_priority != prio) + return ipipe_setscheduler_root(current, policy, prio); + + return 0; +} +EXPORT_SYMBOL_GPL(ipipe_reenter_root); + +#endif /* CONFIG_IPIPE */ diff --git a/kernel/signal.c b/kernel/signal.c index 934ae5e..7181e5f 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -541,6 +541,7 @@ void signal_wake_up(struct task_struct *t, int resume) unsigned int mask; set_tsk_thread_flag(t, TIF_SIGPENDING); + ipipe_sigwake_notify(t); /* TIF_SIGPENDING must be set first. */ /* * For SIGKILL, we want to wake it up in the stopped/traced/killable diff --git a/kernel/spinlock.c b/kernel/spinlock.c index be6517f..862aed4 100644 --- a/kernel/spinlock.c +++ b/kernel/spinlock.c @@ -26,7 +26,9 @@ * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) /* * The __lock_function inlines are taken from * include/linux/spinlock_api_smp.h diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c index b6b898d..75c2031 100644 --- a/kernel/time/tick-common.c +++ b/kernel/time/tick-common.c @@ -69,7 +69,7 @@ static void tick_periodic(int cpu) write_sequnlock(&xtime_lock); } - update_process_times(user_mode(get_irq_regs())); + update_root_process_times(get_irq_regs()); profile_tick(CPU_PROFILING); } @@ -177,6 +177,10 @@ static void tick_setup_device(struct tick_device *td, td->evtdev = newdev; + /* I-pipe: derive global tick IRQ from CPU 0 */ + if (cpu == 0) + ipipe_update_tick_evtdev(newdev); + /* * When the device is not per cpu, pin the interrupt to the * current cpu: diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index f992762..a9314ea 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -551,7 +551,7 @@ static void tick_nohz_handler(struct clock_event_device *dev) ts->idle_jiffies++; } - update_process_times(user_mode(regs)); + update_root_process_times(regs); profile_tick(CPU_PROFILING); while (tick_nohz_reprogram(ts, now)) { @@ -711,7 +711,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer) touch_softlockup_watchdog(); ts->idle_jiffies++; } - update_process_times(user_mode(regs)); + update_root_process_times(regs); profile_tick(CPU_PROFILING); } diff --git a/kernel/timer.c b/kernel/timer.c index c61a794..fb57c0c 100644 --- a/kernel/timer.c +++ b/kernel/timer.c @@ -1203,6 +1203,25 @@ void update_process_times(int user_tick) run_posix_cpu_timers(p); } +#ifdef CONFIG_IPIPE + +void update_root_process_times(struct pt_regs *regs) +{ + int cpu, user_tick = user_mode(regs); + + if (__ipipe_root_tick_p(regs)) { + update_process_times(user_tick); + return; + } + + run_local_timers(); + cpu = smp_processor_id(); + rcu_check_callbacks(cpu, user_tick); + run_posix_cpu_timers(current); +} + +#endif + /* * This function runs timers and the timer-tq in bottom half context. */ diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 60e2ce0..5b7a125 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -468,6 +468,7 @@ config DYNAMIC_FTRACE bool "enable/disable ftrace tracepoints dynamically" depends on FUNCTION_TRACER depends on HAVE_DYNAMIC_FTRACE + depends on !IPIPE_TRACE_MCOUNT default y help This option will modify all the calls to ftrace dynamically diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 1e6640f..5a3aa6b 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -28,6 +28,7 @@ #include <linux/ctype.h> #include <linux/list.h> #include <linux/hash.h> +#include <linux/ipipe.h> #include <trace/events/sched.h> @@ -1151,6 +1152,9 @@ static int __ftrace_modify_code(void *data) static void ftrace_run_update_code(int command) { +#ifdef CONFIG_IPIPE + unsigned long flags; +#endif /* CONFIG_IPIPE */ int ret; ret = ftrace_arch_code_modify_prepare(); @@ -1158,7 +1162,13 @@ static void ftrace_run_update_code(int command) if (ret) return; +#ifdef CONFIG_IPIPE + flags = ipipe_critical_enter(NULL); + __ftrace_modify_code(&command); + ipipe_critical_exit(flags); +#else /* !CONFIG_IPIPE */ stop_machine(__ftrace_modify_code, &command, NULL); +#endif /* !CONFIG_IPIPE */ ret = ftrace_arch_code_modify_post_process(); FTRACE_WARN_ON(ret); @@ -2663,9 +2673,9 @@ static int ftrace_process_locs(struct module *mod, } /* disable interrupts to prevent kstop machine */ - local_irq_save(flags); + local_irq_save_hw_notrace(flags); ftrace_update_code(mod); - local_irq_restore(flags); + local_irq_restore_hw_notrace(flags); mutex_unlock(&ftrace_lock); return 0; @@ -2744,9 +2754,9 @@ void __init ftrace_init(void) /* Keep the ftrace pointer to the stub */ addr = (unsigned long)ftrace_stub; - local_irq_save(flags); + local_irq_save_hw_notrace(flags); ftrace_dyn_arch_init(&addr); - local_irq_restore(flags); + local_irq_restore_hw_notrace(flags); /* ftrace_dyn_arch_init places the return code in addr */ if (addr) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 25c3ed5..6b9a5d0 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -136,6 +136,8 @@ config DEBUG_SECTION_MISMATCH - Enable verbose reporting from modpost to help solving the section mismatches reported. +source "kernel/ipipe/Kconfig.debug" + config DEBUG_KERNEL bool "Kernel debugging" help diff --git a/lib/bust_spinlocks.c b/lib/bust_spinlocks.c index 9681d54..2dba50c 100644 --- a/lib/bust_spinlocks.c +++ b/lib/bust_spinlocks.c @@ -13,6 +13,7 @@ #include <linux/wait.h> #include <linux/vt_kern.h> #include <linux/console.h> +#include <linux/ipipe_trace.h> void __attribute__((weak)) bust_spinlocks(int yes) @@ -24,6 +25,7 @@ void __attribute__((weak)) bust_spinlocks(int yes) unblank_screen(); #endif console_unblank(); + ipipe_trace_panic_dump(); if (--oops_in_progress == 0) wake_up_klogd(); } diff --git a/lib/ioremap.c b/lib/ioremap.c index 14c6078..a275469 100644 --- a/lib/ioremap.c +++ b/lib/ioremap.c @@ -85,8 +85,8 @@ int ioremap_page_range(unsigned long addr, if (err) break; } while (pgd++, addr = next, addr != end); - - flush_cache_vmap(start, end); + __ipipe_pin_range_globally(start, end); + flush_cache_vmap(start, end); return err; } diff --git a/lib/smp_processor_id.c b/lib/smp_processor_id.c index 4689cb0..3d12764 100644 --- a/lib/smp_processor_id.c +++ b/lib/smp_processor_id.c @@ -12,10 +12,13 @@ notrace unsigned int debug_smp_processor_id(void) unsigned long preempt_count = preempt_count(); int this_cpu = raw_smp_processor_id(); + if (!ipipe_root_domain_p) + goto out; + if (likely(preempt_count)) goto out; - if (irqs_disabled()) + if (irqs_disabled() || irqs_disabled_hw()) goto out; /* diff --git a/mm/memory.c b/mm/memory.c index 09e4b1b..bef0089 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -56,6 +56,7 @@ #include <linux/kallsyms.h> #include <linux/swapops.h> #include <linux/elf.h> +#include <linux/vmalloc.h> #include <asm/io.h> #include <asm/pgalloc.h> @@ -566,6 +567,32 @@ out: return pfn_to_page(pfn); } +static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) +{ + /* + * If the source page was a PFN mapping, we don't have + * a "struct page" for it. We do a best-effort copy by + * just copying from the original user address. If that + * fails, we just zero-fill it. Live with it. + */ + if (unlikely(!src)) { + void *kaddr = kmap_atomic(dst, KM_USER0); + void __user *uaddr = (void __user *)(va & PAGE_MASK); + + /* + * This really shouldn't fail, because the page is there + * in the page tables. But it might just be unreadable, + * in which case we just give up and fill the result with + * zeroes. + */ + if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) + memset(kaddr, 0, PAGE_SIZE); + kunmap_atomic(kaddr, KM_USER0); + flush_dcache_page(dst); + } else + copy_user_highpage(dst, src, va, vma); +} + /* * copy one vm_area from one task to the other. Assumes the page tables * already present in the new task to be cleared in the whole range @@ -574,8 +601,8 @@ out: static inline unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, - pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, - unsigned long addr, int *rss) + pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, + unsigned long addr, int *rss, struct page *uncow_page) { unsigned long vm_flags = vma->vm_flags; pte_t pte = *src_pte; @@ -616,6 +643,21 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * in the parent and the child */ if (is_cow_mapping(vm_flags)) { +#ifdef CONFIG_IPIPE + if (uncow_page) { + struct page *old_page = vm_normal_page(vma, addr, pte); + cow_user_page(uncow_page, old_page, addr, vma); + pte = mk_pte(uncow_page, vma->vm_page_prot); + + if (vm_flags & VM_SHARED) + pte = pte_mkclean(pte); + pte = pte_mkold(pte); + + page_add_new_anon_rmap(uncow_page, vma, addr); + rss[!!PageAnon(uncow_page)]++; + goto out_set_pte; + } +#endif /* CONFIG_IPIPE */ ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -650,12 +692,26 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, int progress = 0; int rss[2]; swp_entry_t entry = (swp_entry_t){0}; - + struct page *uncow_page = NULL; +#ifdef CONFIG_IPIPE + int do_cow_break = 0; +again: + if (do_cow_break) { + uncow_page = alloc_page_vma(GFP_HIGHUSER, vma, addr); + if (uncow_page == NULL) + return -ENOMEM; + do_cow_break = 0; + } +#else again: rss[1] = rss[0] = 0; +#endif dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); - if (!dst_pte) + if (!dst_pte) { + if (uncow_page) + page_cache_release(uncow_page); return -ENOMEM; + } src_pte = pte_offset_map_nested(src_pmd, addr); src_ptl = pte_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -678,8 +734,25 @@ again: progress++; continue; } +#ifdef CONFIG_IPIPE + if (likely(uncow_page == NULL) && likely(pte_present(*src_pte))) { + if (is_cow_mapping(vma->vm_flags) && + test_bit(MMF_VM_PINNED, &src_mm->flags) && + ((vma->vm_flags|src_mm->def_flags) & VM_LOCKED)) { + arch_leave_lazy_mmu_mode(); + spin_unlock(src_ptl); + pte_unmap_nested(src_pte); + add_mm_rss(dst_mm, rss[0], rss[1]); + pte_unmap_unlock(dst_pte, dst_ptl); + cond_resched(); + do_cow_break = 1; + goto again; + } + } +#endif entry.val = copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, - vma, addr, rss); + vma, addr, rss, uncow_page); + uncow_page = NULL; if (entry.val) break; progress += 8; @@ -1956,32 +2029,6 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) return pte; } -static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) -{ - /* - * If the source page was a PFN mapping, we don't have - * a "struct page" for it. We do a best-effort copy by - * just copying from the original user address. If that - * fails, we just zero-fill it. Live with it. - */ - if (unlikely(!src)) { - void *kaddr = kmap_atomic(dst, KM_USER0); - void __user *uaddr = (void __user *)(va & PAGE_MASK); - - /* - * This really shouldn't fail, because the page is there - * in the page tables. But it might just be unreadable, - * in which case we just give up and fill the result with - * zeroes. - */ - if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) - memset(kaddr, 0, PAGE_SIZE); - kunmap_atomic(kaddr, KM_USER0); - flush_dcache_page(dst); - } else - copy_user_highpage(dst, src, va, vma); -} - /* * This routine handles present pages, when users try to write * to a shared page. It is done by copying the page to a new address @@ -3402,3 +3449,111 @@ void might_fault(void) } EXPORT_SYMBOL(might_fault); #endif + +#ifdef CONFIG_IPIPE + +static inline int ipipe_pin_pte_range(struct mm_struct *mm, pmd_t *pmd, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + spinlock_t *ptl; + pte_t *pte; + + do { + pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + continue; + + if (!pte_present(*pte) || pte_write(*pte)) { + pte_unmap_unlock(pte, ptl); + continue; + } + + if (do_wp_page(mm, vma, addr, pte, pmd, ptl, *pte) == VM_FAULT_OOM) + return -ENOMEM; + } while (addr += PAGE_SIZE, addr != end); + return 0; +} + +static inline int ipipe_pin_pmd_range(struct mm_struct *mm, pud_t *pud, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + unsigned long next; + pmd_t *pmd; + + pmd = pmd_offset(pud, addr); + do { + next = pmd_addr_end(addr, end); + if (pmd_none_or_clear_bad(pmd)) + continue; + if (ipipe_pin_pte_range(mm, pmd, vma, addr, next)) + return -ENOMEM; + } while (pmd++, addr = next, addr != end); + return 0; +} + +static inline int ipipe_pin_pud_range(struct mm_struct *mm, pgd_t *pgd, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + unsigned long next; + pud_t *pud; + + pud = pud_offset(pgd, addr); + do { + next = pud_addr_end(addr, end); + if (pud_none_or_clear_bad(pud)) + continue; + if (ipipe_pin_pmd_range(mm, pud, vma, addr, next)) + return -ENOMEM; + } while (pud++, addr = next, addr != end); + return 0; +} + +int ipipe_disable_ondemand_mappings(struct task_struct *tsk) +{ + unsigned long addr, next, end; + struct vm_area_struct *vma; + struct mm_struct *mm; + int result = 0; + pgd_t *pgd; + + mm = get_task_mm(tsk); + if (!mm) + return -EPERM; + + down_write(&mm->mmap_sem); + if (test_bit(MMF_VM_PINNED, &mm->flags)) + goto done_mm; + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (!is_cow_mapping(vma->vm_flags) + || !(vma->vm_flags & VM_WRITE)) + continue; + + addr = vma->vm_start; + end = vma->vm_end; + + pgd = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none_or_clear_bad(pgd)) + continue; + if (ipipe_pin_pud_range(mm, pgd, vma, addr, next)) { + result = -ENOMEM; + goto done_mm; + } + } while (pgd++, addr = next, addr != end); + } + set_bit(MMF_VM_PINNED, &mm->flags); + + done_mm: + up_write(&mm->mmap_sem); + mmput(mm); + return result; +} + +EXPORT_SYMBOL(ipipe_disable_ondemand_mappings); + +#endif diff --git a/mm/mmu_context.c b/mm/mmu_context.c index ded9081..cb2ac0e 100644 --- a/mm/mmu_context.c +++ b/mm/mmu_context.c @@ -23,15 +23,18 @@ void use_mm(struct mm_struct *mm) { struct mm_struct *active_mm; struct task_struct *tsk = current; + unsigned long flags; task_lock(tsk); active_mm = tsk->active_mm; + ipipe_mm_switch_protect(flags); if (active_mm != mm) { atomic_inc(&mm->mm_count); tsk->active_mm = mm; } tsk->mm = mm; - switch_mm(active_mm, mm, tsk); + __switch_mm(active_mm, mm, tsk); + ipipe_mm_switch_unprotect(flags); task_unlock(tsk); if (active_mm != mm) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ae00746..ca1597c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -172,6 +172,8 @@ static int vmap_page_range_noflush(unsigned long start, unsigned long end, return err; } while (pgd++, addr = next, addr != end); + __ipipe_pin_range_globally(start, end); + return nr; } ^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 8:59 ` Fredrik Asplund @ 2010-07-01 9:54 ` Gilles Chanteperdrix 2010-07-01 10:16 ` Fredrik Asplund 0 siblings, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-01 9:54 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org Fredrik Asplund wrote: >>> I am actually trying to do the exact same thing. >> Looking at forums and email archives it seems a lot of >> people have been trying to do this, but that no-one has posted their results. :/ >> >>> Anyway, I've managed to change the files manually - the changes seem >>> very minor (the changes in include/asm/cacheflush.h just move a >>> function elsewhere, no code changes at all). >> I have attached a patch file showing the changes I have done (and an updated Adeos patch file). This was >> fairly trivial, but I guess the problem could be if Gumstix have added some changes which will require >> additional patching for Adeos. Anyway, could you share a patch file of exactly what you have changed? >> >>> So, any clues? I can send my changes and logs tomorrow morning, if they are of any help. >> Sure, please do so. I don´t have much time to put on this, so I have actually only have >> had to go through the source files since yesterday. Today (or during the weekend if I don´t >> get the time) I will create a new defconfig file, compile, test and debug. If you have >> updated BitBake file, Defconfig file, Logs, etc., please share them. > > Sorry, > I attached the wrong updated Xenomai Patch file, this is the one I use. > / Fredrik Ok. Next time you post this patch, could you give the attached file a less confusing name? Such as adeos-ipipe-2.6.33-arm-1.16-01-overo-sakoman.diff calling it adeos_blabla_updated.diff could give the impression that it is an updated adeos patch. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 9:54 ` Gilles Chanteperdrix @ 2010-07-01 10:16 ` Fredrik Asplund 0 siblings, 0 replies; 32+ messages in thread From: Fredrik Asplund @ 2010-07-01 10:16 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai@xenomai.org >Ok. Next time you post this patch, could you give the attached file a >less confusing name? Such as >adeos-ipipe-2.6.33-arm-1.16-01-overo-sakoman.diff >calling it adeos_blabla_updated.diff could give the impression that it >is an updated adeos patch. Will do. Sorry. / Fredrik ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-06-30 21:58 ` Felipe Brandão Cavalcanti ` (2 preceding siblings ...) 2010-07-01 8:11 ` Fredrik Asplund @ 2010-07-01 12:22 ` Fredrik Asplund 2010-07-01 23:25 ` Felipe Brandão Cavalcanti 3 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-01 12:22 UTC (permalink / raw) To: cavalkaf@domain.hid; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 551 bytes --] >>So, any clues? I can send my changes and logs tomorrow morning, if they are of any help. >Sure, please do so. I don´t have much time to put on this, so I have actually only >have had to go through the source files since yesterday. Today (or during the weekend >if I don´t get the time) I will create a new defconfig file, compile, test and debug. >If you have updated BitBake file, Defconfig file, Logs, etc., please share them. Felipe - Here is my defconfig (which I have not had time to test yet though), please share yours. / Fredrik [-- Attachment #2: defconfig_overo_sakoman --] [-- Type: application/octet-stream, Size: 63028 bytes --] # # Automatically generated make config: don't edit # Linux kernel version: 2.6.33 # Thu Jul 1 13:56:31 2010 # CONFIG_ARM=y CONFIG_SYS_SUPPORTS_APM_EMULATION=y CONFIG_GENERIC_GPIO=y CONFIG_GENERIC_TIME=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_HAVE_PROC_CPU=y CONFIG_GENERIC_HARDIRQS=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_HARDIRQS_SW_RESEND=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_RWSEM_GENERIC_SPINLOCK=y CONFIG_ARCH_HAS_CPUFREQ=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y CONFIG_ARM_L1_CACHE_SHIFT_6=y CONFIG_OPROFILE_ARMV7=y CONFIG_VECTORS_BASE=0xffff0000 CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_CONSTRUCTORS=y # # General setup # CONFIG_EXPERIMENTAL=y CONFIG_BROKEN_ON_SMP=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_LZO=y CONFIG_KERNEL_GZIP=y # CONFIG_KERNEL_BZIP2 is not set # CONFIG_KERNEL_LZMA is not set # CONFIG_KERNEL_LZO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y # CONFIG_POSIX_MQUEUE is not set CONFIG_BSD_PROCESS_ACCT=y # CONFIG_BSD_PROCESS_ACCT_V3 is not set # CONFIG_TASKSTATS is not set # CONFIG_AUDIT is not set # # RCU Subsystem # CONFIG_TREE_RCU=y # CONFIG_TREE_PREEMPT_RCU is not set # CONFIG_TINY_RCU is not set # CONFIG_RCU_TRACE is not set CONFIG_RCU_FANOUT=32 # CONFIG_RCU_FANOUT_EXACT is not set # CONFIG_TREE_RCU_TRACE is not set CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y CONFIG_LOG_BUF_SHIFT=15 # CONFIG_CGROUPS is not set # CONFIG_SYSFS_DEPRECATED_V2 is not set # CONFIG_RELAY is not set # CONFIG_NAMESPACES is not set CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="" CONFIG_RD_GZIP=y # CONFIG_RD_BZIP2 is not set # CONFIG_RD_LZMA is not set # CONFIG_RD_LZO is not set CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_ANON_INODES=y CONFIG_EMBEDDED=y CONFIG_UID16=y # CONFIG_SYSCTL_SYSCALL is not set CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y # CONFIG_KALLSYMS_EXTRA_PASS is not set CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y # CONFIG_ELF_CORE is not set CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_HAVE_PERF_EVENTS=y CONFIG_PERF_USE_VMALLOC=y # # Kernel Performance Events And Counters # CONFIG_PERF_EVENTS=y # CONFIG_PERF_COUNTERS is not set CONFIG_VM_EVENT_COUNTERS=y # CONFIG_COMPAT_BRK is not set CONFIG_SLAB=y # CONFIG_SLUB is not set # CONFIG_SLOB is not set CONFIG_PROFILING=y CONFIG_OPROFILE=y CONFIG_HAVE_OPROFILE=y # CONFIG_KPROBES is not set CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_CLK=y # # GCOV-based kernel profiling # # CONFIG_GCOV_KERNEL is not set # CONFIG_SLOW_WORK is not set CONFIG_HAVE_GENERIC_DMA_COHERENT=y CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y CONFIG_BASE_SMALL=0 CONFIG_MODULES=y # CONFIG_MODULE_FORCE_LOAD is not set CONFIG_MODULE_UNLOAD=y # CONFIG_MODULE_FORCE_UNLOAD is not set CONFIG_MODVERSIONS=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_BLOCK=y CONFIG_LBDAF=y CONFIG_BLK_DEV_BSG=y # CONFIG_BLK_DEV_INTEGRITY is not set # # IO Schedulers # CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y # CONFIG_DEFAULT_DEADLINE is not set CONFIG_DEFAULT_CFQ=y # CONFIG_DEFAULT_NOOP is not set CONFIG_DEFAULT_IOSCHED="cfq" # CONFIG_INLINE_SPIN_TRYLOCK is not set # CONFIG_INLINE_SPIN_TRYLOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK is not set # CONFIG_INLINE_SPIN_LOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK_IRQ is not set # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set CONFIG_INLINE_SPIN_UNLOCK=y # CONFIG_INLINE_SPIN_UNLOCK_BH is not set CONFIG_INLINE_SPIN_UNLOCK_IRQ=y # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_READ_TRYLOCK is not set # CONFIG_INLINE_READ_LOCK is not set # CONFIG_INLINE_READ_LOCK_BH is not set # CONFIG_INLINE_READ_LOCK_IRQ is not set # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set CONFIG_INLINE_READ_UNLOCK=y # CONFIG_INLINE_READ_UNLOCK_BH is not set CONFIG_INLINE_READ_UNLOCK_IRQ=y # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_WRITE_TRYLOCK is not set # CONFIG_INLINE_WRITE_LOCK is not set # CONFIG_INLINE_WRITE_LOCK_BH is not set # CONFIG_INLINE_WRITE_LOCK_IRQ is not set # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set CONFIG_INLINE_WRITE_UNLOCK=y # CONFIG_INLINE_WRITE_UNLOCK_BH is not set CONFIG_INLINE_WRITE_UNLOCK_IRQ=y # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set # CONFIG_MUTEX_SPIN_ON_OWNER is not set # # Real-time sub-system # CONFIG_XENOMAI=y CONFIG_XENO_GENERIC_STACKPOOL=y CONFIG_XENO_FASTSYNCH_DEP=y CONFIG_XENO_FASTSYNCH=y CONFIG_XENO_OPT_NUCLEUS=y CONFIG_XENO_OPT_PERVASIVE=y CONFIG_XENO_OPT_PRIOCPL=y CONFIG_XENO_OPT_PIPELINE_HEAD=y # CONFIG_XENO_OPT_SCHED_CLASSES is not set CONFIG_XENO_OPT_PIPE=y CONFIG_XENO_OPT_PIPE_NRDEV=32 CONFIG_XENO_OPT_REGISTRY_NRSLOTS=512 CONFIG_XENO_OPT_SYS_HEAPSZ=256 CONFIG_XENO_OPT_SYS_STACKPOOLSZ=128 CONFIG_XENO_OPT_SEM_HEAPSZ=12 CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ=12 CONFIG_XENO_OPT_STATS=y # CONFIG_XENO_OPT_DEBUG is not set # CONFIG_XENO_OPT_SHIRQ is not set # # Timing # # CONFIG_XENO_OPT_TIMING_PERIODIC is not set CONFIG_XENO_OPT_TIMING_VIRTICK=1000 CONFIG_XENO_OPT_TIMING_SCHEDLAT=0 # # Scalability # # CONFIG_XENO_OPT_SCALABLE_SCHED is not set CONFIG_XENO_OPT_TIMER_LIST=y # CONFIG_XENO_OPT_TIMER_HEAP is not set # CONFIG_XENO_OPT_TIMER_WHEEL is not set # # Machine # CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH=y CONFIG_XENO_HW_FPU=y CONFIG_XENO_HW_UNLOCKED_SWITCH=y # # Interfaces # CONFIG_XENO_SKIN_NATIVE=y CONFIG_XENO_OPT_NATIVE_PERIOD=0 CONFIG_XENO_OPT_NATIVE_PIPE=y CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ=1024 CONFIG_XENO_OPT_NATIVE_SEM=y CONFIG_XENO_OPT_NATIVE_EVENT=y CONFIG_XENO_OPT_NATIVE_MUTEX=y CONFIG_XENO_OPT_NATIVE_COND=y CONFIG_XENO_OPT_NATIVE_QUEUE=y CONFIG_XENO_OPT_NATIVE_BUFFER=y CONFIG_XENO_OPT_NATIVE_HEAP=y CONFIG_XENO_OPT_NATIVE_ALARM=y CONFIG_XENO_OPT_NATIVE_MPS=y # CONFIG_XENO_OPT_NATIVE_INTR is not set CONFIG_XENO_SKIN_POSIX=y CONFIG_XENO_OPT_POSIX_PERIOD=0 # CONFIG_XENO_OPT_POSIX_SHM is not set # CONFIG_XENO_OPT_POSIX_INTR is not set # CONFIG_XENO_OPT_POSIX_SELECT is not set # CONFIG_XENO_OPT_DEBUG_POSIX is not set # CONFIG_XENO_SKIN_PSOS is not set # CONFIG_XENO_SKIN_UITRON is not set # CONFIG_XENO_SKIN_VRTX is not set # CONFIG_XENO_SKIN_VXWORKS is not set # CONFIG_XENO_SKIN_RTAI is not set # CONFIG_XENO_OPT_NOWARN_DEPRECATED is not set CONFIG_XENO_SKIN_RTDM=y CONFIG_XENO_OPT_RTDM_PERIOD=0 CONFIG_XENO_OPT_RTDM_FILDES=128 # CONFIG_XENO_OPT_RTDM_SELECT is not set # # Drivers # # # Serial drivers # # CONFIG_XENO_DRIVERS_16550A is not set # # Testing drivers # # CONFIG_XENO_DRIVERS_TESTING_LEGACY_NAMES is not set CONFIG_XENO_DRIVERS_TIMERBENCH=y # CONFIG_XENO_DRIVERS_KLATENCY is not set # CONFIG_XENO_DRIVERS_IRQBENCH is not set CONFIG_XENO_DRIVERS_SWITCHTEST=y # CONFIG_XENO_DRIVERS_SIGTEST is not set # CONFIG_XENO_DRIVERS_RTDMTEST is not set # # CAN drivers # # CONFIG_XENO_DRIVERS_CAN is not set # # ANALOGY drivers # # CONFIG_XENO_DRIVERS_ANALOGY is not set # # Real-time IPC drivers # # CONFIG_XENO_DRIVERS_RTIPC is not set CONFIG_FREEZER=y # # System Type # CONFIG_MMU=y # CONFIG_ARCH_AAEC2000 is not set # CONFIG_ARCH_INTEGRATOR is not set # CONFIG_ARCH_REALVIEW is not set # CONFIG_ARCH_VERSATILE is not set # CONFIG_ARCH_AT91 is not set # CONFIG_ARCH_CLPS711X is not set # CONFIG_ARCH_GEMINI is not set # CONFIG_ARCH_EBSA110 is not set # CONFIG_ARCH_EP93XX is not set # CONFIG_ARCH_FOOTBRIDGE is not set # CONFIG_ARCH_MXC is not set # CONFIG_ARCH_STMP3XXX is not set # CONFIG_ARCH_NETX is not set # CONFIG_ARCH_H720X is not set # CONFIG_ARCH_NOMADIK is not set # CONFIG_ARCH_IOP13XX is not set # CONFIG_ARCH_IOP32X is not set # CONFIG_ARCH_IOP33X is not set # CONFIG_ARCH_IXP23XX is not set # CONFIG_ARCH_IXP2000 is not set # CONFIG_ARCH_IXP4XX is not set # CONFIG_ARCH_L7200 is not set # CONFIG_ARCH_DOVE is not set # CONFIG_ARCH_KIRKWOOD is not set # CONFIG_ARCH_LOKI is not set # CONFIG_ARCH_MV78XX0 is not set # CONFIG_ARCH_ORION5X is not set # CONFIG_ARCH_MMP is not set # CONFIG_ARCH_KS8695 is not set # CONFIG_ARCH_NS9XXX is not set # CONFIG_ARCH_W90X900 is not set # CONFIG_ARCH_NUC93X is not set # CONFIG_ARCH_PNX4008 is not set # CONFIG_ARCH_PXA is not set # CONFIG_ARCH_MSM is not set # CONFIG_ARCH_RPC is not set # CONFIG_ARCH_SA1100 is not set # CONFIG_ARCH_S3C2410 is not set # CONFIG_ARCH_S3C64XX is not set # CONFIG_ARCH_S5PC1XX is not set # CONFIG_ARCH_SHARK is not set # CONFIG_ARCH_LH7A40X is not set # CONFIG_ARCH_U300 is not set # CONFIG_ARCH_DAVINCI is not set CONFIG_ARCH_OMAP=y # CONFIG_ARCH_BCMRING is not set # CONFIG_ARCH_U8500 is not set # # TI OMAP Implementations # CONFIG_ARCH_OMAP_OTG=y # CONFIG_ARCH_OMAP1 is not set CONFIG_ARCH_OMAP2PLUS=y # CONFIG_ARCH_OMAP2 is not set CONFIG_ARCH_OMAP3=y # CONFIG_ARCH_OMAP4 is not set # # OMAP Feature Selections # # CONFIG_OMAP_RESET_CLOCKS is not set # CONFIG_OMAP_MUX is not set CONFIG_OMAP_MCBSP=y # CONFIG_OMAP_MBOX_FWK is not set CONFIG_OMAP_MPU_TIMER=y # CONFIG_OMAP_32K_TIMER is not set # CONFIG_OMAP3_L2_AUX_SECURE_SAVE_RESTORE is not set CONFIG_OMAP_DM_TIMER=y # CONFIG_OMAP_PM_NONE is not set CONFIG_OMAP_PM_NOOP=y CONFIG_ARCH_OMAP3430=y CONFIG_OMAP_PACKAGE_CBB=y # # OMAP Board Type # # CONFIG_MACH_OMAP3_BEAGLE is not set # CONFIG_MACH_DEVKIT8000 is not set # CONFIG_MACH_OMAP_LDP is not set CONFIG_MACH_OVERO=y # CONFIG_MACH_OMAP3EVM is not set # CONFIG_MACH_OMAP3517EVM is not set # CONFIG_MACH_OMAP3_PANDORA is not set # CONFIG_MACH_OMAP3_TOUCHBOOK is not set # CONFIG_MACH_OMAP_3430SDP is not set # CONFIG_MACH_NOKIA_RX51 is not set # CONFIG_MACH_OMAP_ZOOM2 is not set # CONFIG_MACH_OMAP_ZOOM3 is not set # CONFIG_MACH_CM_T35 is not set CONFIG_MACH_IGEP0020=y # CONFIG_MACH_OMAP_3630SDP is not set # CONFIG_OMAP3_EMU is not set # CONFIG_OMAP3_SDRC_AC_TIMING is not set # # Processor Type # CONFIG_CPU_32v6K=y CONFIG_CPU_V7=y CONFIG_CPU_32v7=y CONFIG_CPU_ABRT_EV7=y CONFIG_CPU_PABRT_V7=y CONFIG_CPU_CACHE_V7=y CONFIG_CPU_CACHE_VIPT=y CONFIG_CPU_COPY_V6=y CONFIG_CPU_TLB_V7=y CONFIG_CPU_HAS_ASID=y CONFIG_CPU_CP15=y CONFIG_CPU_CP15_MMU=y # # Processor Features # CONFIG_ARM_THUMB=y CONFIG_ARM_THUMBEE=y # CONFIG_CPU_ICACHE_DISABLE is not set # CONFIG_CPU_DCACHE_DISABLE is not set # CONFIG_CPU_BPREDICT_DISABLE is not set CONFIG_HAS_TLS_REG=y CONFIG_ARM_L1_CACHE_SHIFT=6 CONFIG_CPU_HAS_PMU=y CONFIG_ARM_ERRATA_430973=y # CONFIG_ARM_ERRATA_458693 is not set # CONFIG_ARM_ERRATA_460075 is not set CONFIG_COMMON_CLKDEV=y # # Bus support # # CONFIG_PCI_SYSCALL is not set # CONFIG_ARCH_SUPPORTS_MSI is not set # CONFIG_PCCARD is not set # # Kernel Features # CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_VMSPLIT_3G=y # CONFIG_VMSPLIT_2G is not set # CONFIG_VMSPLIT_1G is not set CONFIG_PAGE_OFFSET=0xC0000000 CONFIG_IPIPE=y CONFIG_IPIPE_DOMAINS=4 CONFIG_IPIPE_DELAYED_ATOMICSW=y # CONFIG_IPIPE_UNMASKED_CONTEXT_SWITCH is not set CONFIG_PREEMPT_NONE=y # CONFIG_PREEMPT_VOLUNTARY is not set # CONFIG_PREEMPT is not set CONFIG_HZ=100 # CONFIG_THUMB2_KERNEL is not set CONFIG_AEABI=y # CONFIG_OABI_COMPAT is not set CONFIG_ARCH_HAS_HOLES_MEMORYMODEL=y # CONFIG_ARCH_SPARSEMEM_DEFAULT is not set # CONFIG_ARCH_SELECT_MEMORY_MODEL is not set # CONFIG_HIGHMEM is not set CONFIG_HW_PERF_EVENTS=y CONFIG_SELECT_MEMORY_MODEL=y CONFIG_FLATMEM_MANUAL=y # CONFIG_DISCONTIGMEM_MANUAL is not set # CONFIG_SPARSEMEM_MANUAL is not set CONFIG_FLATMEM=y CONFIG_FLAT_NODE_MEM_MAP=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 # CONFIG_PHYS_ADDR_T_64BIT is not set CONFIG_ZONE_DMA_FLAG=0 CONFIG_VIRT_TO_BUS=y # CONFIG_KSM is not set CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_LEDS=y CONFIG_ALIGNMENT_TRAP=y # CONFIG_UACCESS_WITH_MEMCPY is not set # # Boot options # CONFIG_ZBOOT_ROM_TEXT=0x0 CONFIG_ZBOOT_ROM_BSS=0x0 CONFIG_CMDLINE=" debug " # CONFIG_XIP_KERNEL is not set CONFIG_KEXEC=y CONFIG_ATAGS_PROC=y # # CPU Power Management # # CONFIG_CPU_FREQ is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set # CONFIG_CPU_IDLE is not set # # Floating point emulation # # # At least one emulation must be selected # CONFIG_VFP=y CONFIG_VFPv3=y CONFIG_NEON=y # # Userspace binary formats # CONFIG_BINFMT_ELF=y CONFIG_HAVE_AOUT=y CONFIG_BINFMT_AOUT=m CONFIG_BINFMT_MISC=y # # Power management options # CONFIG_PM=y CONFIG_PM_DEBUG=y # CONFIG_PM_ADVANCED_DEBUG is not set # CONFIG_PM_VERBOSE is not set CONFIG_CAN_PM_TRACE=y CONFIG_PM_SLEEP=y CONFIG_SUSPEND=y # CONFIG_PM_TEST_SUSPEND is not set CONFIG_SUSPEND_FREEZER=y # CONFIG_APM_EMULATION is not set # CONFIG_PM_RUNTIME is not set CONFIG_PM_OPS=y CONFIG_ARCH_SUSPEND_POSSIBLE=y CONFIG_NET=y # # Networking options # CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_XFRM=y # CONFIG_XFRM_USER is not set # CONFIG_XFRM_SUB_POLICY is not set # CONFIG_XFRM_MIGRATE is not set # CONFIG_XFRM_STATISTICS is not set CONFIG_NET_KEY=y # CONFIG_NET_KEY_MIGRATE is not set CONFIG_INET=y # CONFIG_IP_MULTICAST is not set # CONFIG_IP_ADVANCED_ROUTER is not set CONFIG_IP_FIB_HASH=y CONFIG_IP_PNP=y CONFIG_IP_PNP_DHCP=y CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_RARP=y # CONFIG_NET_IPIP is not set # CONFIG_NET_IPGRE is not set # CONFIG_ARPD is not set # CONFIG_SYN_COOKIES is not set # CONFIG_INET_AH is not set # CONFIG_INET_ESP is not set # CONFIG_INET_IPCOMP is not set # CONFIG_INET_XFRM_TUNNEL is not set CONFIG_INET_TUNNEL=m CONFIG_INET_XFRM_MODE_TRANSPORT=y CONFIG_INET_XFRM_MODE_TUNNEL=y CONFIG_INET_XFRM_MODE_BEET=y # CONFIG_INET_LRO is not set CONFIG_INET_DIAG=y CONFIG_INET_TCP_DIAG=y # CONFIG_TCP_CONG_ADVANCED is not set CONFIG_TCP_CONG_CUBIC=y CONFIG_DEFAULT_TCP_CONG="cubic" # CONFIG_TCP_MD5SIG is not set CONFIG_IPV6=m # CONFIG_IPV6_PRIVACY is not set # CONFIG_IPV6_ROUTER_PREF is not set # CONFIG_IPV6_OPTIMISTIC_DAD is not set # CONFIG_INET6_AH is not set # CONFIG_INET6_ESP is not set # CONFIG_INET6_IPCOMP is not set # CONFIG_IPV6_MIP6 is not set # CONFIG_INET6_XFRM_TUNNEL is not set # CONFIG_INET6_TUNNEL is not set CONFIG_INET6_XFRM_MODE_TRANSPORT=m CONFIG_INET6_XFRM_MODE_TUNNEL=m CONFIG_INET6_XFRM_MODE_BEET=m # CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set CONFIG_IPV6_SIT=m # CONFIG_IPV6_SIT_6RD is not set CONFIG_IPV6_NDISC_NODETYPE=y # CONFIG_IPV6_TUNNEL is not set # CONFIG_IPV6_MULTIPLE_TABLES is not set # CONFIG_IPV6_MROUTE is not set # CONFIG_NETWORK_SECMARK is not set # CONFIG_NETFILTER is not set # CONFIG_IP_DCCP is not set # CONFIG_IP_SCTP is not set # CONFIG_RDS is not set # CONFIG_TIPC is not set # CONFIG_ATM is not set # CONFIG_BRIDGE is not set # CONFIG_NET_DSA is not set # CONFIG_VLAN_8021Q is not set # CONFIG_DECNET is not set # CONFIG_LLC2 is not set # CONFIG_IPX is not set # CONFIG_ATALK is not set # CONFIG_X25 is not set # CONFIG_LAPB is not set # CONFIG_ECONET is not set # CONFIG_WAN_ROUTER is not set # CONFIG_PHONET is not set # CONFIG_IEEE802154 is not set # CONFIG_NET_SCHED is not set # CONFIG_DCB is not set # # Network testing # # CONFIG_NET_PKTGEN is not set # CONFIG_HAMRADIO is not set # CONFIG_CAN is not set # CONFIG_IRDA is not set CONFIG_BT=y CONFIG_BT_L2CAP=y CONFIG_BT_SCO=y CONFIG_BT_RFCOMM=y CONFIG_BT_RFCOMM_TTY=y CONFIG_BT_BNEP=y CONFIG_BT_BNEP_MC_FILTER=y CONFIG_BT_BNEP_PROTO_FILTER=y CONFIG_BT_HIDP=y # # Bluetooth device drivers # CONFIG_BT_HCIBTUSB=m CONFIG_BT_HCIBTSDIO=m CONFIG_BT_HCIUART=y CONFIG_BT_HCIUART_H4=y CONFIG_BT_HCIUART_BCSP=y CONFIG_BT_HCIUART_LL=y CONFIG_BT_HCIBCM203X=m CONFIG_BT_HCIBPA10X=m CONFIG_BT_HCIBFUSB=m CONFIG_BT_HCIVHCI=m CONFIG_BT_MRVL=m CONFIG_BT_MRVL_SDIO=m CONFIG_BT_ATH3K=m # CONFIG_AF_RXRPC is not set CONFIG_WIRELESS=y CONFIG_WIRELESS_EXT=y CONFIG_WEXT_CORE=y CONFIG_WEXT_PROC=y CONFIG_WEXT_SPY=y CONFIG_WEXT_PRIV=y CONFIG_CFG80211=y # CONFIG_NL80211_TESTMODE is not set # CONFIG_CFG80211_DEVELOPER_WARNINGS is not set # CONFIG_CFG80211_REG_DEBUG is not set CONFIG_CFG80211_DEFAULT_PS=y # CONFIG_CFG80211_DEBUGFS is not set # CONFIG_CFG80211_INTERNAL_REGDB is not set CONFIG_CFG80211_WEXT=y CONFIG_WIRELESS_EXT_SYSFS=y CONFIG_LIB80211=m CONFIG_LIB80211_CRYPT_WEP=m CONFIG_LIB80211_CRYPT_CCMP=m CONFIG_LIB80211_CRYPT_TKIP=m # CONFIG_LIB80211_DEBUG is not set CONFIG_MAC80211=y CONFIG_MAC80211_RC_PID=y # CONFIG_MAC80211_RC_MINSTREL is not set CONFIG_MAC80211_RC_DEFAULT_PID=y # CONFIG_MAC80211_RC_DEFAULT_MINSTREL is not set CONFIG_MAC80211_RC_DEFAULT="pid" # CONFIG_MAC80211_MESH is not set CONFIG_MAC80211_LEDS=y # CONFIG_MAC80211_DEBUGFS is not set # CONFIG_MAC80211_DEBUG_MENU is not set # CONFIG_WIMAX is not set # CONFIG_RFKILL is not set # CONFIG_NET_9P is not set # # Device Drivers # # # Generic Driver Options # CONFIG_UEVENT_HELPER_PATH="" CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_STANDALONE=y CONFIG_PREVENT_FIRMWARE_BUILD=y CONFIG_FW_LOADER=y CONFIG_FIRMWARE_IN_KERNEL=y CONFIG_EXTRA_FIRMWARE="" # CONFIG_SYS_HYPERVISOR is not set # CONFIG_CONNECTOR is not set CONFIG_MTD=y # CONFIG_MTD_DEBUG is not set # CONFIG_MTD_TESTS is not set CONFIG_MTD_CONCAT=y CONFIG_MTD_PARTITIONS=y # CONFIG_MTD_REDBOOT_PARTS is not set # CONFIG_MTD_CMDLINE_PARTS is not set # CONFIG_MTD_AFS_PARTS is not set # CONFIG_MTD_AR7_PARTS is not set # # User Modules And Translation Layers # CONFIG_MTD_CHAR=y CONFIG_MTD_BLKDEVS=y CONFIG_MTD_BLOCK=y # CONFIG_FTL is not set # CONFIG_NFTL is not set # CONFIG_INFTL is not set # CONFIG_RFD_FTL is not set # CONFIG_SSFDC is not set # CONFIG_MTD_OOPS is not set # # RAM/ROM/Flash chip drivers # # CONFIG_MTD_CFI is not set # CONFIG_MTD_JEDECPROBE is not set CONFIG_MTD_MAP_BANK_WIDTH_1=y CONFIG_MTD_MAP_BANK_WIDTH_2=y CONFIG_MTD_MAP_BANK_WIDTH_4=y # CONFIG_MTD_MAP_BANK_WIDTH_8 is not set # CONFIG_MTD_MAP_BANK_WIDTH_16 is not set # CONFIG_MTD_MAP_BANK_WIDTH_32 is not set CONFIG_MTD_CFI_I1=y CONFIG_MTD_CFI_I2=y # CONFIG_MTD_CFI_I4 is not set # CONFIG_MTD_CFI_I8 is not set # CONFIG_MTD_RAM is not set # CONFIG_MTD_ROM is not set # CONFIG_MTD_ABSENT is not set # # Mapping drivers for chip access # # CONFIG_MTD_COMPLEX_MAPPINGS is not set # CONFIG_MTD_PLATRAM is not set # # Self-contained MTD device drivers # # CONFIG_MTD_DATAFLASH is not set # CONFIG_MTD_M25P80 is not set # CONFIG_MTD_SST25L is not set # CONFIG_MTD_SLRAM is not set # CONFIG_MTD_PHRAM is not set # CONFIG_MTD_MTDRAM is not set # CONFIG_MTD_BLOCK2MTD is not set # # Disk-On-Chip Device Drivers # # CONFIG_MTD_DOC2000 is not set # CONFIG_MTD_DOC2001 is not set # CONFIG_MTD_DOC2001PLUS is not set CONFIG_MTD_NAND=y # CONFIG_MTD_NAND_VERIFY_WRITE is not set # CONFIG_MTD_NAND_ECC_SMC is not set # CONFIG_MTD_NAND_MUSEUM_IDS is not set # CONFIG_MTD_NAND_GPIO is not set CONFIG_MTD_NAND_OMAP2=y CONFIG_MTD_NAND_OMAP_PREFETCH=y # CONFIG_MTD_NAND_OMAP_PREFETCH_DMA is not set CONFIG_MTD_NAND_IDS=y # CONFIG_MTD_NAND_DISKONCHIP is not set # CONFIG_MTD_NAND_NANDSIM is not set # CONFIG_MTD_NAND_PLATFORM is not set # CONFIG_MTD_ALAUDA is not set # CONFIG_MTD_ONENAND is not set # # LPDDR flash memory drivers # # CONFIG_MTD_LPDDR is not set # # UBI - Unsorted block images # CONFIG_MTD_UBI=y CONFIG_MTD_UBI_WL_THRESHOLD=4096 CONFIG_MTD_UBI_BEB_RESERVE=1 CONFIG_MTD_UBI_GLUEBI=y # # UBI debugging options # # CONFIG_MTD_UBI_DEBUG is not set # CONFIG_PARPORT is not set CONFIG_BLK_DEV=y # CONFIG_BLK_DEV_COW_COMMON is not set CONFIG_BLK_DEV_LOOP=y CONFIG_BLK_DEV_CRYPTOLOOP=m # # DRBD disabled because PROC_FS, INET or CONNECTOR not selected # # CONFIG_BLK_DEV_NBD is not set # CONFIG_BLK_DEV_UB is not set CONFIG_BLK_DEV_RAM=y CONFIG_BLK_DEV_RAM_COUNT=16 CONFIG_BLK_DEV_RAM_SIZE=16384 # CONFIG_BLK_DEV_XIP is not set CONFIG_CDROM_PKTCDVD=m CONFIG_CDROM_PKTCDVD_BUFFERS=8 # CONFIG_CDROM_PKTCDVD_WCACHE is not set # CONFIG_ATA_OVER_ETH is not set # CONFIG_MG_DISK is not set # CONFIG_MISC_DEVICES is not set CONFIG_EEPROM_93CX6=m CONFIG_HAVE_IDE=y # CONFIG_IDE is not set # # SCSI device support # CONFIG_RAID_ATTRS=m CONFIG_SCSI=y CONFIG_SCSI_DMA=y # CONFIG_SCSI_TGT is not set # CONFIG_SCSI_NETLINK is not set CONFIG_SCSI_PROC_FS=y # # SCSI support type (disk, tape, CD-ROM) # CONFIG_BLK_DEV_SD=y # CONFIG_CHR_DEV_ST is not set # CONFIG_CHR_DEV_OSST is not set CONFIG_BLK_DEV_SR=m # CONFIG_BLK_DEV_SR_VENDOR is not set CONFIG_CHR_DEV_SG=m # CONFIG_CHR_DEV_SCH is not set CONFIG_SCSI_MULTI_LUN=y # CONFIG_SCSI_CONSTANTS is not set # CONFIG_SCSI_LOGGING is not set # CONFIG_SCSI_SCAN_ASYNC is not set CONFIG_SCSI_WAIT_SCAN=m # # SCSI Transports # # CONFIG_SCSI_SPI_ATTRS is not set # CONFIG_SCSI_FC_ATTRS is not set # CONFIG_SCSI_ISCSI_ATTRS is not set # CONFIG_SCSI_SAS_ATTRS is not set # CONFIG_SCSI_SAS_LIBSAS is not set # CONFIG_SCSI_SRP_ATTRS is not set CONFIG_SCSI_LOWLEVEL=y # CONFIG_ISCSI_TCP is not set # CONFIG_LIBFC is not set # CONFIG_LIBFCOE is not set # CONFIG_SCSI_DEBUG is not set # CONFIG_SCSI_DH is not set # CONFIG_SCSI_OSD_INITIATOR is not set # CONFIG_ATA is not set CONFIG_MD=y CONFIG_BLK_DEV_MD=m CONFIG_MD_LINEAR=m CONFIG_MD_RAID0=m CONFIG_MD_RAID1=m CONFIG_MD_RAID10=m CONFIG_MD_RAID456=m CONFIG_MD_RAID6_PQ=m # CONFIG_ASYNC_RAID6_TEST is not set CONFIG_MD_MULTIPATH=m CONFIG_MD_FAULTY=m CONFIG_BLK_DEV_DM=m # CONFIG_DM_DEBUG is not set CONFIG_DM_CRYPT=m CONFIG_DM_SNAPSHOT=m CONFIG_DM_MIRROR=m # CONFIG_DM_LOG_USERSPACE is not set CONFIG_DM_ZERO=m CONFIG_DM_MULTIPATH=m # CONFIG_DM_MULTIPATH_QL is not set # CONFIG_DM_MULTIPATH_ST is not set CONFIG_DM_DELAY=m # CONFIG_DM_UEVENT is not set CONFIG_NETDEVICES=y CONFIG_DUMMY=m # CONFIG_BONDING is not set # CONFIG_MACVLAN is not set # CONFIG_EQUALIZER is not set CONFIG_TUN=m # CONFIG_VETH is not set CONFIG_PHYLIB=y # # MII PHY device drivers # # CONFIG_MARVELL_PHY is not set # CONFIG_DAVICOM_PHY is not set # CONFIG_QSEMI_PHY is not set # CONFIG_LXT_PHY is not set # CONFIG_CICADA_PHY is not set # CONFIG_VITESSE_PHY is not set # CONFIG_SMSC_PHY is not set # CONFIG_BROADCOM_PHY is not set # CONFIG_ICPLUS_PHY is not set # CONFIG_REALTEK_PHY is not set # CONFIG_NATIONAL_PHY is not set # CONFIG_STE10XP is not set # CONFIG_LSI_ET1011C_PHY is not set # CONFIG_FIXED_PHY is not set # CONFIG_MDIO_BITBANG is not set CONFIG_NET_ETHERNET=y CONFIG_MII=y # CONFIG_AX88796 is not set # CONFIG_SMC91X is not set # CONFIG_TI_DAVINCI_EMAC is not set # CONFIG_DM9000 is not set CONFIG_ENC28J60=m # CONFIG_ENC28J60_WRITEVERIFY is not set # CONFIG_ETHOC is not set # CONFIG_SMC911X is not set CONFIG_SMSC911X=y # CONFIG_DNET is not set # CONFIG_IBM_NEW_EMAC_ZMII is not set # CONFIG_IBM_NEW_EMAC_RGMII is not set # CONFIG_IBM_NEW_EMAC_TAH is not set # CONFIG_IBM_NEW_EMAC_EMAC4 is not set # CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set # CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set # CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set # CONFIG_B44 is not set # CONFIG_KS8842 is not set # CONFIG_KS8851 is not set # CONFIG_KS8851_MLL is not set # CONFIG_NETDEV_1000 is not set # CONFIG_NETDEV_10000 is not set CONFIG_WLAN=y # CONFIG_LIBERTAS_THINFIRM is not set # CONFIG_AT76C50X_USB is not set CONFIG_USB_ZD1201=m CONFIG_USB_NET_RNDIS_WLAN=m CONFIG_RTL8187=m CONFIG_RTL8187_LEDS=y # CONFIG_MAC80211_HWSIM is not set # CONFIG_ATH_COMMON is not set # CONFIG_B43 is not set # CONFIG_B43LEGACY is not set CONFIG_HOSTAP=m CONFIG_HOSTAP_FIRMWARE=y CONFIG_HOSTAP_FIRMWARE_NVRAM=y # CONFIG_IWM is not set CONFIG_LIBERTAS=m CONFIG_LIBERTAS_USB=m CONFIG_LIBERTAS_SDIO=m # CONFIG_LIBERTAS_SPI is not set CONFIG_LIBERTAS_DEBUG=y # CONFIG_LIBERTAS_MESH is not set CONFIG_P54_COMMON=m CONFIG_P54_USB=m # CONFIG_P54_SPI is not set CONFIG_P54_LEDS=y CONFIG_RT2X00=m CONFIG_RT2500USB=m CONFIG_RT73USB=m CONFIG_RT2800USB=m # CONFIG_RT2800USB_RT30XX is not set # CONFIG_RT2800USB_RT35XX is not set # CONFIG_RT2800USB_UNKNOWN is not set CONFIG_RT2800_LIB=m CONFIG_RT2X00_LIB_USB=m CONFIG_RT2X00_LIB=m CONFIG_RT2X00_LIB_HT=y CONFIG_RT2X00_LIB_FIRMWARE=y CONFIG_RT2X00_LIB_CRYPTO=y CONFIG_RT2X00_LIB_LEDS=y # CONFIG_RT2X00_DEBUG is not set CONFIG_WL12XX=m # CONFIG_WL1251 is not set CONFIG_WL1271=m CONFIG_ZD1211RW=m # CONFIG_ZD1211RW_DEBUG is not set # # Enable WiMAX (Networking options) to see the WiMAX drivers # # # USB Network Adapters # CONFIG_USB_CATC=m CONFIG_USB_KAWETH=m CONFIG_USB_PEGASUS=m CONFIG_USB_RTL8150=m CONFIG_USB_USBNET=y CONFIG_USB_NET_AX8817X=y CONFIG_USB_NET_CDCETHER=y CONFIG_USB_NET_CDC_EEM=m CONFIG_USB_NET_DM9601=m CONFIG_USB_NET_SMSC95XX=m CONFIG_USB_NET_GL620A=m CONFIG_USB_NET_NET1080=m CONFIG_USB_NET_PLUSB=m CONFIG_USB_NET_MCS7830=m CONFIG_USB_NET_RNDIS_HOST=m CONFIG_USB_NET_CDC_SUBSET=m CONFIG_USB_ALI_M5632=y CONFIG_USB_AN2720=y CONFIG_USB_BELKIN=y CONFIG_USB_ARMLINUX=y CONFIG_USB_EPSON2888=y CONFIG_USB_KC2190=y CONFIG_USB_NET_ZAURUS=m CONFIG_USB_NET_INT51X1=m # CONFIG_WAN is not set CONFIG_PPP=m # CONFIG_PPP_MULTILINK is not set # CONFIG_PPP_FILTER is not set CONFIG_PPP_ASYNC=m CONFIG_PPP_SYNC_TTY=m CONFIG_PPP_DEFLATE=m CONFIG_PPP_BSDCOMP=m CONFIG_PPP_MPPE=m CONFIG_PPPOE=m # CONFIG_PPPOL2TP is not set # CONFIG_SLIP is not set CONFIG_SLHC=m # CONFIG_NETCONSOLE is not set # CONFIG_NETPOLL is not set # CONFIG_NET_POLL_CONTROLLER is not set # CONFIG_ISDN is not set # CONFIG_PHONE is not set # # Input device support # CONFIG_INPUT=y # CONFIG_INPUT_FF_MEMLESS is not set CONFIG_INPUT_POLLDEV=m # CONFIG_INPUT_SPARSEKMAP is not set # # Userland interfaces # CONFIG_INPUT_MOUSEDEV=y CONFIG_INPUT_MOUSEDEV_PSAUX=y CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024 CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768 # CONFIG_INPUT_JOYDEV is not set CONFIG_INPUT_EVDEV=y # CONFIG_INPUT_EVBUG is not set # # Input Device Drivers # CONFIG_INPUT_KEYBOARD=y # CONFIG_KEYBOARD_ADP5588 is not set # CONFIG_KEYBOARD_ATKBD is not set # CONFIG_QT2160 is not set # CONFIG_KEYBOARD_LKKBD is not set # CONFIG_KEYBOARD_GPIO is not set # CONFIG_KEYBOARD_MATRIX is not set # CONFIG_KEYBOARD_LM8323 is not set # CONFIG_KEYBOARD_MAX7359 is not set # CONFIG_KEYBOARD_NEWTON is not set # CONFIG_KEYBOARD_OPENCORES is not set # CONFIG_KEYBOARD_STOWAWAY is not set # CONFIG_KEYBOARD_SUNKBD is not set # CONFIG_KEYBOARD_TWL4030 is not set # CONFIG_KEYBOARD_XTKBD is not set CONFIG_INPUT_MOUSE=y CONFIG_MOUSE_PS2=y CONFIG_MOUSE_PS2_ALPS=y CONFIG_MOUSE_PS2_LOGIPS2PP=y CONFIG_MOUSE_PS2_SYNAPTICS=y CONFIG_MOUSE_PS2_TRACKPOINT=y # CONFIG_MOUSE_PS2_ELANTECH is not set # CONFIG_MOUSE_PS2_SENTELIC is not set # CONFIG_MOUSE_PS2_TOUCHKIT is not set # CONFIG_MOUSE_SERIAL is not set # CONFIG_MOUSE_APPLETOUCH is not set # CONFIG_MOUSE_BCM5974 is not set # CONFIG_MOUSE_VSXXXAA is not set # CONFIG_MOUSE_GPIO is not set # CONFIG_MOUSE_SYNAPTICS_I2C is not set # CONFIG_INPUT_JOYSTICK is not set # CONFIG_INPUT_TABLET is not set CONFIG_INPUT_TOUCHSCREEN=y CONFIG_TOUCHSCREEN_ADS7846=m # CONFIG_TOUCHSCREEN_AD7877 is not set # CONFIG_TOUCHSCREEN_AD7879_I2C is not set # CONFIG_TOUCHSCREEN_AD7879_SPI is not set # CONFIG_TOUCHSCREEN_AD7879 is not set # CONFIG_TOUCHSCREEN_DYNAPRO is not set # CONFIG_TOUCHSCREEN_EETI is not set # CONFIG_TOUCHSCREEN_FUJITSU is not set # CONFIG_TOUCHSCREEN_GUNZE is not set # CONFIG_TOUCHSCREEN_ELO is not set # CONFIG_TOUCHSCREEN_WACOM_W8001 is not set # CONFIG_TOUCHSCREEN_MCS5000 is not set # CONFIG_TOUCHSCREEN_MTOUCH is not set # CONFIG_TOUCHSCREEN_INEXIO is not set # CONFIG_TOUCHSCREEN_MK712 is not set # CONFIG_TOUCHSCREEN_PENMOUNT is not set # CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set # CONFIG_TOUCHSCREEN_TOUCHWIN is not set # CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set # CONFIG_TOUCHSCREEN_TOUCHIT213 is not set # CONFIG_TOUCHSCREEN_TSC2007 is not set # CONFIG_TOUCHSCREEN_W90X900 is not set # CONFIG_INPUT_MISC is not set # # Hardware I/O ports # CONFIG_SERIO=y CONFIG_SERIO_SERPORT=y CONFIG_SERIO_LIBPS2=y # CONFIG_SERIO_RAW is not set # CONFIG_SERIO_ALTERA_PS2 is not set # CONFIG_GAMEPORT is not set # # Character devices # CONFIG_VT=y CONFIG_CONSOLE_TRANSLATIONS=y CONFIG_VT_CONSOLE=y CONFIG_HW_CONSOLE=y CONFIG_VT_HW_CONSOLE_BINDING=y CONFIG_DEVKMEM=y # CONFIG_SERIAL_NONSTANDARD is not set # # Serial drivers # CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_8250_NR_UARTS=32 CONFIG_SERIAL_8250_RUNTIME_UARTS=4 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_MANY_PORTS=y CONFIG_SERIAL_8250_SHARE_IRQ=y CONFIG_SERIAL_8250_DETECT_IRQ=y CONFIG_SERIAL_8250_RSA=y # # Non-8250 serial port support # # CONFIG_SERIAL_MAX3100 is not set CONFIG_SERIAL_CORE=y CONFIG_SERIAL_CORE_CONSOLE=y CONFIG_UNIX98_PTYS=y # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set # CONFIG_LEGACY_PTYS is not set # CONFIG_IPMI_HANDLER is not set CONFIG_HW_RANDOM=y # CONFIG_HW_RANDOM_TIMERIOMEM is not set # CONFIG_R3964 is not set # CONFIG_RAW_DRIVER is not set # CONFIG_TCG_TPM is not set CONFIG_I2C=y CONFIG_I2C_BOARDINFO=y CONFIG_I2C_COMPAT=y CONFIG_I2C_CHARDEV=y CONFIG_I2C_HELPER_AUTO=y # # I2C Hardware Bus support # # # I2C system bus drivers (mostly embedded / system-on-chip) # # CONFIG_I2C_DESIGNWARE is not set # CONFIG_I2C_GPIO is not set # CONFIG_I2C_OCORES is not set CONFIG_I2C_OMAP=y # CONFIG_I2C_SIMTEC is not set # # External I2C/SMBus adapter drivers # # CONFIG_I2C_PARPORT_LIGHT is not set # CONFIG_I2C_TAOS_EVM is not set # CONFIG_I2C_TINY_USB is not set # # Other I2C/SMBus bus drivers # # CONFIG_I2C_PCA_PLATFORM is not set # CONFIG_I2C_STUB is not set # # Miscellaneous I2C Chip support # # CONFIG_SENSORS_TSL2550 is not set # CONFIG_I2C_DEBUG_CORE is not set # CONFIG_I2C_DEBUG_ALGO is not set # CONFIG_I2C_DEBUG_BUS is not set # CONFIG_I2C_DEBUG_CHIP is not set CONFIG_SPI=y CONFIG_SPI_MASTER=y # # SPI Master Controller Drivers # # CONFIG_SPI_BITBANG is not set # CONFIG_SPI_GPIO is not set CONFIG_SPI_OMAP24XX=y # CONFIG_SPI_XILINX is not set # CONFIG_SPI_DESIGNWARE is not set # # SPI Protocol Masters # CONFIG_SPI_SPIDEV=y # CONFIG_SPI_TLE62X0 is not set # # PPS support # # CONFIG_PPS is not set CONFIG_ARCH_REQUIRE_GPIOLIB=y CONFIG_GPIOLIB=y CONFIG_GPIO_SYSFS=y # # Memory mapped GPIO expanders: # # # I2C GPIO expanders: # CONFIG_GPIO_MAX732X=m CONFIG_GPIO_PCA953X=m CONFIG_GPIO_PCF857X=m CONFIG_GPIO_TWL4030=y CONFIG_GPIO_ADP5588=m # # PCI GPIO expanders: # # # SPI GPIO expanders: # CONFIG_GPIO_MAX7301=m CONFIG_GPIO_MCP23S08=m # CONFIG_GPIO_MC33880 is not set # # AC97 GPIO expanders: # # CONFIG_W1 is not set CONFIG_POWER_SUPPLY=m # CONFIG_POWER_SUPPLY_DEBUG is not set # CONFIG_PDA_POWER is not set # CONFIG_BATTERY_DS2760 is not set # CONFIG_BATTERY_DS2782 is not set # CONFIG_BATTERY_BQ27x00 is not set # CONFIG_BATTERY_MAX17040 is not set CONFIG_HWMON=y CONFIG_HWMON_VID=m # CONFIG_HWMON_DEBUG_CHIP is not set # # Native drivers # CONFIG_SENSORS_AD7414=m CONFIG_SENSORS_AD7418=m CONFIG_SENSORS_ADCXX=m CONFIG_SENSORS_ADM1021=m CONFIG_SENSORS_ADM1025=m CONFIG_SENSORS_ADM1026=m CONFIG_SENSORS_ADM1029=m CONFIG_SENSORS_ADM1031=m CONFIG_SENSORS_ADM9240=m CONFIG_SENSORS_ADT7462=m CONFIG_SENSORS_ADT7470=m CONFIG_SENSORS_ADT7473=m CONFIG_SENSORS_ADT7475=m CONFIG_SENSORS_ATXP1=m CONFIG_SENSORS_DS1621=m CONFIG_SENSORS_F71805F=m CONFIG_SENSORS_F71882FG=m CONFIG_SENSORS_F75375S=m CONFIG_SENSORS_G760A=m CONFIG_SENSORS_GL518SM=m CONFIG_SENSORS_GL520SM=m CONFIG_SENSORS_IT87=m CONFIG_SENSORS_LM63=m CONFIG_SENSORS_LM70=m CONFIG_SENSORS_LM73=m CONFIG_SENSORS_LM75=m CONFIG_SENSORS_LM77=m CONFIG_SENSORS_LM78=m CONFIG_SENSORS_LM80=m CONFIG_SENSORS_LM83=m CONFIG_SENSORS_LM85=m CONFIG_SENSORS_LM87=m CONFIG_SENSORS_LM90=m CONFIG_SENSORS_LM92=m CONFIG_SENSORS_LM93=m CONFIG_SENSORS_LTC4215=m CONFIG_SENSORS_LTC4245=m CONFIG_SENSORS_LM95241=m CONFIG_SENSORS_MAX1111=m CONFIG_SENSORS_MAX1619=m CONFIG_SENSORS_MAX6650=m CONFIG_SENSORS_PC87360=m CONFIG_SENSORS_PC87427=m CONFIG_SENSORS_PCF8591=m CONFIG_SENSORS_SHT15=m CONFIG_SENSORS_DME1737=m CONFIG_SENSORS_SMSC47M1=m CONFIG_SENSORS_SMSC47M192=m CONFIG_SENSORS_SMSC47B397=m CONFIG_SENSORS_ADS7828=m CONFIG_SENSORS_AMC6821=m CONFIG_SENSORS_THMC50=m CONFIG_SENSORS_TMP401=m CONFIG_SENSORS_TMP421=m CONFIG_SENSORS_VT1211=m CONFIG_SENSORS_W83781D=m CONFIG_SENSORS_W83791D=m CONFIG_SENSORS_W83792D=m CONFIG_SENSORS_W83793=m CONFIG_SENSORS_W83L785TS=m CONFIG_SENSORS_W83L786NG=m CONFIG_SENSORS_W83627HF=m CONFIG_SENSORS_W83627EHF=m CONFIG_SENSORS_LIS3_SPI=m CONFIG_SENSORS_LIS3_I2C=m # CONFIG_THERMAL is not set CONFIG_WATCHDOG=y CONFIG_WATCHDOG_NOWAYOUT=y # # Watchdog Device Drivers # # CONFIG_SOFT_WATCHDOG is not set CONFIG_OMAP_WATCHDOG=y CONFIG_TWL4030_WATCHDOG=y # # USB-based Watchdog Cards # # CONFIG_USBPCWATCHDOG is not set CONFIG_SSB_POSSIBLE=y # # Sonics Silicon Backplane # # CONFIG_SSB is not set # # Multifunction device drivers # CONFIG_MFD_CORE=y # CONFIG_MFD_SM501 is not set # CONFIG_MFD_ASIC3 is not set # CONFIG_HTC_EGPIO is not set # CONFIG_HTC_PASIC3 is not set # CONFIG_TPS65010 is not set CONFIG_TWL4030_CORE=y CONFIG_TWL4030_POWER=y CONFIG_TWL4030_CODEC=y CONFIG_TWL4030_MADC=y # CONFIG_MFD_TMIO is not set # CONFIG_MFD_T7L66XB is not set # CONFIG_MFD_TC6387XB is not set # CONFIG_MFD_TC6393XB is not set # CONFIG_PMIC_DA903X is not set # CONFIG_PMIC_ADP5520 is not set # CONFIG_MFD_WM8400 is not set # CONFIG_MFD_WM831X is not set # CONFIG_MFD_WM8350_I2C is not set # CONFIG_MFD_PCF50633 is not set # CONFIG_MFD_MC13783 is not set # CONFIG_AB3100_CORE is not set # CONFIG_EZX_PCAP is not set # CONFIG_MFD_88PM8607 is not set # CONFIG_AB4500_CORE is not set CONFIG_REGULATOR=y CONFIG_REGULATOR_DEBUG=y # CONFIG_REGULATOR_FIXED_VOLTAGE is not set # CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set # CONFIG_REGULATOR_USERSPACE_CONSUMER is not set # CONFIG_REGULATOR_BQ24022 is not set # CONFIG_REGULATOR_MAX1586 is not set # CONFIG_REGULATOR_MAX8660 is not set CONFIG_REGULATOR_TWL4030=y # CONFIG_REGULATOR_LP3971 is not set # CONFIG_REGULATOR_TPS65023 is not set # CONFIG_REGULATOR_TPS6507X is not set CONFIG_MEDIA_SUPPORT=m # # Multimedia core support # CONFIG_VIDEO_DEV=m CONFIG_VIDEO_V4L2_COMMON=m CONFIG_VIDEO_ALLOW_V4L1=y CONFIG_VIDEO_V4L1_COMPAT=y CONFIG_DVB_CORE=m CONFIG_VIDEO_MEDIA=m # # Multimedia drivers # CONFIG_IR_CORE=m CONFIG_VIDEO_IR=m CONFIG_MEDIA_ATTACH=y CONFIG_MEDIA_TUNER=m # CONFIG_MEDIA_TUNER_CUSTOMISE is not set CONFIG_MEDIA_TUNER_SIMPLE=m CONFIG_MEDIA_TUNER_TDA8290=m CONFIG_MEDIA_TUNER_TDA827X=m CONFIG_MEDIA_TUNER_TDA18271=m CONFIG_MEDIA_TUNER_TDA9887=m CONFIG_MEDIA_TUNER_TEA5761=m CONFIG_MEDIA_TUNER_TEA5767=m CONFIG_MEDIA_TUNER_MT20XX=m CONFIG_MEDIA_TUNER_MT2060=m CONFIG_MEDIA_TUNER_MT2266=m CONFIG_MEDIA_TUNER_QT1010=m CONFIG_MEDIA_TUNER_XC2028=m CONFIG_MEDIA_TUNER_XC5000=m CONFIG_MEDIA_TUNER_MXL5005S=m CONFIG_MEDIA_TUNER_MXL5007T=m CONFIG_MEDIA_TUNER_MC44S803=m CONFIG_MEDIA_TUNER_MAX2165=m CONFIG_VIDEO_V4L2=m CONFIG_VIDEO_V4L1=m CONFIG_VIDEOBUF_GEN=m CONFIG_VIDEOBUF_VMALLOC=m CONFIG_VIDEOBUF_DVB=m CONFIG_VIDEO_TVEEPROM=m CONFIG_VIDEO_TUNER=m CONFIG_VIDEO_CAPTURE_DRIVERS=y # CONFIG_VIDEO_ADV_DEBUG is not set # CONFIG_VIDEO_FIXED_MINOR_RANGES is not set CONFIG_VIDEO_HELPER_CHIPS_AUTO=y CONFIG_VIDEO_IR_I2C=m CONFIG_VIDEO_MSP3400=m CONFIG_VIDEO_CS53L32A=m CONFIG_VIDEO_WM8775=m CONFIG_VIDEO_MT9V011=m CONFIG_VIDEO_SAA711X=m CONFIG_VIDEO_TVP5150=m CONFIG_VIDEO_CX25840=m CONFIG_VIDEO_CX2341X=m # CONFIG_VIDEO_VIVI is not set # CONFIG_VIDEO_CPIA is not set # CONFIG_VIDEO_CPIA2 is not set # CONFIG_VIDEO_SAA5246A is not set # CONFIG_VIDEO_SAA5249 is not set # CONFIG_VIDEO_AU0828 is not set # CONFIG_SOC_CAMERA is not set CONFIG_V4L_USB_DRIVERS=y CONFIG_USB_VIDEO_CLASS=m CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y CONFIG_USB_GSPCA=m CONFIG_USB_M5602=m CONFIG_USB_STV06XX=m # CONFIG_USB_GL860 is not set # CONFIG_USB_GSPCA_BENQ is not set CONFIG_USB_GSPCA_CONEX=m # CONFIG_USB_GSPCA_CPIA1 is not set CONFIG_USB_GSPCA_ETOMS=m CONFIG_USB_GSPCA_FINEPIX=m # CONFIG_USB_GSPCA_JEILINJ is not set CONFIG_USB_GSPCA_MARS=m # CONFIG_USB_GSPCA_MR97310A is not set CONFIG_USB_GSPCA_OV519=m CONFIG_USB_GSPCA_OV534=m # CONFIG_USB_GSPCA_OV534_9 is not set CONFIG_USB_GSPCA_PAC207=m # CONFIG_USB_GSPCA_PAC7302 is not set CONFIG_USB_GSPCA_PAC7311=m # CONFIG_USB_GSPCA_SN9C2028 is not set # CONFIG_USB_GSPCA_SN9C20X is not set CONFIG_USB_GSPCA_SONIXB=m CONFIG_USB_GSPCA_SONIXJ=m CONFIG_USB_GSPCA_SPCA500=m CONFIG_USB_GSPCA_SPCA501=m CONFIG_USB_GSPCA_SPCA505=m CONFIG_USB_GSPCA_SPCA506=m CONFIG_USB_GSPCA_SPCA508=m CONFIG_USB_GSPCA_SPCA561=m # CONFIG_USB_GSPCA_SQ905 is not set # CONFIG_USB_GSPCA_SQ905C is not set CONFIG_USB_GSPCA_STK014=m # CONFIG_USB_GSPCA_STV0680 is not set CONFIG_USB_GSPCA_SUNPLUS=m CONFIG_USB_GSPCA_T613=m CONFIG_USB_GSPCA_TV8532=m CONFIG_USB_GSPCA_VC032X=m CONFIG_USB_GSPCA_ZC3XX=m CONFIG_VIDEO_PVRUSB2=m CONFIG_VIDEO_PVRUSB2_SYSFS=y CONFIG_VIDEO_PVRUSB2_DVB=y # CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set CONFIG_VIDEO_HDPVR=m CONFIG_VIDEO_EM28XX=m CONFIG_VIDEO_EM28XX_ALSA=m CONFIG_VIDEO_EM28XX_DVB=m # CONFIG_VIDEO_TLG2300 is not set CONFIG_VIDEO_CX231XX=m CONFIG_VIDEO_CX231XX_ALSA=m CONFIG_VIDEO_CX231XX_DVB=m CONFIG_VIDEO_USBVISION=m CONFIG_VIDEO_USBVIDEO=m CONFIG_USB_VICAM=m CONFIG_USB_IBMCAM=m CONFIG_USB_KONICAWC=m CONFIG_USB_QUICKCAM_MESSENGER=m CONFIG_USB_ET61X251=m CONFIG_VIDEO_OVCAMCHIP=m CONFIG_USB_W9968CF=m CONFIG_USB_OV511=m CONFIG_USB_SE401=m CONFIG_USB_SN9C102=m CONFIG_USB_STV680=m CONFIG_USB_ZC0301=m CONFIG_USB_PWC=m # CONFIG_USB_PWC_DEBUG is not set CONFIG_USB_PWC_INPUT_EVDEV=y CONFIG_USB_ZR364XX=m CONFIG_USB_STKWEBCAM=m CONFIG_USB_S2255=m CONFIG_RADIO_ADAPTERS=y # CONFIG_I2C_SI4713 is not set # CONFIG_RADIO_SI4713 is not set # CONFIG_USB_DSBR is not set # CONFIG_RADIO_SI470X is not set # CONFIG_USB_MR800 is not set # CONFIG_RADIO_TEA5764 is not set # CONFIG_RADIO_SAA7706H is not set # CONFIG_RADIO_TEF6862 is not set CONFIG_DVB_MAX_ADAPTERS=8 # CONFIG_DVB_DYNAMIC_MINORS is not set CONFIG_DVB_CAPTURE_DRIVERS=y # CONFIG_TTPCI_EEPROM is not set # # Supported USB Adapters # CONFIG_DVB_USB=m # CONFIG_DVB_USB_DEBUG is not set CONFIG_DVB_USB_A800=m CONFIG_DVB_USB_DIBUSB_MB=m # CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set CONFIG_DVB_USB_DIBUSB_MC=m CONFIG_DVB_USB_DIB0700=m CONFIG_DVB_USB_UMT_010=m CONFIG_DVB_USB_CXUSB=m CONFIG_DVB_USB_M920X=m CONFIG_DVB_USB_GL861=m CONFIG_DVB_USB_AU6610=m CONFIG_DVB_USB_DIGITV=m CONFIG_DVB_USB_VP7045=m CONFIG_DVB_USB_VP702X=m CONFIG_DVB_USB_GP8PSK=m CONFIG_DVB_USB_NOVA_T_USB2=m CONFIG_DVB_USB_TTUSB2=m CONFIG_DVB_USB_DTT200U=m CONFIG_DVB_USB_OPERA1=m CONFIG_DVB_USB_AF9005=m CONFIG_DVB_USB_AF9005_REMOTE=m CONFIG_DVB_USB_DW2102=m CONFIG_DVB_USB_CINERGY_T2=m CONFIG_DVB_USB_ANYSEE=m CONFIG_DVB_USB_DTV5100=m CONFIG_DVB_USB_AF9015=m CONFIG_DVB_USB_CE6230=m CONFIG_DVB_USB_FRIIO=m CONFIG_DVB_USB_EC168=m # CONFIG_DVB_USB_AZ6027 is not set CONFIG_SMS_SIANO_MDTV=m # # Siano module components # # CONFIG_SMS_USB_DRV is not set # CONFIG_SMS_SDIO_DRV is not set # # Supported FlexCopII (B2C2) Adapters # CONFIG_DVB_B2C2_FLEXCOP=m CONFIG_DVB_B2C2_FLEXCOP_USB=m # CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set # # Supported DVB Frontends # # CONFIG_DVB_FE_CUSTOMISE is not set CONFIG_DVB_STB6100=m CONFIG_DVB_CX24123=m CONFIG_DVB_MT312=m CONFIG_DVB_ZL10039=m CONFIG_DVB_S5H1420=m CONFIG_DVB_STV0288=m CONFIG_DVB_STB6000=m CONFIG_DVB_STV0299=m CONFIG_DVB_STV6110=m CONFIG_DVB_STV0900=m CONFIG_DVB_TDA10086=m CONFIG_DVB_TUNER_ITD1000=m CONFIG_DVB_TUNER_CX24113=m CONFIG_DVB_TDA826X=m CONFIG_DVB_CX24116=m CONFIG_DVB_SI21XX=m CONFIG_DVB_DS3000=m CONFIG_DVB_CX22702=m CONFIG_DVB_TDA1004X=m CONFIG_DVB_NXT6000=m CONFIG_DVB_MT352=m CONFIG_DVB_ZL10353=m CONFIG_DVB_DIB3000MB=m CONFIG_DVB_DIB3000MC=m CONFIG_DVB_DIB7000M=m CONFIG_DVB_DIB7000P=m CONFIG_DVB_TDA10048=m CONFIG_DVB_AF9013=m CONFIG_DVB_EC100=m CONFIG_DVB_TDA10021=m CONFIG_DVB_TDA10023=m CONFIG_DVB_STV0297=m CONFIG_DVB_NXT200X=m CONFIG_DVB_BCM3510=m CONFIG_DVB_LGDT330X=m CONFIG_DVB_LGDT3305=m CONFIG_DVB_S5H1409=m CONFIG_DVB_S5H1411=m CONFIG_DVB_DIB8000=m CONFIG_DVB_PLL=m CONFIG_DVB_TUNER_DIB0070=m CONFIG_DVB_LNBP21=m CONFIG_DVB_ISL6421=m CONFIG_DVB_LGS8GXX=m CONFIG_DVB_ATBM8830=m # CONFIG_DAB is not set # # Graphics support # # CONFIG_VGASTATE is not set # CONFIG_VIDEO_OUTPUT_CONTROL is not set CONFIG_FB=y # CONFIG_FIRMWARE_EDID is not set # CONFIG_FB_DDC is not set # CONFIG_FB_BOOT_VESA_SUPPORT is not set CONFIG_FB_CFB_FILLRECT=y CONFIG_FB_CFB_COPYAREA=y CONFIG_FB_CFB_IMAGEBLIT=y # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set # CONFIG_FB_SYS_FILLRECT is not set # CONFIG_FB_SYS_COPYAREA is not set # CONFIG_FB_SYS_IMAGEBLIT is not set # CONFIG_FB_FOREIGN_ENDIAN is not set # CONFIG_FB_SYS_FOPS is not set # CONFIG_FB_SVGALIB is not set # CONFIG_FB_MACMODES is not set # CONFIG_FB_BACKLIGHT is not set # CONFIG_FB_MODE_HELPERS is not set # CONFIG_FB_TILEBLITTING is not set # # Frame buffer hardware drivers # # CONFIG_FB_S1D13XXX is not set # CONFIG_FB_TMIO is not set # CONFIG_FB_VIRTUAL is not set # CONFIG_FB_METRONOME is not set # CONFIG_FB_MB862XX is not set # CONFIG_FB_BROADSHEET is not set # CONFIG_FB_OMAP_BOOTLOADER_INIT is not set CONFIG_OMAP2_VRAM=y CONFIG_OMAP2_VRFB=y CONFIG_OMAP2_DSS=y CONFIG_OMAP2_VRAM_SIZE=0 CONFIG_OMAP2_DSS_DEBUG_SUPPORT=y # CONFIG_OMAP2_DSS_COLLECT_IRQ_STATS is not set # CONFIG_OMAP2_DSS_RFBI is not set CONFIG_OMAP2_DSS_VENC=y # CONFIG_OMAP2_DSS_SDI is not set CONFIG_OMAP2_DSS_DSI=y CONFIG_OMAP2_DSS_USE_DSI_PLL=y # CONFIG_OMAP2_DSS_FAKE_VSYNC is not set CONFIG_OMAP2_DSS_MIN_FCK_PER_PCK=0 CONFIG_FB_OMAP2=y CONFIG_FB_OMAP2_DEBUG_SUPPORT=y CONFIG_FB_OMAP2_NUM_FBS=3 # # OMAP2/3 Display Device Drivers # CONFIG_PANEL_GENERIC=y CONFIG_PANEL_LGPHILIPS_LB035Q02=y CONFIG_PANEL_SAMSUNG_LTE430WQ_F0C=y CONFIG_PANEL_SHARP_LS037V7DW01=y # CONFIG_PANEL_SHARP_LQ043T1DG01 is not set # CONFIG_PANEL_TAAL is not set # CONFIG_PANEL_TOPPOLY_TDO35S is not set # CONFIG_PANEL_TPO_TD043MTEA1 is not set CONFIG_BACKLIGHT_LCD_SUPPORT=y CONFIG_LCD_CLASS_DEVICE=m # CONFIG_LCD_LMS283GF05 is not set # CONFIG_LCD_LTV350QV is not set # CONFIG_LCD_ILI9320 is not set # CONFIG_LCD_TDO24M is not set # CONFIG_LCD_VGG2432A4 is not set # CONFIG_LCD_PLATFORM is not set CONFIG_BACKLIGHT_CLASS_DEVICE=m CONFIG_BACKLIGHT_GENERIC=m # # Display device support # CONFIG_DISPLAY_SUPPORT=y # # Display hardware drivers # # # Console display driver support # # CONFIG_VGA_CONSOLE is not set CONFIG_DUMMY_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE=y # CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y CONFIG_FONTS=y CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y # CONFIG_FONT_6x11 is not set # CONFIG_FONT_7x14 is not set # CONFIG_FONT_PEARL_8x8 is not set # CONFIG_FONT_ACORN_8x8 is not set # CONFIG_FONT_MINI_4x6 is not set # CONFIG_FONT_SUN8x16 is not set # CONFIG_FONT_SUN12x22 is not set # CONFIG_FONT_10x18 is not set CONFIG_LOGO=y CONFIG_LOGO_LINUX_MONO=y CONFIG_LOGO_LINUX_VGA16=y CONFIG_LOGO_LINUX_CLUT224=y CONFIG_SOUND=y CONFIG_SOUND_OSS_CORE=y CONFIG_SOUND_OSS_CORE_PRECLAIM=y CONFIG_SND=y CONFIG_SND_TIMER=y CONFIG_SND_PCM=y CONFIG_SND_HWDEP=y CONFIG_SND_RAWMIDI=y CONFIG_SND_JACK=y CONFIG_SND_SEQUENCER=m # CONFIG_SND_SEQ_DUMMY is not set CONFIG_SND_OSSEMUL=y CONFIG_SND_MIXER_OSS=y CONFIG_SND_PCM_OSS=y CONFIG_SND_PCM_OSS_PLUGINS=y CONFIG_SND_SEQUENCER_OSS=y # CONFIG_SND_HRTIMER is not set # CONFIG_SND_DYNAMIC_MINORS is not set CONFIG_SND_SUPPORT_OLD_API=y CONFIG_SND_VERBOSE_PROCFS=y CONFIG_SND_VERBOSE_PRINTK=y CONFIG_SND_DEBUG=y # CONFIG_SND_DEBUG_VERBOSE is not set # CONFIG_SND_PCM_XRUN_DEBUG is not set CONFIG_SND_RAWMIDI_SEQ=m # CONFIG_SND_OPL3_LIB_SEQ is not set # CONFIG_SND_OPL4_LIB_SEQ is not set # CONFIG_SND_SBAWE_SEQ is not set # CONFIG_SND_EMU10K1_SEQ is not set # CONFIG_SND_DRIVERS is not set # CONFIG_SND_ARM is not set # CONFIG_SND_SPI is not set CONFIG_SND_USB=y CONFIG_SND_USB_AUDIO=y # CONFIG_SND_USB_UA101 is not set CONFIG_SND_USB_CAIAQ=m CONFIG_SND_USB_CAIAQ_INPUT=y CONFIG_SND_SOC=y CONFIG_SND_OMAP_SOC=y CONFIG_SND_OMAP_SOC_MCBSP=y CONFIG_SND_OMAP_SOC_OVERO=y CONFIG_SND_OMAP_SOC_IGEP0020=y CONFIG_SND_SOC_I2C_AND_SPI=y # CONFIG_SND_SOC_ALL_CODECS is not set CONFIG_SND_SOC_TWL4030=y CONFIG_SOUND_PRIME=m CONFIG_HID_SUPPORT=y CONFIG_HID=y # CONFIG_HIDRAW is not set # # USB Input Devices # CONFIG_USB_HID=y # CONFIG_HID_PID is not set # CONFIG_USB_HIDDEV is not set # # Special HID drivers # # CONFIG_HID_3M_PCT is not set CONFIG_HID_A4TECH=y CONFIG_HID_APPLE=y CONFIG_HID_BELKIN=y CONFIG_HID_CHERRY=y CONFIG_HID_CHICONY=y CONFIG_HID_CYPRESS=y CONFIG_HID_DRAGONRISE=y # CONFIG_DRAGONRISE_FF is not set CONFIG_HID_EZKEY=y CONFIG_HID_KYE=y CONFIG_HID_GYRATION=y CONFIG_HID_TWINHAN=y CONFIG_HID_KENSINGTON=y CONFIG_HID_LOGITECH=y # CONFIG_LOGITECH_FF is not set # CONFIG_LOGIRUMBLEPAD2_FF is not set # CONFIG_LOGIG940_FF is not set # CONFIG_HID_MAGICMOUSE is not set CONFIG_HID_MICROSOFT=y # CONFIG_HID_MOSART is not set CONFIG_HID_MONTEREY=y CONFIG_HID_NTRIG=y # CONFIG_HID_ORTEK is not set CONFIG_HID_PANTHERLORD=y # CONFIG_PANTHERLORD_FF is not set CONFIG_HID_PETALYNX=y # CONFIG_HID_QUANTA is not set CONFIG_HID_SAMSUNG=y CONFIG_HID_SONY=y # CONFIG_HID_STANTUM is not set CONFIG_HID_SUNPLUS=y CONFIG_HID_GREENASIA=y # CONFIG_GREENASIA_FF is not set CONFIG_HID_SMARTJOYPLUS=y # CONFIG_SMARTJOYPLUS_FF is not set CONFIG_HID_TOPSEED=y CONFIG_HID_THRUSTMASTER=y # CONFIG_THRUSTMASTER_FF is not set CONFIG_HID_WACOM=y CONFIG_HID_ZEROPLUS=y # CONFIG_ZEROPLUS_FF is not set CONFIG_USB_SUPPORT=y CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB_ARCH_HAS_OHCI=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_USB=y CONFIG_USB_DEBUG=y CONFIG_USB_ANNOUNCE_NEW_DEVICES=y # # Miscellaneous USB options # CONFIG_USB_DEVICEFS=y CONFIG_USB_DEVICE_CLASS=y # CONFIG_USB_DYNAMIC_MINORS is not set CONFIG_USB_SUSPEND=y # CONFIG_USB_OTG is not set # CONFIG_USB_OTG_WHITELIST is not set # CONFIG_USB_OTG_BLACKLIST_HUB is not set CONFIG_USB_MON=y # CONFIG_USB_WUSB is not set # CONFIG_USB_WUSB_CBAF is not set # # USB Host Controller Drivers # # CONFIG_USB_C67X00_HCD is not set CONFIG_USB_EHCI_HCD=y CONFIG_USB_EHCI_ROOT_HUB_TT=y CONFIG_USB_EHCI_TT_NEWSCHED=y # CONFIG_USB_OXU210HP_HCD is not set # CONFIG_USB_ISP116X_HCD is not set # CONFIG_USB_ISP1760_HCD is not set # CONFIG_USB_ISP1362_HCD is not set # CONFIG_USB_OHCI_HCD is not set # CONFIG_USB_U132_HCD is not set # CONFIG_USB_SL811_HCD is not set # CONFIG_USB_R8A66597_HCD is not set # CONFIG_USB_HWA_HCD is not set CONFIG_USB_MUSB_HDRC=y CONFIG_USB_MUSB_SOC=y # # OMAP 343x high speed USB support # CONFIG_USB_MUSB_HOST=y # CONFIG_USB_MUSB_PERIPHERAL is not set # CONFIG_USB_MUSB_OTG is not set # CONFIG_USB_GADGET_MUSB_HDRC is not set CONFIG_USB_MUSB_HDRC_HCD=y # CONFIG_MUSB_PIO_ONLY is not set CONFIG_USB_INVENTRA_DMA=y # CONFIG_USB_TI_CPPI_DMA is not set CONFIG_USB_MUSB_DEBUG=y # # USB Device Class drivers # CONFIG_USB_ACM=m CONFIG_USB_PRINTER=m CONFIG_USB_WDM=m CONFIG_USB_TMC=m # # NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may # # # also be needed; see USB_STORAGE Help for more info # CONFIG_USB_STORAGE=y # CONFIG_USB_STORAGE_DEBUG is not set # CONFIG_USB_STORAGE_DATAFAB is not set CONFIG_USB_STORAGE_FREECOM=m CONFIG_USB_STORAGE_ISD200=m CONFIG_USB_STORAGE_USBAT=m CONFIG_USB_STORAGE_SDDR09=m CONFIG_USB_STORAGE_SDDR55=m CONFIG_USB_STORAGE_JUMPSHOT=m CONFIG_USB_STORAGE_ALAUDA=m CONFIG_USB_STORAGE_ONETOUCH=m CONFIG_USB_STORAGE_KARMA=m CONFIG_USB_STORAGE_CYPRESS_ATACB=m # CONFIG_USB_LIBUSUAL is not set # # USB Imaging devices # CONFIG_USB_MDC800=m CONFIG_USB_MICROTEK=m # # USB port drivers # CONFIG_USB_SERIAL=m CONFIG_USB_EZUSB=y CONFIG_USB_SERIAL_GENERIC=y CONFIG_USB_SERIAL_AIRCABLE=m CONFIG_USB_SERIAL_ARK3116=m CONFIG_USB_SERIAL_BELKIN=m CONFIG_USB_SERIAL_CH341=m CONFIG_USB_SERIAL_WHITEHEAT=m CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m CONFIG_USB_SERIAL_CP210X=m CONFIG_USB_SERIAL_CYPRESS_M8=m CONFIG_USB_SERIAL_EMPEG=m CONFIG_USB_SERIAL_FTDI_SIO=m CONFIG_USB_SERIAL_FUNSOFT=m CONFIG_USB_SERIAL_VISOR=m CONFIG_USB_SERIAL_IPAQ=m CONFIG_USB_SERIAL_IR=m CONFIG_USB_SERIAL_EDGEPORT=m CONFIG_USB_SERIAL_EDGEPORT_TI=m CONFIG_USB_SERIAL_GARMIN=m CONFIG_USB_SERIAL_IPW=m CONFIG_USB_SERIAL_IUU=m CONFIG_USB_SERIAL_KEYSPAN_PDA=m CONFIG_USB_SERIAL_KEYSPAN=m CONFIG_USB_SERIAL_KEYSPAN_MPR=y CONFIG_USB_SERIAL_KEYSPAN_USA28=y CONFIG_USB_SERIAL_KEYSPAN_USA28X=y CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y CONFIG_USB_SERIAL_KEYSPAN_USA19=y CONFIG_USB_SERIAL_KEYSPAN_USA18X=y CONFIG_USB_SERIAL_KEYSPAN_USA19W=y CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y CONFIG_USB_SERIAL_KEYSPAN_USA49W=y CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y CONFIG_USB_SERIAL_KLSI=m CONFIG_USB_SERIAL_KOBIL_SCT=m CONFIG_USB_SERIAL_MCT_U232=m CONFIG_USB_SERIAL_MOS7720=m CONFIG_USB_SERIAL_MOS7840=m CONFIG_USB_SERIAL_MOTOROLA=m CONFIG_USB_SERIAL_NAVMAN=m CONFIG_USB_SERIAL_PL2303=m CONFIG_USB_SERIAL_OTI6858=m CONFIG_USB_SERIAL_QUALCOMM=m CONFIG_USB_SERIAL_SPCP8X5=m CONFIG_USB_SERIAL_HP4X=m CONFIG_USB_SERIAL_SAFE=m # CONFIG_USB_SERIAL_SAFE_PADDED is not set CONFIG_USB_SERIAL_SIEMENS_MPI=m CONFIG_USB_SERIAL_SIERRAWIRELESS=m CONFIG_USB_SERIAL_SYMBOL=m CONFIG_USB_SERIAL_TI=m CONFIG_USB_SERIAL_CYBERJACK=m CONFIG_USB_SERIAL_XIRCOM=m CONFIG_USB_SERIAL_OPTION=m CONFIG_USB_SERIAL_OMNINET=m CONFIG_USB_SERIAL_OPTICON=m # CONFIG_USB_SERIAL_DEBUG is not set # # USB Miscellaneous drivers # CONFIG_USB_EMI62=m CONFIG_USB_EMI26=m CONFIG_USB_ADUTUX=m CONFIG_USB_SEVSEG=m CONFIG_USB_RIO500=m CONFIG_USB_LEGOTOWER=m CONFIG_USB_LCD=m CONFIG_USB_BERRY_CHARGE=m CONFIG_USB_LED=m CONFIG_USB_CYPRESS_CY7C63=m CONFIG_USB_CYTHERM=m CONFIG_USB_IDMOUSE=m CONFIG_USB_FTDI_ELAN=m CONFIG_USB_APPLEDISPLAY=m CONFIG_USB_SISUSBVGA=m CONFIG_USB_SISUSBVGA_CON=y CONFIG_USB_LD=m CONFIG_USB_TRANCEVIBRATOR=m CONFIG_USB_IOWARRIOR=m # CONFIG_USB_TEST is not set CONFIG_USB_ISIGHTFW=m CONFIG_USB_VST=m CONFIG_USB_GADGET=y # CONFIG_USB_GADGET_DEBUG_FILES is not set # CONFIG_USB_GADGET_DEBUG_FS is not set CONFIG_USB_GADGET_VBUS_DRAW=500 CONFIG_USB_GADGET_SELECTED=y # CONFIG_USB_GADGET_AT91 is not set # CONFIG_USB_GADGET_ATMEL_USBA is not set # CONFIG_USB_GADGET_FSL_USB2 is not set # CONFIG_USB_GADGET_LH7A40X is not set CONFIG_USB_GADGET_OMAP=y CONFIG_USB_OMAP=y # CONFIG_USB_GADGET_PXA25X is not set # CONFIG_USB_GADGET_R8A66597 is not set # CONFIG_USB_GADGET_PXA27X is not set # CONFIG_USB_GADGET_S3C_HSOTG is not set # CONFIG_USB_GADGET_IMX is not set # CONFIG_USB_GADGET_S3C2410 is not set # CONFIG_USB_GADGET_M66592 is not set # CONFIG_USB_GADGET_AMD5536UDC is not set # CONFIG_USB_GADGET_FSL_QE is not set # CONFIG_USB_GADGET_CI13XXX is not set # CONFIG_USB_GADGET_NET2280 is not set # CONFIG_USB_GADGET_GOKU is not set # CONFIG_USB_GADGET_LANGWELL is not set # CONFIG_USB_GADGET_DUMMY_HCD is not set # CONFIG_USB_GADGET_DUALSPEED is not set # CONFIG_USB_ZERO is not set CONFIG_USB_AUDIO=m CONFIG_USB_ETH=m CONFIG_USB_ETH_RNDIS=y # CONFIG_USB_ETH_EEM is not set # CONFIG_USB_GADGETFS is not set CONFIG_USB_FILE_STORAGE=m # CONFIG_USB_FILE_STORAGE_TEST is not set # CONFIG_USB_MASS_STORAGE is not set CONFIG_USB_G_SERIAL=m # CONFIG_USB_MIDI_GADGET is not set CONFIG_USB_G_PRINTER=m CONFIG_USB_CDC_COMPOSITE=m # CONFIG_USB_G_MULTI is not set # # OTG and related infrastructure # CONFIG_USB_OTG_UTILS=y # CONFIG_USB_GPIO_VBUS is not set # CONFIG_ISP1301_OMAP is not set # CONFIG_USB_ULPI is not set CONFIG_TWL4030_USB=y # CONFIG_NOP_USB_XCEIV is not set CONFIG_MMC=y # CONFIG_MMC_DEBUG is not set # CONFIG_MMC_UNSAFE_RESUME is not set # # MMC/SD/SDIO Card Drivers # CONFIG_MMC_BLOCK=y CONFIG_MMC_BLOCK_BOUNCE=y CONFIG_SDIO_UART=y # CONFIG_MMC_TEST is not set # # MMC/SD/SDIO Host Controller Drivers # # CONFIG_MMC_SDHCI is not set # CONFIG_MMC_OMAP is not set CONFIG_MMC_OMAP_HS=y # CONFIG_MMC_AT91 is not set # CONFIG_MMC_ATMELMCI is not set # CONFIG_MMC_SPI is not set # CONFIG_MEMSTICK is not set CONFIG_NEW_LEDS=y CONFIG_LEDS_CLASS=y # # LED drivers # # CONFIG_LEDS_PCA9532 is not set CONFIG_LEDS_GPIO=y CONFIG_LEDS_GPIO_PLATFORM=y # CONFIG_LEDS_LP3944 is not set CONFIG_LEDS_PCA955X=m # CONFIG_LEDS_DAC124S085 is not set # CONFIG_LEDS_REGULATOR is not set # CONFIG_LEDS_BD2802 is not set # CONFIG_LEDS_LT3593 is not set # # LED Triggers # CONFIG_LEDS_TRIGGERS=y CONFIG_LEDS_TRIGGER_TIMER=y CONFIG_LEDS_TRIGGER_HEARTBEAT=y CONFIG_LEDS_TRIGGER_BACKLIGHT=y CONFIG_LEDS_TRIGGER_GPIO=y CONFIG_LEDS_TRIGGER_DEFAULT_ON=y # # iptables trigger is under Netfilter config (LED target) # # CONFIG_ACCESSIBILITY is not set CONFIG_RTC_LIB=y CONFIG_RTC_CLASS=y CONFIG_RTC_HCTOSYS=y CONFIG_RTC_HCTOSYS_DEVICE="rtc0" # CONFIG_RTC_DEBUG is not set # # RTC interfaces # CONFIG_RTC_INTF_SYSFS=y CONFIG_RTC_INTF_PROC=y CONFIG_RTC_INTF_DEV=y # CONFIG_RTC_INTF_DEV_UIE_EMUL is not set # CONFIG_RTC_DRV_TEST is not set # # I2C RTC drivers # # CONFIG_RTC_DRV_DS1307 is not set # CONFIG_RTC_DRV_DS1374 is not set # CONFIG_RTC_DRV_DS1672 is not set # CONFIG_RTC_DRV_MAX6900 is not set # CONFIG_RTC_DRV_RS5C372 is not set # CONFIG_RTC_DRV_ISL1208 is not set # CONFIG_RTC_DRV_X1205 is not set # CONFIG_RTC_DRV_PCF8563 is not set # CONFIG_RTC_DRV_PCF8583 is not set # CONFIG_RTC_DRV_M41T80 is not set # CONFIG_RTC_DRV_BQ32K is not set CONFIG_RTC_DRV_TWL4030=y # CONFIG_RTC_DRV_S35390A is not set # CONFIG_RTC_DRV_FM3130 is not set # CONFIG_RTC_DRV_RX8581 is not set # CONFIG_RTC_DRV_RX8025 is not set # # SPI RTC drivers # # CONFIG_RTC_DRV_M41T94 is not set # CONFIG_RTC_DRV_DS1305 is not set # CONFIG_RTC_DRV_DS1390 is not set # CONFIG_RTC_DRV_MAX6902 is not set # CONFIG_RTC_DRV_R9701 is not set # CONFIG_RTC_DRV_RS5C348 is not set # CONFIG_RTC_DRV_DS3234 is not set # CONFIG_RTC_DRV_PCF2123 is not set # # Platform RTC drivers # # CONFIG_RTC_DRV_CMOS is not set # CONFIG_RTC_DRV_DS1286 is not set # CONFIG_RTC_DRV_DS1511 is not set # CONFIG_RTC_DRV_DS1553 is not set # CONFIG_RTC_DRV_DS1742 is not set # CONFIG_RTC_DRV_STK17TA8 is not set # CONFIG_RTC_DRV_M48T86 is not set # CONFIG_RTC_DRV_M48T35 is not set # CONFIG_RTC_DRV_M48T59 is not set # CONFIG_RTC_DRV_MSM6242 is not set # CONFIG_RTC_DRV_BQ4802 is not set # CONFIG_RTC_DRV_RP5C01 is not set # CONFIG_RTC_DRV_V3020 is not set # # on-CPU RTC drivers # # CONFIG_DMADEVICES is not set # CONFIG_AUXDISPLAY is not set # CONFIG_UIO is not set # # TI VLYNQ # # CONFIG_STAGING is not set # # CBUS support # # CONFIG_CBUS is not set # # File systems # CONFIG_EXT2_FS=y # CONFIG_EXT2_FS_XATTR is not set # CONFIG_EXT2_FS_XIP is not set CONFIG_EXT3_FS=y # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set # CONFIG_EXT3_FS_XATTR is not set # CONFIG_EXT4_FS is not set CONFIG_JBD=y # CONFIG_JBD_DEBUG is not set # CONFIG_REISERFS_FS is not set # CONFIG_JFS_FS is not set CONFIG_FS_POSIX_ACL=y # CONFIG_XFS_FS is not set # CONFIG_GFS2_FS is not set # CONFIG_OCFS2_FS is not set # CONFIG_BTRFS_FS is not set # CONFIG_NILFS2_FS is not set CONFIG_FILE_LOCKING=y CONFIG_FSNOTIFY=y CONFIG_DNOTIFY=y CONFIG_INOTIFY=y CONFIG_INOTIFY_USER=y CONFIG_QUOTA=y # CONFIG_QUOTA_NETLINK_INTERFACE is not set CONFIG_PRINT_QUOTA_WARNING=y CONFIG_QUOTA_TREE=y # CONFIG_QFMT_V1 is not set CONFIG_QFMT_V2=y CONFIG_QUOTACTL=y # CONFIG_AUTOFS_FS is not set # CONFIG_AUTOFS4_FS is not set CONFIG_FUSE_FS=m # CONFIG_CUSE is not set CONFIG_GENERIC_ACL=y # # Caches # # CONFIG_FSCACHE is not set # # CD-ROM/DVD Filesystems # CONFIG_ISO9660_FS=m CONFIG_JOLIET=y CONFIG_ZISOFS=y CONFIG_UDF_FS=m CONFIG_UDF_NLS=y # # DOS/FAT/NT Filesystems # CONFIG_FAT_FS=y CONFIG_MSDOS_FS=y CONFIG_VFAT_FS=y CONFIG_FAT_DEFAULT_CODEPAGE=437 CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1" # CONFIG_NTFS_FS is not set # # Pseudo filesystems # CONFIG_PROC_FS=y CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y CONFIG_SYSFS=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y # CONFIG_HUGETLB_PAGE is not set # CONFIG_CONFIGFS_FS is not set CONFIG_MISC_FILESYSTEMS=y # CONFIG_ADFS_FS is not set # CONFIG_AFFS_FS is not set # CONFIG_HFS_FS is not set # CONFIG_HFSPLUS_FS is not set # CONFIG_BEFS_FS is not set # CONFIG_BFS_FS is not set # CONFIG_EFS_FS is not set CONFIG_JFFS2_FS=y CONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y # CONFIG_JFFS2_FS_WBUF_VERIFY is not set CONFIG_JFFS2_SUMMARY=y CONFIG_JFFS2_FS_XATTR=y CONFIG_JFFS2_FS_POSIX_ACL=y CONFIG_JFFS2_FS_SECURITY=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=y CONFIG_JFFS2_LZO=y CONFIG_JFFS2_RTIME=y CONFIG_JFFS2_RUBIN=y # CONFIG_JFFS2_CMODE_NONE is not set CONFIG_JFFS2_CMODE_PRIORITY=y # CONFIG_JFFS2_CMODE_SIZE is not set # CONFIG_JFFS2_CMODE_FAVOURLZO is not set CONFIG_UBIFS_FS=y # CONFIG_UBIFS_FS_XATTR is not set # CONFIG_UBIFS_FS_ADVANCED_COMPR is not set CONFIG_UBIFS_FS_LZO=y CONFIG_UBIFS_FS_ZLIB=y CONFIG_UBIFS_FS_DEBUG=y CONFIG_UBIFS_FS_DEBUG_MSG_LVL=3 # CONFIG_UBIFS_FS_DEBUG_CHKS is not set # CONFIG_CRAMFS is not set # CONFIG_SQUASHFS is not set # CONFIG_VXFS_FS is not set # CONFIG_MINIX_FS is not set # CONFIG_OMFS_FS is not set # CONFIG_HPFS_FS is not set # CONFIG_QNX4FS_FS is not set # CONFIG_ROMFS_FS is not set # CONFIG_SYSV_FS is not set # CONFIG_UFS_FS is not set CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=y CONFIG_NFS_V3=y # CONFIG_NFS_V3_ACL is not set CONFIG_NFS_V4=y # CONFIG_NFS_V4_1 is not set CONFIG_ROOT_NFS=y # CONFIG_NFSD is not set CONFIG_LOCKD=y CONFIG_LOCKD_V4=y CONFIG_NFS_COMMON=y CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y CONFIG_RPCSEC_GSS_KRB5=y # CONFIG_RPCSEC_GSS_SPKM3 is not set # CONFIG_SMB_FS is not set # CONFIG_CIFS is not set # CONFIG_NCP_FS is not set # CONFIG_CODA_FS is not set # CONFIG_AFS_FS is not set # # Partition Types # CONFIG_PARTITION_ADVANCED=y # CONFIG_ACORN_PARTITION is not set # CONFIG_OSF_PARTITION is not set # CONFIG_AMIGA_PARTITION is not set # CONFIG_ATARI_PARTITION is not set # CONFIG_MAC_PARTITION is not set CONFIG_MSDOS_PARTITION=y # CONFIG_BSD_DISKLABEL is not set # CONFIG_MINIX_SUBPARTITION is not set # CONFIG_SOLARIS_X86_PARTITION is not set # CONFIG_UNIXWARE_DISKLABEL is not set # CONFIG_LDM_PARTITION is not set # CONFIG_SGI_PARTITION is not set # CONFIG_ULTRIX_PARTITION is not set # CONFIG_SUN_PARTITION is not set # CONFIG_KARMA_PARTITION is not set # CONFIG_EFI_PARTITION is not set # CONFIG_SYSV68_PARTITION is not set CONFIG_NLS=y CONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_437=y # CONFIG_NLS_CODEPAGE_737 is not set # CONFIG_NLS_CODEPAGE_775 is not set # CONFIG_NLS_CODEPAGE_850 is not set # CONFIG_NLS_CODEPAGE_852 is not set # CONFIG_NLS_CODEPAGE_855 is not set # CONFIG_NLS_CODEPAGE_857 is not set # CONFIG_NLS_CODEPAGE_860 is not set # CONFIG_NLS_CODEPAGE_861 is not set # CONFIG_NLS_CODEPAGE_862 is not set # CONFIG_NLS_CODEPAGE_863 is not set # CONFIG_NLS_CODEPAGE_864 is not set # CONFIG_NLS_CODEPAGE_865 is not set # CONFIG_NLS_CODEPAGE_866 is not set # CONFIG_NLS_CODEPAGE_869 is not set # CONFIG_NLS_CODEPAGE_936 is not set # CONFIG_NLS_CODEPAGE_950 is not set # CONFIG_NLS_CODEPAGE_932 is not set # CONFIG_NLS_CODEPAGE_949 is not set # CONFIG_NLS_CODEPAGE_874 is not set # CONFIG_NLS_ISO8859_8 is not set # CONFIG_NLS_CODEPAGE_1250 is not set # CONFIG_NLS_CODEPAGE_1251 is not set # CONFIG_NLS_ASCII is not set CONFIG_NLS_ISO8859_1=y # CONFIG_NLS_ISO8859_2 is not set # CONFIG_NLS_ISO8859_3 is not set # CONFIG_NLS_ISO8859_4 is not set # CONFIG_NLS_ISO8859_5 is not set # CONFIG_NLS_ISO8859_6 is not set # CONFIG_NLS_ISO8859_7 is not set # CONFIG_NLS_ISO8859_9 is not set # CONFIG_NLS_ISO8859_13 is not set # CONFIG_NLS_ISO8859_14 is not set # CONFIG_NLS_ISO8859_15 is not set # CONFIG_NLS_KOI8_R is not set # CONFIG_NLS_KOI8_U is not set # CONFIG_NLS_UTF8 is not set # CONFIG_DLM is not set # # Kernel hacking # # CONFIG_PRINTK_TIME is not set CONFIG_ENABLE_WARN_DEPRECATED=y CONFIG_ENABLE_MUST_CHECK=y CONFIG_FRAME_WARN=1024 CONFIG_MAGIC_SYSRQ=y # CONFIG_STRIP_ASM_SYMS is not set # CONFIG_UNUSED_SYMBOLS is not set CONFIG_DEBUG_FS=y # CONFIG_HEADERS_CHECK is not set # CONFIG_IPIPE_DEBUG is not set # CONFIG_DEBUG_KERNEL is not set CONFIG_DEBUG_BUGVERBOSE=y # CONFIG_DEBUG_MEMORY_INIT is not set # CONFIG_RCU_CPU_STALL_DETECTOR is not set # CONFIG_LATENCYTOP is not set # CONFIG_SYSCTL_SYSCALL_CHECK is not set CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_RING_BUFFER=y CONFIG_RING_BUFFER_ALLOW_SWAP=y CONFIG_TRACING_SUPPORT=y CONFIG_FTRACE=y # CONFIG_FUNCTION_TRACER is not set # CONFIG_IRQSOFF_TRACER is not set # CONFIG_SCHED_TRACER is not set # CONFIG_ENABLE_DEFAULT_TRACERS is not set # CONFIG_BOOT_TRACER is not set CONFIG_BRANCH_PROFILE_NONE=y # CONFIG_PROFILE_ANNOTATED_BRANCHES is not set # CONFIG_PROFILE_ALL_BRANCHES is not set # CONFIG_STACK_TRACER is not set # CONFIG_KMEMTRACE is not set # CONFIG_WORKQUEUE_TRACER is not set # CONFIG_BLK_DEV_IO_TRACE is not set # CONFIG_RING_BUFFER_BENCHMARK is not set # CONFIG_DYNAMIC_DEBUG is not set # CONFIG_SAMPLES is not set CONFIG_HAVE_ARCH_KGDB=y CONFIG_ARM_UNWIND=y # CONFIG_DEBUG_USER is not set # CONFIG_OC_ETM is not set # # Security options # # CONFIG_KEYS is not set # CONFIG_SECURITY is not set # CONFIG_SECURITYFS is not set # CONFIG_DEFAULT_SECURITY_SELINUX is not set # CONFIG_DEFAULT_SECURITY_SMACK is not set # CONFIG_DEFAULT_SECURITY_TOMOYO is not set CONFIG_DEFAULT_SECURITY_DAC=y CONFIG_DEFAULT_SECURITY="" CONFIG_XOR_BLOCKS=m CONFIG_ASYNC_CORE=m CONFIG_ASYNC_MEMCPY=m CONFIG_ASYNC_XOR=m CONFIG_ASYNC_PQ=m CONFIG_ASYNC_RAID6_RECOV=m CONFIG_CRYPTO=y # # Crypto core or helper # CONFIG_CRYPTO_ALGAPI=y CONFIG_CRYPTO_ALGAPI2=y CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_BLKCIPHER=y CONFIG_CRYPTO_BLKCIPHER2=y CONFIG_CRYPTO_HASH=y CONFIG_CRYPTO_HASH2=y CONFIG_CRYPTO_RNG2=y CONFIG_CRYPTO_PCOMP=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_GF128MUL=m CONFIG_CRYPTO_NULL=m CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_CRYPTD=m # CONFIG_CRYPTO_AUTHENC is not set CONFIG_CRYPTO_TEST=m # # Authenticated Encryption with Associated Data # # CONFIG_CRYPTO_CCM is not set # CONFIG_CRYPTO_GCM is not set # CONFIG_CRYPTO_SEQIV is not set # # Block modes # CONFIG_CRYPTO_CBC=y # CONFIG_CRYPTO_CTR is not set # CONFIG_CRYPTO_CTS is not set CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m # CONFIG_CRYPTO_XTS is not set # # Hash modes # CONFIG_CRYPTO_HMAC=m CONFIG_CRYPTO_XCBC=m # CONFIG_CRYPTO_VMAC is not set # # Digest # CONFIG_CRYPTO_CRC32C=y # CONFIG_CRYPTO_GHASH is not set CONFIG_CRYPTO_MD4=m CONFIG_CRYPTO_MD5=y CONFIG_CRYPTO_MICHAEL_MIC=y # CONFIG_CRYPTO_RMD128 is not set # CONFIG_CRYPTO_RMD160 is not set # CONFIG_CRYPTO_RMD256 is not set # CONFIG_CRYPTO_RMD320 is not set CONFIG_CRYPTO_SHA1=m CONFIG_CRYPTO_SHA256=m CONFIG_CRYPTO_SHA512=m CONFIG_CRYPTO_TGR192=m CONFIG_CRYPTO_WP512=m # # Ciphers # CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_ANUBIS=m CONFIG_CRYPTO_ARC4=y CONFIG_CRYPTO_BLOWFISH=m CONFIG_CRYPTO_CAMELLIA=m CONFIG_CRYPTO_CAST5=m CONFIG_CRYPTO_CAST6=m CONFIG_CRYPTO_DES=y CONFIG_CRYPTO_FCRYPT=m CONFIG_CRYPTO_KHAZAD=m # CONFIG_CRYPTO_SALSA20 is not set # CONFIG_CRYPTO_SEED is not set CONFIG_CRYPTO_SERPENT=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_TWOFISH_COMMON=m # # Compression # CONFIG_CRYPTO_DEFLATE=y # CONFIG_CRYPTO_ZLIB is not set CONFIG_CRYPTO_LZO=y # # Random Number Generation # # CONFIG_CRYPTO_ANSI_CPRNG is not set CONFIG_CRYPTO_HW=y # CONFIG_BINARY_PRINTF is not set # # Library routines # CONFIG_BITREVERSE=y CONFIG_GENERIC_FIND_LAST_BIT=y CONFIG_CRC_CCITT=y CONFIG_CRC16=y CONFIG_CRC_T10DIF=y CONFIG_CRC_ITU_T=y CONFIG_CRC32=y CONFIG_CRC7=y CONFIG_LIBCRC32C=y CONFIG_ZLIB_INFLATE=y CONFIG_ZLIB_DEFLATE=y CONFIG_LZO_COMPRESS=y CONFIG_LZO_DECOMPRESS=y CONFIG_DECOMPRESS_GZIP=y CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT=y CONFIG_HAS_DMA=y CONFIG_NLATTR=y ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 12:22 ` Fredrik Asplund @ 2010-07-01 23:25 ` Felipe Brandão Cavalcanti 2010-07-02 7:33 ` Fredrik Asplund 2010-07-02 8:14 ` Gilles Chanteperdrix 0 siblings, 2 replies; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-01 23:25 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org [-- Attachment #1.1: Type: text/plain, Size: 1775 bytes --] Sorry for the late reply. Anyway, I did enable Xenomai in the kernel itself, so no modules involved - can't modprobe anything. As for the vanilla kernel, I've tried for a while to get it working right on my board, but no luck. I am not sure if it has support to all the peripherals, or if there is something that I am doing wrong (used the defconfig, no modifications). I've edited the bitbake files for the kernel, but what I usually do is to run the unpack script, replace the flush.c file, and keep the process going. The defconfig is pretty much standard, except that I've enabled several debugging options and got rid of a few troublesome pieces of code (power management, CPU Freq. scaling, etc...). As for the edits, they seem very similar to yours, but I didn't look at them very carefully. Fredrik, did you managed to get a kernel compiled at all? On Thu, Jul 1, 2010 at 9:22 AM, Fredrik Asplund <fasplund@domain.hid> wrote: > >>So, any clues? I can send my changes and logs tomorrow morning, if they > are of any help. > >Sure, please do so. I don´t have much time to put on this, so I have > actually only > >have had to go through the source files since yesterday. Today (or during > the weekend > >if I don´t get the time) I will create a new defconfig file, compile, test > and debug. > >If you have updated BitBake file, Defconfig file, Logs, etc., please share > them. > > Felipe - Here is my defconfig (which I have not had time to test yet > though), please share yours. > > / Fredrik > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #1.2: Type: text/html, Size: 2238 bytes --] [-- Attachment #2: xenomai.bb --] [-- Type: application/octet-stream, Size: 788 bytes --] SECTION = "base" DESCRIPTION = "Xenomai 2.5.3" PN = "xenomai" PV = "2.5.3" PR = "r0" DEPENDS = "" FILES_${PN}-dbg = "/usr/xenomai /dev" SRC_URI = " http://download.gna.org/xenomai/stable/xenomai-${PV}.tar.bz2 \ " SRC_URI[md5sum] = "88228badfee1587334460cc5bd0a9fb2" SRC_URI[sha256sum] = "204a62f20cf9e997c06aa9606e136c6a8cc1288d16617eb4c1b8a49fc28c08c4" S = "${WORKDIR}/xenomai-${PV}" inherit autotools do_configure () { ./configure --enable-linux-build=/media/Data-Linux/overo/overo-oe/tmp/work/overo-angstrom-linux-gnueabi/linux-omap3-2.6.33-r78/git --enable-arm-mach=omap3 --host=arm-linux --disable-dox-doc --disable-dbx } do_compile() { oe_runmake } do_install() { oe_runmake 'DESTDIR=/media/Data-Linux/overo/tmp/xenomai/' 'SUDO=/bin/true' install } PACKAGES = "${PN}" [-- Attachment #3: linux-omap3_2.6.33.bb --] [-- Type: application/octet-stream, Size: 2702 bytes --] DESCRIPTION = "Linux kernel for OMAP3 processors" KERNEL_IMAGETYPE = "uImage" COMPATIBLE_MACHINE = "beagleboard|omap3-multi|overo" require linux.inc MUSB_MODE ?= "host" SRCREV = "248cd24b4fd3376f41fd1b4e3554f4a8c4faa8b9" PV = "2.6.33" # The main PR is now using MACHINE_KERNEL_PR, for omap3 see conf/machine/include/omap3.inc SRC_URI = "git://www.sakoman.com/git/linux-omap-2.6.git;branch=omap3-2.6.33;protocol=git \ file://defconfig \ " SRC_URI_append = " \ file://${BOOT_SPLASH} \ " do_configure_prepend() { if [ "${MUSB_MODE}" = "host" ]; then sed -i 's:# CONFIG_USB_MUSB_HOST is not set:CONFIG_USB_MUSB_HOST=y:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_MUSB_PERIPHERAL=y:# CONFIG_USB_MUSB_PERIPHERAL is not set:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_MUSB_OTG=y:# CONFIG_USB_MUSB_OTG is not set:g' ${WORKDIR}/defconfig sed -i 's:# CONFIG_USB_MUSB_HDRC_HCD is not set:CONFIG_USB_MUSB_HDRC_HCD=y:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_GADGET_MUSB_HDRC=y:# CONFIG_USB_GADGET_MUSB_HDRC is not set:g' ${WORKDIR}/defconfig fi if [ "${MUSB_MODE}" = "peripheral" ]; then sed -i 's:CONFIG_USB_MUSB_HOST=y:# CONFIG_USB_MUSB_HOST is not set:g' ${WORKDIR}/defconfig sed -i 's:# CONFIG_USB_MUSB_PERIPHERAL is not set:CONFIG_USB_MUSB_PERIPHERAL=y:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_MUSB_OTG=y:# CONFIG_USB_MUSB_OTG is not set:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_MUSB_HDRC_HCD=y:# CONFIG_USB_MUSB_HDRC_HCD is not set:g' ${WORKDIR}/defconfig sed -i 's:# CONFIG_USB_GADGET_MUSB_HDRC is not set:CONFIG_USB_GADGET_MUSB_HDRC=y:g' ${WORKDIR}/defconfig fi if [ "${MUSB_MODE}" = "otg" ]; then sed -i 's:CONFIG_USB_MUSB_HOST=y:# CONFIG_USB_MUSB_HOST is not set:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_MUSB_PERIPHERAL=y:# CONFIG_USB_MUSB_PERIPHERAL is not set:g' ${WORKDIR}/defconfig sed -i 's:# CONFIG_USB_MUSB_OTG is not set:CONFIG_USB_MUSB_OTG=y:g' ${WORKDIR}/defconfig sed -i 's:CONFIG_USB_MUSB_HDRC_HCD=y:# CONFIG_USB_MUSB_HDRC_HCD is not set:g' ${WORKDIR}/defconfig sed -i 's:# CONFIG_USB_GADGET_MUSB_HDRC is not set:CONFIG_USB_GADGET_MUSB_HDRC=y:g' ${WORKDIR}/defconfig fi /media/Data-Linux/overo/xenomai/xenomai-2.5.3/scripts/prepare-kernel.sh --linux=/media/Data-Linux/overo/overo-oe/tmp/work/overo-angstrom-linux-gnueabi/linux-omap3-2.6.33-r78/git --adeos=/media/Data-Linux/overo/xenomai/xenomai-2.5.3/ksrc/arch/arm/patches/adeos-ipipe-2.6.33-arm-1.16-01.patch --arch=arm } S = "${WORKDIR}/git" [-- Attachment #4: defconfig --] [-- Type: application/octet-stream, Size: 65389 bytes --] # # Automatically generated make config: don't edit # Linux kernel version: 2.6.33 # Mon Jun 28 12:31:37 2010 # CONFIG_ARM=y CONFIG_SYS_SUPPORTS_APM_EMULATION=y CONFIG_GENERIC_GPIO=y CONFIG_GENERIC_TIME=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_HAVE_PROC_CPU=y CONFIG_GENERIC_HARDIRQS=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_HARDIRQS_SW_RESEND=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_RWSEM_GENERIC_SPINLOCK=y CONFIG_ARCH_HAS_CPUFREQ=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y CONFIG_ARM_L1_CACHE_SHIFT_6=y CONFIG_OPROFILE_ARMV7=y CONFIG_VECTORS_BASE=0xffff0000 CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_CONSTRUCTORS=y # # General setup # CONFIG_EXPERIMENTAL=y CONFIG_BROKEN_ON_SMP=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_LZO=y CONFIG_KERNEL_GZIP=y # CONFIG_KERNEL_BZIP2 is not set # CONFIG_KERNEL_LZMA is not set # CONFIG_KERNEL_LZO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y # CONFIG_POSIX_MQUEUE is not set CONFIG_BSD_PROCESS_ACCT=y # CONFIG_BSD_PROCESS_ACCT_V3 is not set # CONFIG_TASKSTATS is not set # CONFIG_AUDIT is not set # # RCU Subsystem # CONFIG_TREE_RCU=y # CONFIG_TREE_PREEMPT_RCU is not set # CONFIG_TINY_RCU is not set # CONFIG_RCU_TRACE is not set CONFIG_RCU_FANOUT=32 # CONFIG_RCU_FANOUT_EXACT is not set # CONFIG_TREE_RCU_TRACE is not set CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y CONFIG_LOG_BUF_SHIFT=15 # CONFIG_CGROUPS is not set # CONFIG_SYSFS_DEPRECATED_V2 is not set # CONFIG_RELAY is not set # CONFIG_NAMESPACES is not set CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="" CONFIG_RD_GZIP=y # CONFIG_RD_BZIP2 is not set # CONFIG_RD_LZMA is not set # CONFIG_RD_LZO is not set CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_ANON_INODES=y CONFIG_EMBEDDED=y CONFIG_UID16=y # CONFIG_SYSCTL_SYSCALL is not set CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y # CONFIG_KALLSYMS_EXTRA_PASS is not set CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y # CONFIG_ELF_CORE is not set CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_HAVE_PERF_EVENTS=y CONFIG_PERF_USE_VMALLOC=y # # Kernel Performance Events And Counters # CONFIG_PERF_EVENTS=y # CONFIG_PERF_COUNTERS is not set # CONFIG_DEBUG_PERF_USE_VMALLOC is not set CONFIG_VM_EVENT_COUNTERS=y # CONFIG_COMPAT_BRK is not set CONFIG_SLAB=y # CONFIG_SLUB is not set # CONFIG_SLOB is not set CONFIG_PROFILING=y CONFIG_TRACEPOINTS=y CONFIG_OPROFILE=y CONFIG_HAVE_OPROFILE=y # CONFIG_KPROBES is not set CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_CLK=y # # GCOV-based kernel profiling # # CONFIG_GCOV_KERNEL is not set # CONFIG_SLOW_WORK is not set CONFIG_HAVE_GENERIC_DMA_COHERENT=y CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y CONFIG_BASE_SMALL=0 CONFIG_MODULES=y # CONFIG_MODULE_FORCE_LOAD is not set CONFIG_MODULE_UNLOAD=y # CONFIG_MODULE_FORCE_UNLOAD is not set CONFIG_MODVERSIONS=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_BLOCK=y CONFIG_LBDAF=y CONFIG_BLK_DEV_BSG=y # CONFIG_BLK_DEV_INTEGRITY is not set # # IO Schedulers # CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y # CONFIG_DEFAULT_DEADLINE is not set CONFIG_DEFAULT_CFQ=y # CONFIG_DEFAULT_NOOP is not set CONFIG_DEFAULT_IOSCHED="cfq" # CONFIG_INLINE_SPIN_TRYLOCK is not set # CONFIG_INLINE_SPIN_TRYLOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK is not set # CONFIG_INLINE_SPIN_LOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK_IRQ is not set # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set CONFIG_INLINE_SPIN_UNLOCK=y # CONFIG_INLINE_SPIN_UNLOCK_BH is not set CONFIG_INLINE_SPIN_UNLOCK_IRQ=y # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_READ_TRYLOCK is not set # CONFIG_INLINE_READ_LOCK is not set # CONFIG_INLINE_READ_LOCK_BH is not set # CONFIG_INLINE_READ_LOCK_IRQ is not set # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set CONFIG_INLINE_READ_UNLOCK=y # CONFIG_INLINE_READ_UNLOCK_BH is not set CONFIG_INLINE_READ_UNLOCK_IRQ=y # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_WRITE_TRYLOCK is not set # CONFIG_INLINE_WRITE_LOCK is not set # CONFIG_INLINE_WRITE_LOCK_BH is not set # CONFIG_INLINE_WRITE_LOCK_IRQ is not set # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set CONFIG_INLINE_WRITE_UNLOCK=y # CONFIG_INLINE_WRITE_UNLOCK_BH is not set CONFIG_INLINE_WRITE_UNLOCK_IRQ=y # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set # CONFIG_MUTEX_SPIN_ON_OWNER is not set # # Real-time sub-system # CONFIG_XENOMAI=y CONFIG_XENO_GENERIC_STACKPOOL=y CONFIG_XENO_FASTSYNCH_DEP=y CONFIG_XENO_FASTSYNCH=y CONFIG_XENO_OPT_NUCLEUS=y CONFIG_XENO_OPT_PERVASIVE=y CONFIG_XENO_OPT_PRIOCPL=y CONFIG_XENO_OPT_PIPELINE_HEAD=y # CONFIG_XENO_OPT_SCHED_CLASSES is not set CONFIG_XENO_OPT_PIPE=y CONFIG_XENO_OPT_PIPE_NRDEV=32 CONFIG_XENO_OPT_REGISTRY_NRSLOTS=512 CONFIG_XENO_OPT_SYS_HEAPSZ=256 CONFIG_XENO_OPT_SYS_STACKPOOLSZ=128 CONFIG_XENO_OPT_SEM_HEAPSZ=12 CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ=12 CONFIG_XENO_OPT_STATS=y CONFIG_XENO_OPT_DEBUG=y CONFIG_XENO_OPT_DEBUG_NUCLEUS=y CONFIG_XENO_OPT_DEBUG_XNLOCK=y CONFIG_XENO_OPT_DEBUG_QUEUES=y CONFIG_XENO_OPT_DEBUG_REGISTRY=y CONFIG_XENO_OPT_DEBUG_TIMERS=y CONFIG_XENO_OPT_DEBUG_SYNCH_RELAX=y CONFIG_XENO_OPT_WATCHDOG=y CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4 # CONFIG_XENO_OPT_SHIRQ is not set # # Timing # # CONFIG_XENO_OPT_TIMING_PERIODIC is not set CONFIG_XENO_OPT_TIMING_VIRTICK=1000 CONFIG_XENO_OPT_TIMING_SCHEDLAT=0 # # Scalability # # CONFIG_XENO_OPT_SCALABLE_SCHED is not set CONFIG_XENO_OPT_TIMER_LIST=y # CONFIG_XENO_OPT_TIMER_HEAP is not set # CONFIG_XENO_OPT_TIMER_WHEEL is not set # # Machine # CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH=y CONFIG_XENO_HW_FPU=y CONFIG_XENO_HW_UNLOCKED_SWITCH=y # # Interfaces # CONFIG_XENO_SKIN_NATIVE=y CONFIG_XENO_OPT_NATIVE_PERIOD=0 CONFIG_XENO_OPT_NATIVE_PIPE=y CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ=1024 CONFIG_XENO_OPT_NATIVE_SEM=y CONFIG_XENO_OPT_NATIVE_EVENT=y CONFIG_XENO_OPT_NATIVE_MUTEX=y CONFIG_XENO_OPT_NATIVE_COND=y CONFIG_XENO_OPT_NATIVE_QUEUE=y CONFIG_XENO_OPT_NATIVE_BUFFER=y CONFIG_XENO_OPT_NATIVE_HEAP=y CONFIG_XENO_OPT_NATIVE_ALARM=y CONFIG_XENO_OPT_NATIVE_MPS=y # CONFIG_XENO_OPT_NATIVE_INTR is not set CONFIG_XENO_OPT_DEBUG_NATIVE=y CONFIG_XENO_SKIN_POSIX=y CONFIG_XENO_OPT_POSIX_PERIOD=0 # CONFIG_XENO_OPT_POSIX_SHM is not set # CONFIG_XENO_OPT_POSIX_INTR is not set # CONFIG_XENO_OPT_POSIX_SELECT is not set # CONFIG_XENO_OPT_DEBUG_POSIX is not set # CONFIG_XENO_SKIN_PSOS is not set # CONFIG_XENO_SKIN_UITRON is not set # CONFIG_XENO_SKIN_VRTX is not set # CONFIG_XENO_SKIN_VXWORKS is not set # CONFIG_XENO_SKIN_RTAI is not set # CONFIG_XENO_OPT_NOWARN_DEPRECATED is not set CONFIG_XENO_SKIN_RTDM=y CONFIG_XENO_OPT_RTDM_PERIOD=0 CONFIG_XENO_OPT_RTDM_FILDES=128 # CONFIG_XENO_OPT_RTDM_SELECT is not set CONFIG_XENO_OPT_DEBUG_RTDM=y CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y # # Drivers # # # Serial drivers # # CONFIG_XENO_DRIVERS_16550A is not set # # Testing drivers # # CONFIG_XENO_DRIVERS_TESTING_LEGACY_NAMES is not set CONFIG_XENO_DRIVERS_TIMERBENCH=y # CONFIG_XENO_DRIVERS_KLATENCY is not set # CONFIG_XENO_DRIVERS_IRQBENCH is not set CONFIG_XENO_DRIVERS_SWITCHTEST=y # CONFIG_XENO_DRIVERS_SIGTEST is not set # CONFIG_XENO_DRIVERS_RTDMTEST is not set # # CAN drivers # # CONFIG_XENO_DRIVERS_CAN is not set # # ANALOGY drivers # # CONFIG_XENO_DRIVERS_ANALOGY is not set # # Real-time IPC drivers # # CONFIG_XENO_DRIVERS_RTIPC is not set CONFIG_FREEZER=y # # System Type # CONFIG_MMU=y # CONFIG_ARCH_AAEC2000 is not set # CONFIG_ARCH_INTEGRATOR is not set # CONFIG_ARCH_REALVIEW is not set # CONFIG_ARCH_VERSATILE is not set # CONFIG_ARCH_AT91 is not set # CONFIG_ARCH_CLPS711X is not set # CONFIG_ARCH_GEMINI is not set # CONFIG_ARCH_EBSA110 is not set # CONFIG_ARCH_EP93XX is not set # CONFIG_ARCH_FOOTBRIDGE is not set # CONFIG_ARCH_MXC is not set # CONFIG_ARCH_STMP3XXX is not set # CONFIG_ARCH_NETX is not set # CONFIG_ARCH_H720X is not set # CONFIG_ARCH_NOMADIK is not set # CONFIG_ARCH_IOP13XX is not set # CONFIG_ARCH_IOP32X is not set # CONFIG_ARCH_IOP33X is not set # CONFIG_ARCH_IXP23XX is not set # CONFIG_ARCH_IXP2000 is not set # CONFIG_ARCH_IXP4XX is not set # CONFIG_ARCH_L7200 is not set # CONFIG_ARCH_DOVE is not set # CONFIG_ARCH_KIRKWOOD is not set # CONFIG_ARCH_LOKI is not set # CONFIG_ARCH_MV78XX0 is not set # CONFIG_ARCH_ORION5X is not set # CONFIG_ARCH_MMP is not set # CONFIG_ARCH_KS8695 is not set # CONFIG_ARCH_NS9XXX is not set # CONFIG_ARCH_W90X900 is not set # CONFIG_ARCH_NUC93X is not set # CONFIG_ARCH_PNX4008 is not set # CONFIG_ARCH_PXA is not set # CONFIG_ARCH_MSM is not set # CONFIG_ARCH_RPC is not set # CONFIG_ARCH_SA1100 is not set # CONFIG_ARCH_S3C2410 is not set # CONFIG_ARCH_S3C64XX is not set # CONFIG_ARCH_S5PC1XX is not set # CONFIG_ARCH_SHARK is not set # CONFIG_ARCH_LH7A40X is not set # CONFIG_ARCH_U300 is not set # CONFIG_ARCH_DAVINCI is not set CONFIG_ARCH_OMAP=y # CONFIG_ARCH_BCMRING is not set # CONFIG_ARCH_U8500 is not set # # TI OMAP Implementations # CONFIG_ARCH_OMAP_OTG=y # CONFIG_ARCH_OMAP1 is not set CONFIG_ARCH_OMAP2PLUS=y # CONFIG_ARCH_OMAP2 is not set CONFIG_ARCH_OMAP3=y # CONFIG_ARCH_OMAP4 is not set # # OMAP Feature Selections # # CONFIG_OMAP_RESET_CLOCKS is not set # CONFIG_OMAP_MUX is not set CONFIG_OMAP_MCBSP=y # CONFIG_OMAP_MBOX_FWK is not set CONFIG_OMAP_MPU_TIMER=y # CONFIG_OMAP_32K_TIMER is not set # CONFIG_OMAP3_L2_AUX_SECURE_SAVE_RESTORE is not set CONFIG_OMAP_DM_TIMER=y # CONFIG_OMAP_PM_NONE is not set CONFIG_OMAP_PM_NOOP=y CONFIG_ARCH_OMAP3430=y CONFIG_OMAP_PACKAGE_CBB=y # # OMAP Board Type # # CONFIG_MACH_OMAP3_BEAGLE is not set # CONFIG_MACH_DEVKIT8000 is not set # CONFIG_MACH_OMAP_LDP is not set CONFIG_MACH_OVERO=y # CONFIG_MACH_OMAP3EVM is not set # CONFIG_MACH_OMAP3517EVM is not set # CONFIG_MACH_OMAP3_PANDORA is not set # CONFIG_MACH_OMAP3_TOUCHBOOK is not set # CONFIG_MACH_OMAP_3430SDP is not set # CONFIG_MACH_NOKIA_RX51 is not set # CONFIG_MACH_OMAP_ZOOM2 is not set # CONFIG_MACH_OMAP_ZOOM3 is not set # CONFIG_MACH_CM_T35 is not set # CONFIG_MACH_IGEP0020 is not set # CONFIG_MACH_OMAP_3630SDP is not set # CONFIG_OMAP3_EMU is not set # CONFIG_OMAP3_SDRC_AC_TIMING is not set # # Processor Type # CONFIG_CPU_32v6K=y CONFIG_CPU_V7=y CONFIG_CPU_32v7=y CONFIG_CPU_ABRT_EV7=y CONFIG_CPU_PABRT_V7=y CONFIG_CPU_CACHE_V7=y CONFIG_CPU_CACHE_VIPT=y CONFIG_CPU_COPY_V6=y CONFIG_CPU_TLB_V7=y CONFIG_CPU_HAS_ASID=y CONFIG_CPU_CP15=y CONFIG_CPU_CP15_MMU=y # # Processor Features # CONFIG_ARM_THUMB=y CONFIG_ARM_THUMBEE=y # CONFIG_CPU_ICACHE_DISABLE is not set # CONFIG_CPU_DCACHE_DISABLE is not set # CONFIG_CPU_BPREDICT_DISABLE is not set CONFIG_HAS_TLS_REG=y CONFIG_ARM_L1_CACHE_SHIFT=6 CONFIG_CPU_HAS_PMU=y CONFIG_ARM_ERRATA_430973=y # CONFIG_ARM_ERRATA_458693 is not set # CONFIG_ARM_ERRATA_460075 is not set CONFIG_COMMON_CLKDEV=y # # Bus support # # CONFIG_PCI_SYSCALL is not set # CONFIG_ARCH_SUPPORTS_MSI is not set # CONFIG_PCCARD is not set # # Kernel Features # CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_VMSPLIT_3G=y # CONFIG_VMSPLIT_2G is not set # CONFIG_VMSPLIT_1G is not set CONFIG_PAGE_OFFSET=0xC0000000 CONFIG_IPIPE=y CONFIG_IPIPE_DOMAINS=4 CONFIG_IPIPE_DELAYED_ATOMICSW=y # CONFIG_IPIPE_UNMASKED_CONTEXT_SWITCH is not set CONFIG_PREEMPT_NONE=y # CONFIG_PREEMPT_VOLUNTARY is not set # CONFIG_PREEMPT is not set CONFIG_HZ=100 # CONFIG_THUMB2_KERNEL is not set CONFIG_AEABI=y # CONFIG_OABI_COMPAT is not set CONFIG_ARCH_HAS_HOLES_MEMORYMODEL=y # CONFIG_ARCH_SPARSEMEM_DEFAULT is not set # CONFIG_ARCH_SELECT_MEMORY_MODEL is not set # CONFIG_HIGHMEM is not set CONFIG_HW_PERF_EVENTS=y CONFIG_SELECT_MEMORY_MODEL=y CONFIG_FLATMEM_MANUAL=y # CONFIG_DISCONTIGMEM_MANUAL is not set # CONFIG_SPARSEMEM_MANUAL is not set CONFIG_FLATMEM=y CONFIG_FLAT_NODE_MEM_MAP=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 # CONFIG_PHYS_ADDR_T_64BIT is not set CONFIG_ZONE_DMA_FLAG=0 CONFIG_VIRT_TO_BUS=y # CONFIG_KSM is not set CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_LEDS=y CONFIG_ALIGNMENT_TRAP=y # CONFIG_UACCESS_WITH_MEMCPY is not set # # Boot options # CONFIG_ZBOOT_ROM_TEXT=0x0 CONFIG_ZBOOT_ROM_BSS=0x0 CONFIG_CMDLINE=" debug " # CONFIG_XIP_KERNEL is not set CONFIG_KEXEC=y CONFIG_ATAGS_PROC=y # # CPU Power Management # # CONFIG_CPU_FREQ is not set # CONFIG_CPU_IDLE is not set # # Floating point emulation # # # At least one emulation must be selected # CONFIG_VFP=y CONFIG_VFPv3=y CONFIG_NEON=y # # Userspace binary formats # CONFIG_BINFMT_ELF=y CONFIG_HAVE_AOUT=y CONFIG_BINFMT_AOUT=m CONFIG_BINFMT_MISC=y # # Power management options # CONFIG_PM=y CONFIG_PM_DEBUG=y # CONFIG_PM_ADVANCED_DEBUG is not set # CONFIG_PM_VERBOSE is not set CONFIG_CAN_PM_TRACE=y CONFIG_PM_SLEEP=y CONFIG_SUSPEND=y # CONFIG_PM_TEST_SUSPEND is not set CONFIG_SUSPEND_FREEZER=y # CONFIG_APM_EMULATION is not set # CONFIG_PM_RUNTIME is not set CONFIG_PM_OPS=y CONFIG_ARCH_SUSPEND_POSSIBLE=y CONFIG_NET=y # # Networking options # CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_XFRM=y # CONFIG_XFRM_USER is not set # CONFIG_XFRM_SUB_POLICY is not set # CONFIG_XFRM_MIGRATE is not set # CONFIG_XFRM_STATISTICS is not set CONFIG_NET_KEY=y # CONFIG_NET_KEY_MIGRATE is not set CONFIG_INET=y # CONFIG_IP_MULTICAST is not set # CONFIG_IP_ADVANCED_ROUTER is not set CONFIG_IP_FIB_HASH=y CONFIG_IP_PNP=y CONFIG_IP_PNP_DHCP=y CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_RARP=y # CONFIG_NET_IPIP is not set # CONFIG_NET_IPGRE is not set # CONFIG_ARPD is not set # CONFIG_SYN_COOKIES is not set # CONFIG_INET_AH is not set # CONFIG_INET_ESP is not set # CONFIG_INET_IPCOMP is not set # CONFIG_INET_XFRM_TUNNEL is not set CONFIG_INET_TUNNEL=m CONFIG_INET_XFRM_MODE_TRANSPORT=y CONFIG_INET_XFRM_MODE_TUNNEL=y CONFIG_INET_XFRM_MODE_BEET=y # CONFIG_INET_LRO is not set CONFIG_INET_DIAG=y CONFIG_INET_TCP_DIAG=y # CONFIG_TCP_CONG_ADVANCED is not set CONFIG_TCP_CONG_CUBIC=y CONFIG_DEFAULT_TCP_CONG="cubic" # CONFIG_TCP_MD5SIG is not set CONFIG_IPV6=m # CONFIG_IPV6_PRIVACY is not set # CONFIG_IPV6_ROUTER_PREF is not set # CONFIG_IPV6_OPTIMISTIC_DAD is not set # CONFIG_INET6_AH is not set # CONFIG_INET6_ESP is not set # CONFIG_INET6_IPCOMP is not set # CONFIG_IPV6_MIP6 is not set # CONFIG_INET6_XFRM_TUNNEL is not set # CONFIG_INET6_TUNNEL is not set CONFIG_INET6_XFRM_MODE_TRANSPORT=m CONFIG_INET6_XFRM_MODE_TUNNEL=m CONFIG_INET6_XFRM_MODE_BEET=m # CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set CONFIG_IPV6_SIT=m # CONFIG_IPV6_SIT_6RD is not set CONFIG_IPV6_NDISC_NODETYPE=y # CONFIG_IPV6_TUNNEL is not set # CONFIG_IPV6_MULTIPLE_TABLES is not set # CONFIG_IPV6_MROUTE is not set # CONFIG_NETWORK_SECMARK is not set # CONFIG_NETFILTER is not set # CONFIG_IP_DCCP is not set # CONFIG_IP_SCTP is not set # CONFIG_RDS is not set # CONFIG_TIPC is not set # CONFIG_ATM is not set # CONFIG_BRIDGE is not set # CONFIG_NET_DSA is not set # CONFIG_VLAN_8021Q is not set # CONFIG_DECNET is not set # CONFIG_LLC2 is not set # CONFIG_IPX is not set # CONFIG_ATALK is not set # CONFIG_X25 is not set # CONFIG_LAPB is not set # CONFIG_ECONET is not set # CONFIG_WAN_ROUTER is not set # CONFIG_PHONET is not set # CONFIG_IEEE802154 is not set # CONFIG_NET_SCHED is not set # CONFIG_DCB is not set # # Network testing # # CONFIG_NET_PKTGEN is not set # CONFIG_NET_DROP_MONITOR is not set # CONFIG_HAMRADIO is not set # CONFIG_CAN is not set # CONFIG_IRDA is not set CONFIG_BT=y CONFIG_BT_L2CAP=y CONFIG_BT_SCO=y CONFIG_BT_RFCOMM=y CONFIG_BT_RFCOMM_TTY=y CONFIG_BT_BNEP=y CONFIG_BT_BNEP_MC_FILTER=y CONFIG_BT_BNEP_PROTO_FILTER=y CONFIG_BT_HIDP=y # # Bluetooth device drivers # CONFIG_BT_HCIBTUSB=m CONFIG_BT_HCIBTSDIO=m CONFIG_BT_HCIUART=y CONFIG_BT_HCIUART_H4=y CONFIG_BT_HCIUART_BCSP=y CONFIG_BT_HCIUART_LL=y CONFIG_BT_HCIBCM203X=m CONFIG_BT_HCIBPA10X=m CONFIG_BT_HCIBFUSB=m CONFIG_BT_HCIVHCI=m CONFIG_BT_MRVL=m CONFIG_BT_MRVL_SDIO=m CONFIG_BT_ATH3K=m # CONFIG_AF_RXRPC is not set CONFIG_WIRELESS=y CONFIG_WIRELESS_EXT=y CONFIG_WEXT_CORE=y CONFIG_WEXT_PROC=y CONFIG_WEXT_SPY=y CONFIG_WEXT_PRIV=y CONFIG_CFG80211=y # CONFIG_NL80211_TESTMODE is not set # CONFIG_CFG80211_DEVELOPER_WARNINGS is not set # CONFIG_CFG80211_REG_DEBUG is not set CONFIG_CFG80211_DEFAULT_PS=y # CONFIG_CFG80211_DEBUGFS is not set # CONFIG_CFG80211_INTERNAL_REGDB is not set CONFIG_CFG80211_WEXT=y CONFIG_WIRELESS_EXT_SYSFS=y CONFIG_LIB80211=m CONFIG_LIB80211_CRYPT_WEP=m CONFIG_LIB80211_CRYPT_CCMP=m CONFIG_LIB80211_CRYPT_TKIP=m # CONFIG_LIB80211_DEBUG is not set CONFIG_MAC80211=y CONFIG_MAC80211_RC_PID=y # CONFIG_MAC80211_RC_MINSTREL is not set CONFIG_MAC80211_RC_DEFAULT_PID=y # CONFIG_MAC80211_RC_DEFAULT_MINSTREL is not set CONFIG_MAC80211_RC_DEFAULT="pid" # CONFIG_MAC80211_MESH is not set CONFIG_MAC80211_LEDS=y # CONFIG_MAC80211_DEBUGFS is not set # CONFIG_MAC80211_DEBUG_MENU is not set # CONFIG_WIMAX is not set # CONFIG_RFKILL is not set # CONFIG_NET_9P is not set # # Device Drivers # # # Generic Driver Options # CONFIG_UEVENT_HELPER_PATH="" CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_STANDALONE=y CONFIG_PREVENT_FIRMWARE_BUILD=y CONFIG_FW_LOADER=y CONFIG_FIRMWARE_IN_KERNEL=y CONFIG_EXTRA_FIRMWARE="" # CONFIG_DEBUG_DRIVER is not set # CONFIG_DEBUG_DEVRES is not set # CONFIG_SYS_HYPERVISOR is not set # CONFIG_CONNECTOR is not set CONFIG_MTD=y # CONFIG_MTD_DEBUG is not set # CONFIG_MTD_TESTS is not set CONFIG_MTD_CONCAT=y CONFIG_MTD_PARTITIONS=y # CONFIG_MTD_REDBOOT_PARTS is not set # CONFIG_MTD_CMDLINE_PARTS is not set # CONFIG_MTD_AFS_PARTS is not set # CONFIG_MTD_AR7_PARTS is not set # # User Modules And Translation Layers # CONFIG_MTD_CHAR=y CONFIG_MTD_BLKDEVS=y CONFIG_MTD_BLOCK=y # CONFIG_FTL is not set # CONFIG_NFTL is not set # CONFIG_INFTL is not set # CONFIG_RFD_FTL is not set # CONFIG_SSFDC is not set # CONFIG_MTD_OOPS is not set # # RAM/ROM/Flash chip drivers # # CONFIG_MTD_CFI is not set # CONFIG_MTD_JEDECPROBE is not set CONFIG_MTD_MAP_BANK_WIDTH_1=y CONFIG_MTD_MAP_BANK_WIDTH_2=y CONFIG_MTD_MAP_BANK_WIDTH_4=y # CONFIG_MTD_MAP_BANK_WIDTH_8 is not set # CONFIG_MTD_MAP_BANK_WIDTH_16 is not set # CONFIG_MTD_MAP_BANK_WIDTH_32 is not set CONFIG_MTD_CFI_I1=y CONFIG_MTD_CFI_I2=y # CONFIG_MTD_CFI_I4 is not set # CONFIG_MTD_CFI_I8 is not set # CONFIG_MTD_RAM is not set # CONFIG_MTD_ROM is not set # CONFIG_MTD_ABSENT is not set # # Mapping drivers for chip access # # CONFIG_MTD_COMPLEX_MAPPINGS is not set # CONFIG_MTD_PLATRAM is not set # # Self-contained MTD device drivers # # CONFIG_MTD_DATAFLASH is not set # CONFIG_MTD_M25P80 is not set # CONFIG_MTD_SST25L is not set # CONFIG_MTD_SLRAM is not set # CONFIG_MTD_PHRAM is not set # CONFIG_MTD_MTDRAM is not set # CONFIG_MTD_BLOCK2MTD is not set # # Disk-On-Chip Device Drivers # # CONFIG_MTD_DOC2000 is not set # CONFIG_MTD_DOC2001 is not set # CONFIG_MTD_DOC2001PLUS is not set CONFIG_MTD_NAND=y # CONFIG_MTD_NAND_VERIFY_WRITE is not set # CONFIG_MTD_NAND_ECC_SMC is not set # CONFIG_MTD_NAND_MUSEUM_IDS is not set # CONFIG_MTD_NAND_GPIO is not set CONFIG_MTD_NAND_OMAP2=y CONFIG_MTD_NAND_OMAP_PREFETCH=y # CONFIG_MTD_NAND_OMAP_PREFETCH_DMA is not set CONFIG_MTD_NAND_IDS=y # CONFIG_MTD_NAND_DISKONCHIP is not set # CONFIG_MTD_NAND_NANDSIM is not set # CONFIG_MTD_NAND_PLATFORM is not set # CONFIG_MTD_ALAUDA is not set # CONFIG_MTD_ONENAND is not set # # LPDDR flash memory drivers # # CONFIG_MTD_LPDDR is not set # # UBI - Unsorted block images # CONFIG_MTD_UBI=y CONFIG_MTD_UBI_WL_THRESHOLD=4096 CONFIG_MTD_UBI_BEB_RESERVE=1 CONFIG_MTD_UBI_GLUEBI=y # # UBI debugging options # # CONFIG_MTD_UBI_DEBUG is not set # CONFIG_PARPORT is not set CONFIG_BLK_DEV=y # CONFIG_BLK_DEV_COW_COMMON is not set CONFIG_BLK_DEV_LOOP=y CONFIG_BLK_DEV_CRYPTOLOOP=m # # DRBD disabled because PROC_FS, INET or CONNECTOR not selected # # CONFIG_BLK_DEV_NBD is not set # CONFIG_BLK_DEV_UB is not set CONFIG_BLK_DEV_RAM=y CONFIG_BLK_DEV_RAM_COUNT=16 CONFIG_BLK_DEV_RAM_SIZE=16384 # CONFIG_BLK_DEV_XIP is not set CONFIG_CDROM_PKTCDVD=m CONFIG_CDROM_PKTCDVD_BUFFERS=8 # CONFIG_CDROM_PKTCDVD_WCACHE is not set # CONFIG_ATA_OVER_ETH is not set # CONFIG_MG_DISK is not set # CONFIG_MISC_DEVICES is not set CONFIG_EEPROM_93CX6=m CONFIG_HAVE_IDE=y # CONFIG_IDE is not set # # SCSI device support # CONFIG_RAID_ATTRS=m CONFIG_SCSI=y CONFIG_SCSI_DMA=y # CONFIG_SCSI_TGT is not set # CONFIG_SCSI_NETLINK is not set CONFIG_SCSI_PROC_FS=y # # SCSI support type (disk, tape, CD-ROM) # CONFIG_BLK_DEV_SD=y # CONFIG_CHR_DEV_ST is not set # CONFIG_CHR_DEV_OSST is not set CONFIG_BLK_DEV_SR=m # CONFIG_BLK_DEV_SR_VENDOR is not set CONFIG_CHR_DEV_SG=m # CONFIG_CHR_DEV_SCH is not set CONFIG_SCSI_MULTI_LUN=y # CONFIG_SCSI_CONSTANTS is not set # CONFIG_SCSI_LOGGING is not set # CONFIG_SCSI_SCAN_ASYNC is not set CONFIG_SCSI_WAIT_SCAN=m # # SCSI Transports # # CONFIG_SCSI_SPI_ATTRS is not set # CONFIG_SCSI_FC_ATTRS is not set # CONFIG_SCSI_ISCSI_ATTRS is not set # CONFIG_SCSI_SAS_ATTRS is not set # CONFIG_SCSI_SAS_LIBSAS is not set # CONFIG_SCSI_SRP_ATTRS is not set CONFIG_SCSI_LOWLEVEL=y # CONFIG_ISCSI_TCP is not set # CONFIG_LIBFC is not set # CONFIG_LIBFCOE is not set # CONFIG_SCSI_DEBUG is not set # CONFIG_SCSI_DH is not set # CONFIG_SCSI_OSD_INITIATOR is not set # CONFIG_ATA is not set CONFIG_MD=y CONFIG_BLK_DEV_MD=m CONFIG_MD_LINEAR=m CONFIG_MD_RAID0=m CONFIG_MD_RAID1=m CONFIG_MD_RAID10=m CONFIG_MD_RAID456=m CONFIG_MD_RAID6_PQ=m # CONFIG_ASYNC_RAID6_TEST is not set CONFIG_MD_MULTIPATH=m CONFIG_MD_FAULTY=m CONFIG_BLK_DEV_DM=m # CONFIG_DM_DEBUG is not set CONFIG_DM_CRYPT=m CONFIG_DM_SNAPSHOT=m CONFIG_DM_MIRROR=m # CONFIG_DM_LOG_USERSPACE is not set CONFIG_DM_ZERO=m CONFIG_DM_MULTIPATH=m # CONFIG_DM_MULTIPATH_QL is not set # CONFIG_DM_MULTIPATH_ST is not set CONFIG_DM_DELAY=m # CONFIG_DM_UEVENT is not set CONFIG_NETDEVICES=y CONFIG_DUMMY=m # CONFIG_BONDING is not set # CONFIG_MACVLAN is not set # CONFIG_EQUALIZER is not set CONFIG_TUN=m # CONFIG_VETH is not set CONFIG_PHYLIB=y # # MII PHY device drivers # # CONFIG_MARVELL_PHY is not set # CONFIG_DAVICOM_PHY is not set # CONFIG_QSEMI_PHY is not set # CONFIG_LXT_PHY is not set # CONFIG_CICADA_PHY is not set # CONFIG_VITESSE_PHY is not set # CONFIG_SMSC_PHY is not set # CONFIG_BROADCOM_PHY is not set # CONFIG_ICPLUS_PHY is not set # CONFIG_REALTEK_PHY is not set # CONFIG_NATIONAL_PHY is not set # CONFIG_STE10XP is not set # CONFIG_LSI_ET1011C_PHY is not set # CONFIG_FIXED_PHY is not set # CONFIG_MDIO_BITBANG is not set CONFIG_NET_ETHERNET=y CONFIG_MII=y # CONFIG_AX88796 is not set # CONFIG_SMC91X is not set # CONFIG_TI_DAVINCI_EMAC is not set # CONFIG_DM9000 is not set CONFIG_ENC28J60=m # CONFIG_ENC28J60_WRITEVERIFY is not set # CONFIG_ETHOC is not set # CONFIG_SMC911X is not set CONFIG_SMSC911X=y # CONFIG_DNET is not set # CONFIG_IBM_NEW_EMAC_ZMII is not set # CONFIG_IBM_NEW_EMAC_RGMII is not set # CONFIG_IBM_NEW_EMAC_TAH is not set # CONFIG_IBM_NEW_EMAC_EMAC4 is not set # CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set # CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set # CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set # CONFIG_B44 is not set # CONFIG_KS8842 is not set # CONFIG_KS8851 is not set # CONFIG_KS8851_MLL is not set # CONFIG_NETDEV_1000 is not set # CONFIG_NETDEV_10000 is not set CONFIG_WLAN=y # CONFIG_LIBERTAS_THINFIRM is not set # CONFIG_AT76C50X_USB is not set CONFIG_USB_ZD1201=m CONFIG_USB_NET_RNDIS_WLAN=m CONFIG_RTL8187=m CONFIG_RTL8187_LEDS=y # CONFIG_MAC80211_HWSIM is not set # CONFIG_ATH_COMMON is not set # CONFIG_B43 is not set # CONFIG_B43LEGACY is not set CONFIG_HOSTAP=m CONFIG_HOSTAP_FIRMWARE=y CONFIG_HOSTAP_FIRMWARE_NVRAM=y # CONFIG_IWM is not set CONFIG_LIBERTAS=m CONFIG_LIBERTAS_USB=m CONFIG_LIBERTAS_SDIO=m # CONFIG_LIBERTAS_SPI is not set CONFIG_LIBERTAS_DEBUG=y # CONFIG_LIBERTAS_MESH is not set CONFIG_P54_COMMON=m CONFIG_P54_USB=m # CONFIG_P54_SPI is not set CONFIG_P54_LEDS=y CONFIG_RT2X00=m CONFIG_RT2500USB=m CONFIG_RT73USB=m CONFIG_RT2800USB=m # CONFIG_RT2800USB_RT30XX is not set # CONFIG_RT2800USB_RT35XX is not set # CONFIG_RT2800USB_UNKNOWN is not set CONFIG_RT2800_LIB=m CONFIG_RT2X00_LIB_USB=m CONFIG_RT2X00_LIB=m CONFIG_RT2X00_LIB_HT=y CONFIG_RT2X00_LIB_FIRMWARE=y CONFIG_RT2X00_LIB_CRYPTO=y CONFIG_RT2X00_LIB_LEDS=y # CONFIG_RT2X00_DEBUG is not set CONFIG_WL12XX=m # CONFIG_WL1251 is not set CONFIG_WL1271=m CONFIG_ZD1211RW=m # CONFIG_ZD1211RW_DEBUG is not set # # Enable WiMAX (Networking options) to see the WiMAX drivers # # # USB Network Adapters # CONFIG_USB_CATC=m CONFIG_USB_KAWETH=m CONFIG_USB_PEGASUS=m CONFIG_USB_RTL8150=m CONFIG_USB_USBNET=y CONFIG_USB_NET_AX8817X=y CONFIG_USB_NET_CDCETHER=y CONFIG_USB_NET_CDC_EEM=m CONFIG_USB_NET_DM9601=m CONFIG_USB_NET_SMSC95XX=m CONFIG_USB_NET_GL620A=m CONFIG_USB_NET_NET1080=m CONFIG_USB_NET_PLUSB=m CONFIG_USB_NET_MCS7830=m CONFIG_USB_NET_RNDIS_HOST=m CONFIG_USB_NET_CDC_SUBSET=m CONFIG_USB_ALI_M5632=y CONFIG_USB_AN2720=y CONFIG_USB_BELKIN=y CONFIG_USB_ARMLINUX=y CONFIG_USB_EPSON2888=y CONFIG_USB_KC2190=y CONFIG_USB_NET_ZAURUS=m CONFIG_USB_NET_INT51X1=m # CONFIG_WAN is not set CONFIG_PPP=m # CONFIG_PPP_MULTILINK is not set # CONFIG_PPP_FILTER is not set CONFIG_PPP_ASYNC=m CONFIG_PPP_SYNC_TTY=m CONFIG_PPP_DEFLATE=m CONFIG_PPP_BSDCOMP=m CONFIG_PPP_MPPE=m CONFIG_PPPOE=m # CONFIG_PPPOL2TP is not set # CONFIG_SLIP is not set CONFIG_SLHC=m # CONFIG_NETCONSOLE is not set # CONFIG_NETPOLL is not set # CONFIG_NET_POLL_CONTROLLER is not set # CONFIG_ISDN is not set # CONFIG_PHONE is not set # # Input device support # CONFIG_INPUT=y # CONFIG_INPUT_FF_MEMLESS is not set CONFIG_INPUT_POLLDEV=m # CONFIG_INPUT_SPARSEKMAP is not set # # Userland interfaces # CONFIG_INPUT_MOUSEDEV=y CONFIG_INPUT_MOUSEDEV_PSAUX=y CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024 CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768 # CONFIG_INPUT_JOYDEV is not set CONFIG_INPUT_EVDEV=y # CONFIG_INPUT_EVBUG is not set # # Input Device Drivers # CONFIG_INPUT_KEYBOARD=y # CONFIG_KEYBOARD_ADP5588 is not set # CONFIG_KEYBOARD_ATKBD is not set # CONFIG_QT2160 is not set # CONFIG_KEYBOARD_LKKBD is not set # CONFIG_KEYBOARD_GPIO is not set # CONFIG_KEYBOARD_MATRIX is not set # CONFIG_KEYBOARD_LM8323 is not set # CONFIG_KEYBOARD_MAX7359 is not set # CONFIG_KEYBOARD_NEWTON is not set # CONFIG_KEYBOARD_OPENCORES is not set # CONFIG_KEYBOARD_STOWAWAY is not set # CONFIG_KEYBOARD_SUNKBD is not set # CONFIG_KEYBOARD_TWL4030 is not set # CONFIG_KEYBOARD_XTKBD is not set CONFIG_INPUT_MOUSE=y CONFIG_MOUSE_PS2=y CONFIG_MOUSE_PS2_ALPS=y CONFIG_MOUSE_PS2_LOGIPS2PP=y CONFIG_MOUSE_PS2_SYNAPTICS=y CONFIG_MOUSE_PS2_TRACKPOINT=y # CONFIG_MOUSE_PS2_ELANTECH is not set # CONFIG_MOUSE_PS2_SENTELIC is not set # CONFIG_MOUSE_PS2_TOUCHKIT is not set # CONFIG_MOUSE_SERIAL is not set # CONFIG_MOUSE_APPLETOUCH is not set # CONFIG_MOUSE_BCM5974 is not set # CONFIG_MOUSE_VSXXXAA is not set # CONFIG_MOUSE_GPIO is not set # CONFIG_MOUSE_SYNAPTICS_I2C is not set # CONFIG_INPUT_JOYSTICK is not set # CONFIG_INPUT_TABLET is not set CONFIG_INPUT_TOUCHSCREEN=y CONFIG_TOUCHSCREEN_ADS7846=m # CONFIG_TOUCHSCREEN_AD7877 is not set # CONFIG_TOUCHSCREEN_AD7879_I2C is not set # CONFIG_TOUCHSCREEN_AD7879_SPI is not set # CONFIG_TOUCHSCREEN_AD7879 is not set # CONFIG_TOUCHSCREEN_DYNAPRO is not set # CONFIG_TOUCHSCREEN_EETI is not set # CONFIG_TOUCHSCREEN_FUJITSU is not set # CONFIG_TOUCHSCREEN_GUNZE is not set # CONFIG_TOUCHSCREEN_ELO is not set # CONFIG_TOUCHSCREEN_WACOM_W8001 is not set # CONFIG_TOUCHSCREEN_MCS5000 is not set # CONFIG_TOUCHSCREEN_MTOUCH is not set # CONFIG_TOUCHSCREEN_INEXIO is not set # CONFIG_TOUCHSCREEN_MK712 is not set # CONFIG_TOUCHSCREEN_PENMOUNT is not set # CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set # CONFIG_TOUCHSCREEN_TOUCHWIN is not set # CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set # CONFIG_TOUCHSCREEN_TOUCHIT213 is not set # CONFIG_TOUCHSCREEN_TSC2007 is not set # CONFIG_TOUCHSCREEN_W90X900 is not set # CONFIG_INPUT_MISC is not set # # Hardware I/O ports # CONFIG_SERIO=y CONFIG_SERIO_SERPORT=y CONFIG_SERIO_LIBPS2=y # CONFIG_SERIO_RAW is not set # CONFIG_SERIO_ALTERA_PS2 is not set # CONFIG_GAMEPORT is not set # # Character devices # CONFIG_VT=y CONFIG_CONSOLE_TRANSLATIONS=y CONFIG_VT_CONSOLE=y CONFIG_HW_CONSOLE=y CONFIG_VT_HW_CONSOLE_BINDING=y CONFIG_DEVKMEM=y # CONFIG_SERIAL_NONSTANDARD is not set # # Serial drivers # CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_8250_NR_UARTS=32 CONFIG_SERIAL_8250_RUNTIME_UARTS=4 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_MANY_PORTS=y CONFIG_SERIAL_8250_SHARE_IRQ=y CONFIG_SERIAL_8250_DETECT_IRQ=y CONFIG_SERIAL_8250_RSA=y # # Non-8250 serial port support # # CONFIG_SERIAL_MAX3100 is not set CONFIG_SERIAL_CORE=y CONFIG_SERIAL_CORE_CONSOLE=y CONFIG_UNIX98_PTYS=y # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set # CONFIG_LEGACY_PTYS is not set # CONFIG_IPMI_HANDLER is not set CONFIG_HW_RANDOM=y # CONFIG_HW_RANDOM_TIMERIOMEM is not set # CONFIG_R3964 is not set # CONFIG_RAW_DRIVER is not set # CONFIG_TCG_TPM is not set CONFIG_I2C=y CONFIG_I2C_BOARDINFO=y CONFIG_I2C_COMPAT=y CONFIG_I2C_CHARDEV=y CONFIG_I2C_HELPER_AUTO=y # # I2C Hardware Bus support # # # I2C system bus drivers (mostly embedded / system-on-chip) # # CONFIG_I2C_DESIGNWARE is not set # CONFIG_I2C_GPIO is not set # CONFIG_I2C_OCORES is not set CONFIG_I2C_OMAP=y # CONFIG_I2C_SIMTEC is not set # # External I2C/SMBus adapter drivers # # CONFIG_I2C_PARPORT_LIGHT is not set # CONFIG_I2C_TAOS_EVM is not set # CONFIG_I2C_TINY_USB is not set # # Other I2C/SMBus bus drivers # # CONFIG_I2C_PCA_PLATFORM is not set # CONFIG_I2C_STUB is not set # # Miscellaneous I2C Chip support # # CONFIG_SENSORS_TSL2550 is not set # CONFIG_I2C_DEBUG_CORE is not set # CONFIG_I2C_DEBUG_ALGO is not set # CONFIG_I2C_DEBUG_BUS is not set # CONFIG_I2C_DEBUG_CHIP is not set CONFIG_SPI=y # CONFIG_SPI_DEBUG is not set CONFIG_SPI_MASTER=y # # SPI Master Controller Drivers # # CONFIG_SPI_BITBANG is not set # CONFIG_SPI_GPIO is not set CONFIG_SPI_OMAP24XX=y # CONFIG_SPI_XILINX is not set # CONFIG_SPI_DESIGNWARE is not set # # SPI Protocol Masters # CONFIG_SPI_SPIDEV=y # CONFIG_SPI_TLE62X0 is not set # # PPS support # # CONFIG_PPS is not set CONFIG_ARCH_REQUIRE_GPIOLIB=y CONFIG_GPIOLIB=y CONFIG_DEBUG_GPIO=y CONFIG_GPIO_SYSFS=y # # Memory mapped GPIO expanders: # # # I2C GPIO expanders: # CONFIG_GPIO_MAX732X=m CONFIG_GPIO_PCA953X=m CONFIG_GPIO_PCF857X=m CONFIG_GPIO_TWL4030=y CONFIG_GPIO_ADP5588=m # # PCI GPIO expanders: # # # SPI GPIO expanders: # CONFIG_GPIO_MAX7301=m CONFIG_GPIO_MCP23S08=m # CONFIG_GPIO_MC33880 is not set # # AC97 GPIO expanders: # # CONFIG_W1 is not set CONFIG_POWER_SUPPLY=m # CONFIG_POWER_SUPPLY_DEBUG is not set # CONFIG_PDA_POWER is not set # CONFIG_BATTERY_DS2760 is not set # CONFIG_BATTERY_DS2782 is not set # CONFIG_BATTERY_BQ27x00 is not set # CONFIG_BATTERY_MAX17040 is not set CONFIG_HWMON=y CONFIG_HWMON_VID=m # CONFIG_HWMON_DEBUG_CHIP is not set # # Native drivers # CONFIG_SENSORS_AD7414=m CONFIG_SENSORS_AD7418=m CONFIG_SENSORS_ADCXX=m CONFIG_SENSORS_ADM1021=m CONFIG_SENSORS_ADM1025=m CONFIG_SENSORS_ADM1026=m CONFIG_SENSORS_ADM1029=m CONFIG_SENSORS_ADM1031=m CONFIG_SENSORS_ADM9240=m CONFIG_SENSORS_ADT7462=m CONFIG_SENSORS_ADT7470=m CONFIG_SENSORS_ADT7473=m CONFIG_SENSORS_ADT7475=m CONFIG_SENSORS_ATXP1=m CONFIG_SENSORS_DS1621=m CONFIG_SENSORS_F71805F=m CONFIG_SENSORS_F71882FG=m CONFIG_SENSORS_F75375S=m CONFIG_SENSORS_G760A=m CONFIG_SENSORS_GL518SM=m CONFIG_SENSORS_GL520SM=m CONFIG_SENSORS_IT87=m CONFIG_SENSORS_LM63=m CONFIG_SENSORS_LM70=m CONFIG_SENSORS_LM73=m CONFIG_SENSORS_LM75=m CONFIG_SENSORS_LM77=m CONFIG_SENSORS_LM78=m CONFIG_SENSORS_LM80=m CONFIG_SENSORS_LM83=m CONFIG_SENSORS_LM85=m CONFIG_SENSORS_LM87=m CONFIG_SENSORS_LM90=m CONFIG_SENSORS_LM92=m CONFIG_SENSORS_LM93=m CONFIG_SENSORS_LTC4215=m CONFIG_SENSORS_LTC4245=m CONFIG_SENSORS_LM95241=m CONFIG_SENSORS_MAX1111=m CONFIG_SENSORS_MAX1619=m CONFIG_SENSORS_MAX6650=m CONFIG_SENSORS_PC87360=m CONFIG_SENSORS_PC87427=m CONFIG_SENSORS_PCF8591=m CONFIG_SENSORS_SHT15=m CONFIG_SENSORS_DME1737=m CONFIG_SENSORS_SMSC47M1=m CONFIG_SENSORS_SMSC47M192=m CONFIG_SENSORS_SMSC47B397=m CONFIG_SENSORS_ADS7828=m CONFIG_SENSORS_AMC6821=m CONFIG_SENSORS_THMC50=m CONFIG_SENSORS_TMP401=m CONFIG_SENSORS_TMP421=m CONFIG_SENSORS_VT1211=m CONFIG_SENSORS_W83781D=m CONFIG_SENSORS_W83791D=m CONFIG_SENSORS_W83792D=m CONFIG_SENSORS_W83793=m CONFIG_SENSORS_W83L785TS=m CONFIG_SENSORS_W83L786NG=m CONFIG_SENSORS_W83627HF=m CONFIG_SENSORS_W83627EHF=m CONFIG_SENSORS_LIS3_SPI=m CONFIG_SENSORS_LIS3_I2C=m # CONFIG_THERMAL is not set CONFIG_WATCHDOG=y CONFIG_WATCHDOG_NOWAYOUT=y # # Watchdog Device Drivers # # CONFIG_SOFT_WATCHDOG is not set CONFIG_OMAP_WATCHDOG=y CONFIG_TWL4030_WATCHDOG=y # # USB-based Watchdog Cards # # CONFIG_USBPCWATCHDOG is not set CONFIG_SSB_POSSIBLE=y # # Sonics Silicon Backplane # # CONFIG_SSB is not set # # Multifunction device drivers # CONFIG_MFD_CORE=y # CONFIG_MFD_SM501 is not set # CONFIG_MFD_ASIC3 is not set # CONFIG_HTC_EGPIO is not set # CONFIG_HTC_PASIC3 is not set # CONFIG_TPS65010 is not set CONFIG_TWL4030_CORE=y CONFIG_TWL4030_POWER=y CONFIG_TWL4030_CODEC=y CONFIG_TWL4030_MADC=y # CONFIG_MFD_TMIO is not set # CONFIG_MFD_T7L66XB is not set # CONFIG_MFD_TC6387XB is not set # CONFIG_MFD_TC6393XB is not set # CONFIG_PMIC_DA903X is not set # CONFIG_PMIC_ADP5520 is not set # CONFIG_MFD_WM8400 is not set # CONFIG_MFD_WM831X is not set # CONFIG_MFD_WM8350_I2C is not set # CONFIG_MFD_PCF50633 is not set # CONFIG_MFD_MC13783 is not set # CONFIG_AB3100_CORE is not set # CONFIG_EZX_PCAP is not set # CONFIG_MFD_88PM8607 is not set # CONFIG_AB4500_CORE is not set CONFIG_REGULATOR=y CONFIG_REGULATOR_DEBUG=y # CONFIG_REGULATOR_FIXED_VOLTAGE is not set # CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set # CONFIG_REGULATOR_USERSPACE_CONSUMER is not set # CONFIG_REGULATOR_BQ24022 is not set # CONFIG_REGULATOR_MAX1586 is not set # CONFIG_REGULATOR_MAX8660 is not set CONFIG_REGULATOR_TWL4030=y # CONFIG_REGULATOR_LP3971 is not set # CONFIG_REGULATOR_TPS65023 is not set # CONFIG_REGULATOR_TPS6507X is not set CONFIG_MEDIA_SUPPORT=m # # Multimedia core support # CONFIG_VIDEO_DEV=m CONFIG_VIDEO_V4L2_COMMON=m CONFIG_VIDEO_ALLOW_V4L1=y CONFIG_VIDEO_V4L1_COMPAT=y CONFIG_DVB_CORE=m CONFIG_VIDEO_MEDIA=m # # Multimedia drivers # CONFIG_IR_CORE=m CONFIG_VIDEO_IR=m CONFIG_MEDIA_ATTACH=y CONFIG_MEDIA_TUNER=m # CONFIG_MEDIA_TUNER_CUSTOMISE is not set CONFIG_MEDIA_TUNER_SIMPLE=m CONFIG_MEDIA_TUNER_TDA8290=m CONFIG_MEDIA_TUNER_TDA827X=m CONFIG_MEDIA_TUNER_TDA18271=m CONFIG_MEDIA_TUNER_TDA9887=m CONFIG_MEDIA_TUNER_TEA5761=m CONFIG_MEDIA_TUNER_TEA5767=m CONFIG_MEDIA_TUNER_MT20XX=m CONFIG_MEDIA_TUNER_MT2060=m CONFIG_MEDIA_TUNER_MT2266=m CONFIG_MEDIA_TUNER_QT1010=m CONFIG_MEDIA_TUNER_XC2028=m CONFIG_MEDIA_TUNER_XC5000=m CONFIG_MEDIA_TUNER_MXL5005S=m CONFIG_MEDIA_TUNER_MXL5007T=m CONFIG_MEDIA_TUNER_MC44S803=m CONFIG_MEDIA_TUNER_MAX2165=m CONFIG_VIDEO_V4L2=m CONFIG_VIDEO_V4L1=m CONFIG_VIDEOBUF_GEN=m CONFIG_VIDEOBUF_VMALLOC=m CONFIG_VIDEOBUF_DVB=m CONFIG_VIDEO_TVEEPROM=m CONFIG_VIDEO_TUNER=m CONFIG_VIDEO_CAPTURE_DRIVERS=y # CONFIG_VIDEO_ADV_DEBUG is not set # CONFIG_VIDEO_FIXED_MINOR_RANGES is not set CONFIG_VIDEO_HELPER_CHIPS_AUTO=y CONFIG_VIDEO_IR_I2C=m CONFIG_VIDEO_MSP3400=m CONFIG_VIDEO_CS53L32A=m CONFIG_VIDEO_WM8775=m CONFIG_VIDEO_MT9V011=m CONFIG_VIDEO_SAA711X=m CONFIG_VIDEO_TVP5150=m CONFIG_VIDEO_CX25840=m CONFIG_VIDEO_CX2341X=m # CONFIG_VIDEO_VIVI is not set # CONFIG_VIDEO_CPIA is not set # CONFIG_VIDEO_CPIA2 is not set # CONFIG_VIDEO_SAA5246A is not set # CONFIG_VIDEO_SAA5249 is not set # CONFIG_VIDEO_AU0828 is not set # CONFIG_SOC_CAMERA is not set CONFIG_V4L_USB_DRIVERS=y CONFIG_USB_VIDEO_CLASS=m CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y CONFIG_USB_GSPCA=m CONFIG_USB_M5602=m CONFIG_USB_STV06XX=m # CONFIG_USB_GL860 is not set # CONFIG_USB_GSPCA_BENQ is not set CONFIG_USB_GSPCA_CONEX=m # CONFIG_USB_GSPCA_CPIA1 is not set CONFIG_USB_GSPCA_ETOMS=m CONFIG_USB_GSPCA_FINEPIX=m # CONFIG_USB_GSPCA_JEILINJ is not set CONFIG_USB_GSPCA_MARS=m # CONFIG_USB_GSPCA_MR97310A is not set CONFIG_USB_GSPCA_OV519=m CONFIG_USB_GSPCA_OV534=m # CONFIG_USB_GSPCA_OV534_9 is not set CONFIG_USB_GSPCA_PAC207=m # CONFIG_USB_GSPCA_PAC7302 is not set CONFIG_USB_GSPCA_PAC7311=m # CONFIG_USB_GSPCA_SN9C2028 is not set # CONFIG_USB_GSPCA_SN9C20X is not set CONFIG_USB_GSPCA_SONIXB=m CONFIG_USB_GSPCA_SONIXJ=m CONFIG_USB_GSPCA_SPCA500=m CONFIG_USB_GSPCA_SPCA501=m CONFIG_USB_GSPCA_SPCA505=m CONFIG_USB_GSPCA_SPCA506=m CONFIG_USB_GSPCA_SPCA508=m CONFIG_USB_GSPCA_SPCA561=m # CONFIG_USB_GSPCA_SQ905 is not set # CONFIG_USB_GSPCA_SQ905C is not set CONFIG_USB_GSPCA_STK014=m # CONFIG_USB_GSPCA_STV0680 is not set CONFIG_USB_GSPCA_SUNPLUS=m CONFIG_USB_GSPCA_T613=m CONFIG_USB_GSPCA_TV8532=m CONFIG_USB_GSPCA_VC032X=m CONFIG_USB_GSPCA_ZC3XX=m CONFIG_VIDEO_PVRUSB2=m CONFIG_VIDEO_PVRUSB2_SYSFS=y CONFIG_VIDEO_PVRUSB2_DVB=y # CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set CONFIG_VIDEO_HDPVR=m CONFIG_VIDEO_EM28XX=m CONFIG_VIDEO_EM28XX_ALSA=m CONFIG_VIDEO_EM28XX_DVB=m # CONFIG_VIDEO_TLG2300 is not set CONFIG_VIDEO_CX231XX=m CONFIG_VIDEO_CX231XX_ALSA=m CONFIG_VIDEO_CX231XX_DVB=m CONFIG_VIDEO_USBVISION=m CONFIG_VIDEO_USBVIDEO=m CONFIG_USB_VICAM=m CONFIG_USB_IBMCAM=m CONFIG_USB_KONICAWC=m CONFIG_USB_QUICKCAM_MESSENGER=m CONFIG_USB_ET61X251=m CONFIG_VIDEO_OVCAMCHIP=m CONFIG_USB_W9968CF=m CONFIG_USB_OV511=m CONFIG_USB_SE401=m CONFIG_USB_SN9C102=m CONFIG_USB_STV680=m CONFIG_USB_ZC0301=m CONFIG_USB_PWC=m # CONFIG_USB_PWC_DEBUG is not set CONFIG_USB_PWC_INPUT_EVDEV=y CONFIG_USB_ZR364XX=m CONFIG_USB_STKWEBCAM=m CONFIG_USB_S2255=m CONFIG_RADIO_ADAPTERS=y # CONFIG_I2C_SI4713 is not set # CONFIG_RADIO_SI4713 is not set # CONFIG_USB_DSBR is not set # CONFIG_RADIO_SI470X is not set # CONFIG_USB_MR800 is not set # CONFIG_RADIO_TEA5764 is not set # CONFIG_RADIO_SAA7706H is not set # CONFIG_RADIO_TEF6862 is not set CONFIG_DVB_MAX_ADAPTERS=8 # CONFIG_DVB_DYNAMIC_MINORS is not set CONFIG_DVB_CAPTURE_DRIVERS=y # CONFIG_TTPCI_EEPROM is not set # # Supported USB Adapters # CONFIG_DVB_USB=m # CONFIG_DVB_USB_DEBUG is not set CONFIG_DVB_USB_A800=m CONFIG_DVB_USB_DIBUSB_MB=m # CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set CONFIG_DVB_USB_DIBUSB_MC=m CONFIG_DVB_USB_DIB0700=m CONFIG_DVB_USB_UMT_010=m CONFIG_DVB_USB_CXUSB=m CONFIG_DVB_USB_M920X=m CONFIG_DVB_USB_GL861=m CONFIG_DVB_USB_AU6610=m CONFIG_DVB_USB_DIGITV=m CONFIG_DVB_USB_VP7045=m CONFIG_DVB_USB_VP702X=m CONFIG_DVB_USB_GP8PSK=m CONFIG_DVB_USB_NOVA_T_USB2=m CONFIG_DVB_USB_TTUSB2=m CONFIG_DVB_USB_DTT200U=m CONFIG_DVB_USB_OPERA1=m CONFIG_DVB_USB_AF9005=m CONFIG_DVB_USB_AF9005_REMOTE=m CONFIG_DVB_USB_DW2102=m CONFIG_DVB_USB_CINERGY_T2=m CONFIG_DVB_USB_ANYSEE=m CONFIG_DVB_USB_DTV5100=m CONFIG_DVB_USB_AF9015=m CONFIG_DVB_USB_CE6230=m CONFIG_DVB_USB_FRIIO=m CONFIG_DVB_USB_EC168=m # CONFIG_DVB_USB_AZ6027 is not set CONFIG_SMS_SIANO_MDTV=m # # Siano module components # # CONFIG_SMS_USB_DRV is not set # CONFIG_SMS_SDIO_DRV is not set # # Supported FlexCopII (B2C2) Adapters # CONFIG_DVB_B2C2_FLEXCOP=m CONFIG_DVB_B2C2_FLEXCOP_USB=m # CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set # # Supported DVB Frontends # # CONFIG_DVB_FE_CUSTOMISE is not set CONFIG_DVB_STB6100=m CONFIG_DVB_CX24123=m CONFIG_DVB_MT312=m CONFIG_DVB_ZL10039=m CONFIG_DVB_S5H1420=m CONFIG_DVB_STV0288=m CONFIG_DVB_STB6000=m CONFIG_DVB_STV0299=m CONFIG_DVB_STV6110=m CONFIG_DVB_STV0900=m CONFIG_DVB_TDA10086=m CONFIG_DVB_TUNER_ITD1000=m CONFIG_DVB_TUNER_CX24113=m CONFIG_DVB_TDA826X=m CONFIG_DVB_CX24116=m CONFIG_DVB_SI21XX=m CONFIG_DVB_DS3000=m CONFIG_DVB_CX22702=m CONFIG_DVB_TDA1004X=m CONFIG_DVB_NXT6000=m CONFIG_DVB_MT352=m CONFIG_DVB_ZL10353=m CONFIG_DVB_DIB3000MB=m CONFIG_DVB_DIB3000MC=m CONFIG_DVB_DIB7000M=m CONFIG_DVB_DIB7000P=m CONFIG_DVB_TDA10048=m CONFIG_DVB_AF9013=m CONFIG_DVB_EC100=m CONFIG_DVB_TDA10021=m CONFIG_DVB_TDA10023=m CONFIG_DVB_STV0297=m CONFIG_DVB_NXT200X=m CONFIG_DVB_BCM3510=m CONFIG_DVB_LGDT330X=m CONFIG_DVB_LGDT3305=m CONFIG_DVB_S5H1409=m CONFIG_DVB_S5H1411=m CONFIG_DVB_DIB8000=m CONFIG_DVB_PLL=m CONFIG_DVB_TUNER_DIB0070=m CONFIG_DVB_LNBP21=m CONFIG_DVB_ISL6421=m CONFIG_DVB_LGS8GXX=m CONFIG_DVB_ATBM8830=m # CONFIG_DAB is not set # # Graphics support # # CONFIG_VGASTATE is not set # CONFIG_VIDEO_OUTPUT_CONTROL is not set CONFIG_FB=y # CONFIG_FIRMWARE_EDID is not set # CONFIG_FB_DDC is not set # CONFIG_FB_BOOT_VESA_SUPPORT is not set CONFIG_FB_CFB_FILLRECT=y CONFIG_FB_CFB_COPYAREA=y CONFIG_FB_CFB_IMAGEBLIT=y # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set # CONFIG_FB_SYS_FILLRECT is not set # CONFIG_FB_SYS_COPYAREA is not set # CONFIG_FB_SYS_IMAGEBLIT is not set # CONFIG_FB_FOREIGN_ENDIAN is not set # CONFIG_FB_SYS_FOPS is not set # CONFIG_FB_SVGALIB is not set # CONFIG_FB_MACMODES is not set # CONFIG_FB_BACKLIGHT is not set # CONFIG_FB_MODE_HELPERS is not set # CONFIG_FB_TILEBLITTING is not set # # Frame buffer hardware drivers # # CONFIG_FB_S1D13XXX is not set # CONFIG_FB_TMIO is not set # CONFIG_FB_VIRTUAL is not set # CONFIG_FB_METRONOME is not set # CONFIG_FB_MB862XX is not set # CONFIG_FB_BROADSHEET is not set # CONFIG_FB_OMAP_BOOTLOADER_INIT is not set CONFIG_OMAP2_VRAM=y CONFIG_OMAP2_VRFB=y CONFIG_OMAP2_DSS=y CONFIG_OMAP2_VRAM_SIZE=0 CONFIG_OMAP2_DSS_DEBUG_SUPPORT=y # CONFIG_OMAP2_DSS_COLLECT_IRQ_STATS is not set # CONFIG_OMAP2_DSS_RFBI is not set CONFIG_OMAP2_DSS_VENC=y # CONFIG_OMAP2_DSS_SDI is not set CONFIG_OMAP2_DSS_DSI=y CONFIG_OMAP2_DSS_USE_DSI_PLL=y # CONFIG_OMAP2_DSS_FAKE_VSYNC is not set CONFIG_OMAP2_DSS_MIN_FCK_PER_PCK=0 CONFIG_FB_OMAP2=y CONFIG_FB_OMAP2_DEBUG_SUPPORT=y CONFIG_FB_OMAP2_NUM_FBS=3 # # OMAP2/3 Display Device Drivers # CONFIG_PANEL_GENERIC=y CONFIG_PANEL_LGPHILIPS_LB035Q02=y CONFIG_PANEL_SAMSUNG_LTE430WQ_F0C=y CONFIG_PANEL_SHARP_LS037V7DW01=y # CONFIG_PANEL_SHARP_LQ043T1DG01 is not set # CONFIG_PANEL_TAAL is not set # CONFIG_PANEL_TOPPOLY_TDO35S is not set # CONFIG_PANEL_TPO_TD043MTEA1 is not set CONFIG_BACKLIGHT_LCD_SUPPORT=y CONFIG_LCD_CLASS_DEVICE=m # CONFIG_LCD_LMS283GF05 is not set # CONFIG_LCD_LTV350QV is not set # CONFIG_LCD_ILI9320 is not set # CONFIG_LCD_TDO24M is not set # CONFIG_LCD_VGG2432A4 is not set # CONFIG_LCD_PLATFORM is not set CONFIG_BACKLIGHT_CLASS_DEVICE=m CONFIG_BACKLIGHT_GENERIC=m # # Display device support # CONFIG_DISPLAY_SUPPORT=y # # Display hardware drivers # # # Console display driver support # # CONFIG_VGA_CONSOLE is not set CONFIG_DUMMY_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE=y # CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y CONFIG_FONTS=y CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y # CONFIG_FONT_6x11 is not set # CONFIG_FONT_7x14 is not set # CONFIG_FONT_PEARL_8x8 is not set # CONFIG_FONT_ACORN_8x8 is not set # CONFIG_FONT_MINI_4x6 is not set # CONFIG_FONT_SUN8x16 is not set # CONFIG_FONT_SUN12x22 is not set # CONFIG_FONT_10x18 is not set CONFIG_LOGO=y CONFIG_LOGO_LINUX_MONO=y CONFIG_LOGO_LINUX_VGA16=y CONFIG_LOGO_LINUX_CLUT224=y CONFIG_SOUND=y CONFIG_SOUND_OSS_CORE=y CONFIG_SOUND_OSS_CORE_PRECLAIM=y CONFIG_SND=y CONFIG_SND_TIMER=y CONFIG_SND_PCM=y CONFIG_SND_HWDEP=y CONFIG_SND_RAWMIDI=y CONFIG_SND_JACK=y CONFIG_SND_SEQUENCER=m # CONFIG_SND_SEQ_DUMMY is not set CONFIG_SND_OSSEMUL=y CONFIG_SND_MIXER_OSS=y CONFIG_SND_PCM_OSS=y CONFIG_SND_PCM_OSS_PLUGINS=y CONFIG_SND_SEQUENCER_OSS=y # CONFIG_SND_HRTIMER is not set # CONFIG_SND_DYNAMIC_MINORS is not set CONFIG_SND_SUPPORT_OLD_API=y CONFIG_SND_VERBOSE_PROCFS=y CONFIG_SND_VERBOSE_PRINTK=y CONFIG_SND_DEBUG=y # CONFIG_SND_DEBUG_VERBOSE is not set # CONFIG_SND_PCM_XRUN_DEBUG is not set CONFIG_SND_RAWMIDI_SEQ=m # CONFIG_SND_OPL3_LIB_SEQ is not set # CONFIG_SND_OPL4_LIB_SEQ is not set # CONFIG_SND_SBAWE_SEQ is not set # CONFIG_SND_EMU10K1_SEQ is not set # CONFIG_SND_DRIVERS is not set # CONFIG_SND_ARM is not set # CONFIG_SND_SPI is not set CONFIG_SND_USB=y CONFIG_SND_USB_AUDIO=y # CONFIG_SND_USB_UA101 is not set CONFIG_SND_USB_CAIAQ=m CONFIG_SND_USB_CAIAQ_INPUT=y CONFIG_SND_SOC=y CONFIG_SND_OMAP_SOC=y CONFIG_SND_OMAP_SOC_MCBSP=y CONFIG_SND_OMAP_SOC_OVERO=y CONFIG_SND_SOC_I2C_AND_SPI=y # CONFIG_SND_SOC_ALL_CODECS is not set CONFIG_SND_SOC_TWL4030=y CONFIG_SOUND_PRIME=m CONFIG_HID_SUPPORT=y CONFIG_HID=y # CONFIG_HIDRAW is not set # # USB Input Devices # CONFIG_USB_HID=y # CONFIG_HID_PID is not set # CONFIG_USB_HIDDEV is not set # # Special HID drivers # # CONFIG_HID_3M_PCT is not set CONFIG_HID_A4TECH=y CONFIG_HID_APPLE=y CONFIG_HID_BELKIN=y CONFIG_HID_CHERRY=y CONFIG_HID_CHICONY=y CONFIG_HID_CYPRESS=y CONFIG_HID_DRAGONRISE=y # CONFIG_DRAGONRISE_FF is not set CONFIG_HID_EZKEY=y CONFIG_HID_KYE=y CONFIG_HID_GYRATION=y CONFIG_HID_TWINHAN=y CONFIG_HID_KENSINGTON=y CONFIG_HID_LOGITECH=y # CONFIG_LOGITECH_FF is not set # CONFIG_LOGIRUMBLEPAD2_FF is not set # CONFIG_LOGIG940_FF is not set # CONFIG_HID_MAGICMOUSE is not set CONFIG_HID_MICROSOFT=y # CONFIG_HID_MOSART is not set CONFIG_HID_MONTEREY=y CONFIG_HID_NTRIG=y # CONFIG_HID_ORTEK is not set CONFIG_HID_PANTHERLORD=y # CONFIG_PANTHERLORD_FF is not set CONFIG_HID_PETALYNX=y # CONFIG_HID_QUANTA is not set CONFIG_HID_SAMSUNG=y CONFIG_HID_SONY=y # CONFIG_HID_STANTUM is not set CONFIG_HID_SUNPLUS=y CONFIG_HID_GREENASIA=y # CONFIG_GREENASIA_FF is not set CONFIG_HID_SMARTJOYPLUS=y # CONFIG_SMARTJOYPLUS_FF is not set CONFIG_HID_TOPSEED=y CONFIG_HID_THRUSTMASTER=y # CONFIG_THRUSTMASTER_FF is not set CONFIG_HID_WACOM=y CONFIG_HID_ZEROPLUS=y # CONFIG_ZEROPLUS_FF is not set CONFIG_USB_SUPPORT=y CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB_ARCH_HAS_OHCI=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_USB=y CONFIG_USB_DEBUG=y CONFIG_USB_ANNOUNCE_NEW_DEVICES=y # # Miscellaneous USB options # CONFIG_USB_DEVICEFS=y CONFIG_USB_DEVICE_CLASS=y # CONFIG_USB_DYNAMIC_MINORS is not set CONFIG_USB_SUSPEND=y # CONFIG_USB_OTG is not set # CONFIG_USB_OTG_WHITELIST is not set # CONFIG_USB_OTG_BLACKLIST_HUB is not set CONFIG_USB_MON=y # CONFIG_USB_WUSB is not set # CONFIG_USB_WUSB_CBAF is not set # # USB Host Controller Drivers # # CONFIG_USB_C67X00_HCD is not set CONFIG_USB_EHCI_HCD=y CONFIG_USB_EHCI_ROOT_HUB_TT=y CONFIG_USB_EHCI_TT_NEWSCHED=y # CONFIG_USB_OXU210HP_HCD is not set # CONFIG_USB_ISP116X_HCD is not set # CONFIG_USB_ISP1760_HCD is not set # CONFIG_USB_ISP1362_HCD is not set # CONFIG_USB_OHCI_HCD is not set # CONFIG_USB_U132_HCD is not set # CONFIG_USB_SL811_HCD is not set # CONFIG_USB_R8A66597_HCD is not set # CONFIG_USB_HWA_HCD is not set CONFIG_USB_MUSB_HDRC=y CONFIG_USB_MUSB_SOC=y # # OMAP 343x high speed USB support # CONFIG_USB_MUSB_HOST=y # CONFIG_USB_MUSB_PERIPHERAL is not set # CONFIG_USB_MUSB_OTG is not set # CONFIG_USB_GADGET_MUSB_HDRC is not set CONFIG_USB_MUSB_HDRC_HCD=y # CONFIG_MUSB_PIO_ONLY is not set CONFIG_USB_INVENTRA_DMA=y # CONFIG_USB_TI_CPPI_DMA is not set CONFIG_USB_MUSB_DEBUG=y # # USB Device Class drivers # CONFIG_USB_ACM=m CONFIG_USB_PRINTER=m CONFIG_USB_WDM=m CONFIG_USB_TMC=m # # NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may # # # also be needed; see USB_STORAGE Help for more info # CONFIG_USB_STORAGE=y # CONFIG_USB_STORAGE_DEBUG is not set # CONFIG_USB_STORAGE_DATAFAB is not set CONFIG_USB_STORAGE_FREECOM=m CONFIG_USB_STORAGE_ISD200=m CONFIG_USB_STORAGE_USBAT=m CONFIG_USB_STORAGE_SDDR09=m CONFIG_USB_STORAGE_SDDR55=m CONFIG_USB_STORAGE_JUMPSHOT=m CONFIG_USB_STORAGE_ALAUDA=m CONFIG_USB_STORAGE_ONETOUCH=m CONFIG_USB_STORAGE_KARMA=m CONFIG_USB_STORAGE_CYPRESS_ATACB=m # CONFIG_USB_LIBUSUAL is not set # # USB Imaging devices # CONFIG_USB_MDC800=m CONFIG_USB_MICROTEK=m # # USB port drivers # CONFIG_USB_SERIAL=m CONFIG_USB_EZUSB=y CONFIG_USB_SERIAL_GENERIC=y CONFIG_USB_SERIAL_AIRCABLE=m CONFIG_USB_SERIAL_ARK3116=m CONFIG_USB_SERIAL_BELKIN=m CONFIG_USB_SERIAL_CH341=m CONFIG_USB_SERIAL_WHITEHEAT=m CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m CONFIG_USB_SERIAL_CP210X=m CONFIG_USB_SERIAL_CYPRESS_M8=m CONFIG_USB_SERIAL_EMPEG=m CONFIG_USB_SERIAL_FTDI_SIO=m CONFIG_USB_SERIAL_FUNSOFT=m CONFIG_USB_SERIAL_VISOR=m CONFIG_USB_SERIAL_IPAQ=m CONFIG_USB_SERIAL_IR=m CONFIG_USB_SERIAL_EDGEPORT=m CONFIG_USB_SERIAL_EDGEPORT_TI=m CONFIG_USB_SERIAL_GARMIN=m CONFIG_USB_SERIAL_IPW=m CONFIG_USB_SERIAL_IUU=m CONFIG_USB_SERIAL_KEYSPAN_PDA=m CONFIG_USB_SERIAL_KEYSPAN=m CONFIG_USB_SERIAL_KEYSPAN_MPR=y CONFIG_USB_SERIAL_KEYSPAN_USA28=y CONFIG_USB_SERIAL_KEYSPAN_USA28X=y CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y CONFIG_USB_SERIAL_KEYSPAN_USA19=y CONFIG_USB_SERIAL_KEYSPAN_USA18X=y CONFIG_USB_SERIAL_KEYSPAN_USA19W=y CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y CONFIG_USB_SERIAL_KEYSPAN_USA49W=y CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y CONFIG_USB_SERIAL_KLSI=m CONFIG_USB_SERIAL_KOBIL_SCT=m CONFIG_USB_SERIAL_MCT_U232=m CONFIG_USB_SERIAL_MOS7720=m CONFIG_USB_SERIAL_MOS7840=m CONFIG_USB_SERIAL_MOTOROLA=m CONFIG_USB_SERIAL_NAVMAN=m CONFIG_USB_SERIAL_PL2303=m CONFIG_USB_SERIAL_OTI6858=m CONFIG_USB_SERIAL_QUALCOMM=m CONFIG_USB_SERIAL_SPCP8X5=m CONFIG_USB_SERIAL_HP4X=m CONFIG_USB_SERIAL_SAFE=m # CONFIG_USB_SERIAL_SAFE_PADDED is not set CONFIG_USB_SERIAL_SIEMENS_MPI=m CONFIG_USB_SERIAL_SIERRAWIRELESS=m CONFIG_USB_SERIAL_SYMBOL=m CONFIG_USB_SERIAL_TI=m CONFIG_USB_SERIAL_CYBERJACK=m CONFIG_USB_SERIAL_XIRCOM=m CONFIG_USB_SERIAL_OPTION=m CONFIG_USB_SERIAL_OMNINET=m CONFIG_USB_SERIAL_OPTICON=m # CONFIG_USB_SERIAL_DEBUG is not set # # USB Miscellaneous drivers # CONFIG_USB_EMI62=m CONFIG_USB_EMI26=m CONFIG_USB_ADUTUX=m CONFIG_USB_SEVSEG=m CONFIG_USB_RIO500=m CONFIG_USB_LEGOTOWER=m CONFIG_USB_LCD=m CONFIG_USB_BERRY_CHARGE=m CONFIG_USB_LED=m CONFIG_USB_CYPRESS_CY7C63=m CONFIG_USB_CYTHERM=m CONFIG_USB_IDMOUSE=m CONFIG_USB_FTDI_ELAN=m CONFIG_USB_APPLEDISPLAY=m CONFIG_USB_SISUSBVGA=m CONFIG_USB_SISUSBVGA_CON=y CONFIG_USB_LD=m CONFIG_USB_TRANCEVIBRATOR=m CONFIG_USB_IOWARRIOR=m # CONFIG_USB_TEST is not set CONFIG_USB_ISIGHTFW=m CONFIG_USB_VST=m CONFIG_USB_GADGET=y # CONFIG_USB_GADGET_DEBUG is not set # CONFIG_USB_GADGET_DEBUG_FILES is not set # CONFIG_USB_GADGET_DEBUG_FS is not set CONFIG_USB_GADGET_VBUS_DRAW=500 CONFIG_USB_GADGET_SELECTED=y # CONFIG_USB_GADGET_AT91 is not set # CONFIG_USB_GADGET_ATMEL_USBA is not set # CONFIG_USB_GADGET_FSL_USB2 is not set # CONFIG_USB_GADGET_LH7A40X is not set CONFIG_USB_GADGET_OMAP=y CONFIG_USB_OMAP=y # CONFIG_USB_GADGET_PXA25X is not set # CONFIG_USB_GADGET_R8A66597 is not set # CONFIG_USB_GADGET_PXA27X is not set # CONFIG_USB_GADGET_S3C_HSOTG is not set # CONFIG_USB_GADGET_IMX is not set # CONFIG_USB_GADGET_S3C2410 is not set # CONFIG_USB_GADGET_M66592 is not set # CONFIG_USB_GADGET_AMD5536UDC is not set # CONFIG_USB_GADGET_FSL_QE is not set # CONFIG_USB_GADGET_CI13XXX is not set # CONFIG_USB_GADGET_NET2280 is not set # CONFIG_USB_GADGET_GOKU is not set # CONFIG_USB_GADGET_LANGWELL is not set # CONFIG_USB_GADGET_DUMMY_HCD is not set # CONFIG_USB_GADGET_DUALSPEED is not set # CONFIG_USB_ZERO is not set CONFIG_USB_AUDIO=m CONFIG_USB_ETH=m CONFIG_USB_ETH_RNDIS=y # CONFIG_USB_ETH_EEM is not set # CONFIG_USB_GADGETFS is not set CONFIG_USB_FILE_STORAGE=m # CONFIG_USB_FILE_STORAGE_TEST is not set # CONFIG_USB_MASS_STORAGE is not set CONFIG_USB_G_SERIAL=m # CONFIG_USB_MIDI_GADGET is not set CONFIG_USB_G_PRINTER=m CONFIG_USB_CDC_COMPOSITE=m # CONFIG_USB_G_MULTI is not set # # OTG and related infrastructure # CONFIG_USB_OTG_UTILS=y # CONFIG_USB_GPIO_VBUS is not set # CONFIG_ISP1301_OMAP is not set # CONFIG_USB_ULPI is not set CONFIG_TWL4030_USB=y # CONFIG_NOP_USB_XCEIV is not set CONFIG_MMC=y # CONFIG_MMC_DEBUG is not set # CONFIG_MMC_UNSAFE_RESUME is not set # # MMC/SD/SDIO Card Drivers # CONFIG_MMC_BLOCK=y CONFIG_MMC_BLOCK_BOUNCE=y CONFIG_SDIO_UART=y # CONFIG_MMC_TEST is not set # # MMC/SD/SDIO Host Controller Drivers # # CONFIG_MMC_SDHCI is not set # CONFIG_MMC_OMAP is not set CONFIG_MMC_OMAP_HS=y # CONFIG_MMC_AT91 is not set # CONFIG_MMC_ATMELMCI is not set # CONFIG_MMC_SPI is not set # CONFIG_MEMSTICK is not set CONFIG_NEW_LEDS=y CONFIG_LEDS_CLASS=y # # LED drivers # # CONFIG_LEDS_PCA9532 is not set CONFIG_LEDS_GPIO=y CONFIG_LEDS_GPIO_PLATFORM=y # CONFIG_LEDS_LP3944 is not set CONFIG_LEDS_PCA955X=m # CONFIG_LEDS_DAC124S085 is not set # CONFIG_LEDS_REGULATOR is not set # CONFIG_LEDS_BD2802 is not set # CONFIG_LEDS_LT3593 is not set # # LED Triggers # CONFIG_LEDS_TRIGGERS=y CONFIG_LEDS_TRIGGER_TIMER=y CONFIG_LEDS_TRIGGER_HEARTBEAT=y CONFIG_LEDS_TRIGGER_BACKLIGHT=y CONFIG_LEDS_TRIGGER_GPIO=y CONFIG_LEDS_TRIGGER_DEFAULT_ON=y # # iptables trigger is under Netfilter config (LED target) # # CONFIG_ACCESSIBILITY is not set CONFIG_RTC_LIB=y CONFIG_RTC_CLASS=y CONFIG_RTC_HCTOSYS=y CONFIG_RTC_HCTOSYS_DEVICE="rtc0" # CONFIG_RTC_DEBUG is not set # # RTC interfaces # CONFIG_RTC_INTF_SYSFS=y CONFIG_RTC_INTF_PROC=y CONFIG_RTC_INTF_DEV=y # CONFIG_RTC_INTF_DEV_UIE_EMUL is not set # CONFIG_RTC_DRV_TEST is not set # # I2C RTC drivers # # CONFIG_RTC_DRV_DS1307 is not set # CONFIG_RTC_DRV_DS1374 is not set # CONFIG_RTC_DRV_DS1672 is not set # CONFIG_RTC_DRV_MAX6900 is not set # CONFIG_RTC_DRV_RS5C372 is not set # CONFIG_RTC_DRV_ISL1208 is not set # CONFIG_RTC_DRV_X1205 is not set # CONFIG_RTC_DRV_PCF8563 is not set # CONFIG_RTC_DRV_PCF8583 is not set # CONFIG_RTC_DRV_M41T80 is not set # CONFIG_RTC_DRV_BQ32K is not set CONFIG_RTC_DRV_TWL4030=y # CONFIG_RTC_DRV_S35390A is not set # CONFIG_RTC_DRV_FM3130 is not set # CONFIG_RTC_DRV_RX8581 is not set # CONFIG_RTC_DRV_RX8025 is not set # # SPI RTC drivers # # CONFIG_RTC_DRV_M41T94 is not set # CONFIG_RTC_DRV_DS1305 is not set # CONFIG_RTC_DRV_DS1390 is not set # CONFIG_RTC_DRV_MAX6902 is not set # CONFIG_RTC_DRV_R9701 is not set # CONFIG_RTC_DRV_RS5C348 is not set # CONFIG_RTC_DRV_DS3234 is not set # CONFIG_RTC_DRV_PCF2123 is not set # # Platform RTC drivers # # CONFIG_RTC_DRV_CMOS is not set # CONFIG_RTC_DRV_DS1286 is not set # CONFIG_RTC_DRV_DS1511 is not set # CONFIG_RTC_DRV_DS1553 is not set # CONFIG_RTC_DRV_DS1742 is not set # CONFIG_RTC_DRV_STK17TA8 is not set # CONFIG_RTC_DRV_M48T86 is not set # CONFIG_RTC_DRV_M48T35 is not set # CONFIG_RTC_DRV_M48T59 is not set # CONFIG_RTC_DRV_MSM6242 is not set # CONFIG_RTC_DRV_BQ4802 is not set # CONFIG_RTC_DRV_RP5C01 is not set # CONFIG_RTC_DRV_V3020 is not set # # on-CPU RTC drivers # # CONFIG_DMADEVICES is not set # CONFIG_AUXDISPLAY is not set # CONFIG_UIO is not set # # TI VLYNQ # # CONFIG_STAGING is not set # # CBUS support # # CONFIG_CBUS is not set # # File systems # CONFIG_EXT2_FS=y # CONFIG_EXT2_FS_XATTR is not set # CONFIG_EXT2_FS_XIP is not set CONFIG_EXT3_FS=y # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set # CONFIG_EXT3_FS_XATTR is not set # CONFIG_EXT4_FS is not set CONFIG_JBD=y # CONFIG_JBD_DEBUG is not set # CONFIG_REISERFS_FS is not set # CONFIG_JFS_FS is not set CONFIG_FS_POSIX_ACL=y # CONFIG_XFS_FS is not set # CONFIG_GFS2_FS is not set # CONFIG_OCFS2_FS is not set # CONFIG_BTRFS_FS is not set # CONFIG_NILFS2_FS is not set CONFIG_FILE_LOCKING=y CONFIG_FSNOTIFY=y CONFIG_DNOTIFY=y CONFIG_INOTIFY=y CONFIG_INOTIFY_USER=y CONFIG_QUOTA=y # CONFIG_QUOTA_NETLINK_INTERFACE is not set CONFIG_PRINT_QUOTA_WARNING=y CONFIG_QUOTA_TREE=y # CONFIG_QFMT_V1 is not set CONFIG_QFMT_V2=y CONFIG_QUOTACTL=y # CONFIG_AUTOFS_FS is not set # CONFIG_AUTOFS4_FS is not set CONFIG_FUSE_FS=m # CONFIG_CUSE is not set CONFIG_GENERIC_ACL=y # # Caches # # CONFIG_FSCACHE is not set # # CD-ROM/DVD Filesystems # CONFIG_ISO9660_FS=m CONFIG_JOLIET=y CONFIG_ZISOFS=y CONFIG_UDF_FS=m CONFIG_UDF_NLS=y # # DOS/FAT/NT Filesystems # CONFIG_FAT_FS=y CONFIG_MSDOS_FS=y CONFIG_VFAT_FS=y CONFIG_FAT_DEFAULT_CODEPAGE=437 CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1" # CONFIG_NTFS_FS is not set # # Pseudo filesystems # CONFIG_PROC_FS=y CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y CONFIG_SYSFS=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y # CONFIG_HUGETLB_PAGE is not set # CONFIG_CONFIGFS_FS is not set CONFIG_MISC_FILESYSTEMS=y # CONFIG_ADFS_FS is not set # CONFIG_AFFS_FS is not set # CONFIG_HFS_FS is not set # CONFIG_HFSPLUS_FS is not set # CONFIG_BEFS_FS is not set # CONFIG_BFS_FS is not set # CONFIG_EFS_FS is not set CONFIG_JFFS2_FS=y CONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y # CONFIG_JFFS2_FS_WBUF_VERIFY is not set CONFIG_JFFS2_SUMMARY=y CONFIG_JFFS2_FS_XATTR=y CONFIG_JFFS2_FS_POSIX_ACL=y CONFIG_JFFS2_FS_SECURITY=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=y CONFIG_JFFS2_LZO=y CONFIG_JFFS2_RTIME=y CONFIG_JFFS2_RUBIN=y # CONFIG_JFFS2_CMODE_NONE is not set CONFIG_JFFS2_CMODE_PRIORITY=y # CONFIG_JFFS2_CMODE_SIZE is not set # CONFIG_JFFS2_CMODE_FAVOURLZO is not set CONFIG_UBIFS_FS=y # CONFIG_UBIFS_FS_XATTR is not set # CONFIG_UBIFS_FS_ADVANCED_COMPR is not set CONFIG_UBIFS_FS_LZO=y CONFIG_UBIFS_FS_ZLIB=y CONFIG_UBIFS_FS_DEBUG=y CONFIG_UBIFS_FS_DEBUG_MSG_LVL=3 # CONFIG_UBIFS_FS_DEBUG_CHKS is not set # CONFIG_CRAMFS is not set # CONFIG_SQUASHFS is not set # CONFIG_VXFS_FS is not set # CONFIG_MINIX_FS is not set # CONFIG_OMFS_FS is not set # CONFIG_HPFS_FS is not set # CONFIG_QNX4FS_FS is not set # CONFIG_ROMFS_FS is not set # CONFIG_SYSV_FS is not set # CONFIG_UFS_FS is not set CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=y CONFIG_NFS_V3=y # CONFIG_NFS_V3_ACL is not set CONFIG_NFS_V4=y # CONFIG_NFS_V4_1 is not set CONFIG_ROOT_NFS=y # CONFIG_NFSD is not set CONFIG_LOCKD=y CONFIG_LOCKD_V4=y CONFIG_NFS_COMMON=y CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y CONFIG_RPCSEC_GSS_KRB5=y # CONFIG_RPCSEC_GSS_SPKM3 is not set # CONFIG_SMB_FS is not set # CONFIG_CIFS is not set # CONFIG_NCP_FS is not set # CONFIG_CODA_FS is not set # CONFIG_AFS_FS is not set # # Partition Types # CONFIG_PARTITION_ADVANCED=y # CONFIG_ACORN_PARTITION is not set # CONFIG_OSF_PARTITION is not set # CONFIG_AMIGA_PARTITION is not set # CONFIG_ATARI_PARTITION is not set # CONFIG_MAC_PARTITION is not set CONFIG_MSDOS_PARTITION=y # CONFIG_BSD_DISKLABEL is not set # CONFIG_MINIX_SUBPARTITION is not set # CONFIG_SOLARIS_X86_PARTITION is not set # CONFIG_UNIXWARE_DISKLABEL is not set # CONFIG_LDM_PARTITION is not set # CONFIG_SGI_PARTITION is not set # CONFIG_ULTRIX_PARTITION is not set # CONFIG_SUN_PARTITION is not set # CONFIG_KARMA_PARTITION is not set # CONFIG_EFI_PARTITION is not set # CONFIG_SYSV68_PARTITION is not set CONFIG_NLS=y CONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_437=y # CONFIG_NLS_CODEPAGE_737 is not set # CONFIG_NLS_CODEPAGE_775 is not set # CONFIG_NLS_CODEPAGE_850 is not set # CONFIG_NLS_CODEPAGE_852 is not set # CONFIG_NLS_CODEPAGE_855 is not set # CONFIG_NLS_CODEPAGE_857 is not set # CONFIG_NLS_CODEPAGE_860 is not set # CONFIG_NLS_CODEPAGE_861 is not set # CONFIG_NLS_CODEPAGE_862 is not set # CONFIG_NLS_CODEPAGE_863 is not set # CONFIG_NLS_CODEPAGE_864 is not set # CONFIG_NLS_CODEPAGE_865 is not set # CONFIG_NLS_CODEPAGE_866 is not set # CONFIG_NLS_CODEPAGE_869 is not set # CONFIG_NLS_CODEPAGE_936 is not set # CONFIG_NLS_CODEPAGE_950 is not set # CONFIG_NLS_CODEPAGE_932 is not set # CONFIG_NLS_CODEPAGE_949 is not set # CONFIG_NLS_CODEPAGE_874 is not set # CONFIG_NLS_ISO8859_8 is not set # CONFIG_NLS_CODEPAGE_1250 is not set # CONFIG_NLS_CODEPAGE_1251 is not set # CONFIG_NLS_ASCII is not set CONFIG_NLS_ISO8859_1=y # CONFIG_NLS_ISO8859_2 is not set # CONFIG_NLS_ISO8859_3 is not set # CONFIG_NLS_ISO8859_4 is not set # CONFIG_NLS_ISO8859_5 is not set # CONFIG_NLS_ISO8859_6 is not set # CONFIG_NLS_ISO8859_7 is not set # CONFIG_NLS_ISO8859_9 is not set # CONFIG_NLS_ISO8859_13 is not set # CONFIG_NLS_ISO8859_14 is not set # CONFIG_NLS_ISO8859_15 is not set # CONFIG_NLS_KOI8_R is not set # CONFIG_NLS_KOI8_U is not set # CONFIG_NLS_UTF8 is not set # CONFIG_DLM is not set # # Kernel hacking # # CONFIG_PRINTK_TIME is not set CONFIG_ENABLE_WARN_DEPRECATED=y CONFIG_ENABLE_MUST_CHECK=y CONFIG_FRAME_WARN=1024 CONFIG_MAGIC_SYSRQ=y # CONFIG_STRIP_ASM_SYMS is not set # CONFIG_UNUSED_SYMBOLS is not set CONFIG_DEBUG_FS=y # CONFIG_HEADERS_CHECK is not set CONFIG_IPIPE_DEBUG=y CONFIG_IPIPE_DEBUG_CONTEXT=y CONFIG_IPIPE_DEBUG_INTERNAL=y CONFIG_IPIPE_TRACE=y CONFIG_IPIPE_TRACE_ENABLE=y CONFIG_IPIPE_TRACE_MCOUNT=y CONFIG_IPIPE_TRACE_IRQSOFF=y CONFIG_IPIPE_TRACE_SHIFT=14 CONFIG_IPIPE_TRACE_VMALLOC=y CONFIG_IPIPE_TRACE_PANIC=y CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_SHIRQ=y CONFIG_DETECT_SOFTLOCKUP=y # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0 CONFIG_DETECT_HUNG_TASK=y # CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0 CONFIG_SCHED_DEBUG=y CONFIG_SCHEDSTATS=y CONFIG_TIMER_STATS=y CONFIG_DEBUG_OBJECTS=y # CONFIG_DEBUG_OBJECTS_SELFTEST is not set # CONFIG_DEBUG_OBJECTS_FREE is not set # CONFIG_DEBUG_OBJECTS_TIMERS is not set # CONFIG_DEBUG_OBJECTS_WORK is not set CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT=1 # CONFIG_DEBUG_SLAB is not set # CONFIG_DEBUG_KMEMLEAK is not set # CONFIG_DEBUG_RT_MUTEXES is not set # CONFIG_RT_MUTEX_TESTER is not set # CONFIG_DEBUG_SPINLOCK is not set CONFIG_DEBUG_MUTEXES=y # CONFIG_DEBUG_LOCK_ALLOC is not set # CONFIG_PROVE_LOCKING is not set # CONFIG_LOCK_STAT is not set # CONFIG_DEBUG_SPINLOCK_SLEEP is not set # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set CONFIG_STACKTRACE=y # CONFIG_DEBUG_KOBJECT is not set # CONFIG_DEBUG_BUGVERBOSE is not set CONFIG_DEBUG_INFO=y CONFIG_DEBUG_VM=y CONFIG_DEBUG_WRITECOUNT=y CONFIG_DEBUG_MEMORY_INIT=y # CONFIG_DEBUG_LIST is not set # CONFIG_DEBUG_SG is not set # CONFIG_DEBUG_NOTIFIERS is not set # CONFIG_DEBUG_CREDENTIALS is not set CONFIG_FRAME_POINTER=y # CONFIG_BOOT_PRINTK_DELAY is not set # CONFIG_RCU_TORTURE_TEST is not set # CONFIG_RCU_CPU_STALL_DETECTOR is not set # CONFIG_BACKTRACE_SELF_TEST is not set # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set # CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set # CONFIG_FAULT_INJECTION is not set # CONFIG_LATENCYTOP is not set # CONFIG_SYSCTL_SYSCALL_CHECK is not set # CONFIG_PAGE_POISONING is not set CONFIG_NOP_TRACER=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_RING_BUFFER=y CONFIG_EVENT_TRACING=y CONFIG_CONTEXT_SWITCH_TRACER=y CONFIG_RING_BUFFER_ALLOW_SWAP=y CONFIG_TRACING=y CONFIG_GENERIC_TRACER=y CONFIG_TRACING_SUPPORT=y CONFIG_FTRACE=y CONFIG_FUNCTION_TRACER=y # CONFIG_IRQSOFF_TRACER is not set # CONFIG_SCHED_TRACER is not set # CONFIG_BOOT_TRACER is not set CONFIG_BRANCH_PROFILE_NONE=y # CONFIG_PROFILE_ANNOTATED_BRANCHES is not set # CONFIG_PROFILE_ALL_BRANCHES is not set # CONFIG_STACK_TRACER is not set # CONFIG_KMEMTRACE is not set # CONFIG_WORKQUEUE_TRACER is not set # CONFIG_BLK_DEV_IO_TRACE is not set CONFIG_DYNAMIC_FTRACE=y # CONFIG_FUNCTION_PROFILER is not set CONFIG_FTRACE_MCOUNT_RECORD=y # CONFIG_FTRACE_STARTUP_TEST is not set # CONFIG_RING_BUFFER_BENCHMARK is not set # CONFIG_DYNAMIC_DEBUG is not set # CONFIG_SAMPLES is not set CONFIG_HAVE_ARCH_KGDB=y # CONFIG_KGDB is not set CONFIG_ARM_UNWIND=y # CONFIG_DEBUG_USER is not set # CONFIG_DEBUG_ERRORS is not set # CONFIG_DEBUG_STACK_USAGE is not set CONFIG_DEBUG_LL=y CONFIG_EARLY_PRINTK=y # CONFIG_DEBUG_ICEDCC is not set # CONFIG_OC_ETM is not set # # Security options # # CONFIG_KEYS is not set # CONFIG_SECURITY is not set # CONFIG_SECURITYFS is not set # CONFIG_DEFAULT_SECURITY_SELINUX is not set # CONFIG_DEFAULT_SECURITY_SMACK is not set # CONFIG_DEFAULT_SECURITY_TOMOYO is not set CONFIG_DEFAULT_SECURITY_DAC=y CONFIG_DEFAULT_SECURITY="" CONFIG_XOR_BLOCKS=m CONFIG_ASYNC_CORE=m CONFIG_ASYNC_MEMCPY=m CONFIG_ASYNC_XOR=m CONFIG_ASYNC_PQ=m CONFIG_ASYNC_RAID6_RECOV=m CONFIG_CRYPTO=y # # Crypto core or helper # CONFIG_CRYPTO_ALGAPI=y CONFIG_CRYPTO_ALGAPI2=y CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_BLKCIPHER=y CONFIG_CRYPTO_BLKCIPHER2=y CONFIG_CRYPTO_HASH=y CONFIG_CRYPTO_HASH2=y CONFIG_CRYPTO_RNG2=y CONFIG_CRYPTO_PCOMP=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_GF128MUL=m CONFIG_CRYPTO_NULL=m CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_CRYPTD=m # CONFIG_CRYPTO_AUTHENC is not set CONFIG_CRYPTO_TEST=m # # Authenticated Encryption with Associated Data # # CONFIG_CRYPTO_CCM is not set # CONFIG_CRYPTO_GCM is not set # CONFIG_CRYPTO_SEQIV is not set # # Block modes # CONFIG_CRYPTO_CBC=y # CONFIG_CRYPTO_CTR is not set # CONFIG_CRYPTO_CTS is not set CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m # CONFIG_CRYPTO_XTS is not set # # Hash modes # CONFIG_CRYPTO_HMAC=m CONFIG_CRYPTO_XCBC=m # CONFIG_CRYPTO_VMAC is not set # # Digest # CONFIG_CRYPTO_CRC32C=y # CONFIG_CRYPTO_GHASH is not set CONFIG_CRYPTO_MD4=m CONFIG_CRYPTO_MD5=y CONFIG_CRYPTO_MICHAEL_MIC=y # CONFIG_CRYPTO_RMD128 is not set # CONFIG_CRYPTO_RMD160 is not set # CONFIG_CRYPTO_RMD256 is not set # CONFIG_CRYPTO_RMD320 is not set CONFIG_CRYPTO_SHA1=m CONFIG_CRYPTO_SHA256=m CONFIG_CRYPTO_SHA512=m CONFIG_CRYPTO_TGR192=m CONFIG_CRYPTO_WP512=m # # Ciphers # CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_ANUBIS=m CONFIG_CRYPTO_ARC4=y CONFIG_CRYPTO_BLOWFISH=m CONFIG_CRYPTO_CAMELLIA=m CONFIG_CRYPTO_CAST5=m CONFIG_CRYPTO_CAST6=m CONFIG_CRYPTO_DES=y CONFIG_CRYPTO_FCRYPT=m CONFIG_CRYPTO_KHAZAD=m # CONFIG_CRYPTO_SALSA20 is not set # CONFIG_CRYPTO_SEED is not set CONFIG_CRYPTO_SERPENT=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_TWOFISH_COMMON=m # # Compression # CONFIG_CRYPTO_DEFLATE=y # CONFIG_CRYPTO_ZLIB is not set CONFIG_CRYPTO_LZO=y # # Random Number Generation # # CONFIG_CRYPTO_ANSI_CPRNG is not set CONFIG_CRYPTO_HW=y CONFIG_BINARY_PRINTF=y # # Library routines # CONFIG_BITREVERSE=y CONFIG_GENERIC_FIND_LAST_BIT=y CONFIG_CRC_CCITT=y CONFIG_CRC16=y CONFIG_CRC_T10DIF=y CONFIG_CRC_ITU_T=y CONFIG_CRC32=y CONFIG_CRC7=y CONFIG_LIBCRC32C=y CONFIG_ZLIB_INFLATE=y CONFIG_ZLIB_DEFLATE=y CONFIG_LZO_COMPRESS=y CONFIG_LZO_DECOMPRESS=y CONFIG_DECOMPRESS_GZIP=y CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT=y CONFIG_HAS_DMA=y CONFIG_NLATTR=y [-- Attachment #5: adeos-ipipe-2.6.33-arm-1.16-01-gumstix-overo-linux-omap.patch --] [-- Type: text/x-patch, Size: 445061 bytes --] diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 184a6bd..1a8674b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -254,6 +254,8 @@ config ARCH_VERSATILE config ARCH_AT91 bool "Atmel AT91" select GENERIC_GPIO + select GENERIC_TIME if IPIPE + select GENERIC_CLOCKEVENTS if IPIPE select ARCH_REQUIRE_GPIOLIB select HAVE_CLK help @@ -514,7 +516,7 @@ config ARCH_KS8695 bool "Micrel/Kendin KS8695" select CPU_ARM922T select GENERIC_GPIO - select ARCH_REQUIRE_GPIOLIB + select ARCH_REQUIRE_GPIOLIB help Support for Micrel/Kendin KS8695 "Centaur" (ARM922T) based System-on-Chip devices. @@ -1070,6 +1072,8 @@ config LOCAL_TIMERS accounting to be spread across the timer interval, preventing a "thundering herd" at every timer tick. +source kernel/ipipe/Kconfig + source kernel/Kconfig.preempt config HZ diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S index 4fddc50..cbb2e4c 100644 --- a/arch/arm/boot/compressed/head.S +++ b/arch/arm/boot/compressed/head.S @@ -1074,6 +1074,15 @@ memdump: mov r12, r0 mov pc, r10 #endif +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + .text + .align 0 + .type mcount %function + .global mcount +mcount: + mov pc, lr @ just return +#endif + .ltorg reloc_end: diff --git a/arch/arm/common/it8152.c b/arch/arm/common/it8152.c index 2793447..5ec1999 100644 --- a/arch/arm/common/it8152.c +++ b/arch/arm/common/it8152.c @@ -26,6 +26,7 @@ #include <linux/ioport.h> #include <linux/irq.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <asm/mach/pci.h> #include <asm/hardware/it8152.h> @@ -120,21 +121,21 @@ void it8152_irq_demux(unsigned int irq, struct irq_desc *desc) bits_pd &= ((1 << IT8152_PD_IRQ_COUNT) - 1); while (bits_pd) { i = __ffs(bits_pd); - generic_handle_irq(IT8152_PD_IRQ(i)); + ipipe_handle_irq_cond(IT8152_PD_IRQ(i)); bits_pd &= ~(1 << i); } bits_lp &= ((1 << IT8152_LP_IRQ_COUNT) - 1); while (bits_lp) { i = __ffs(bits_lp); - generic_handle_irq(IT8152_LP_IRQ(i)); + ipipe_handle_irq_cond(IT8152_LP_IRQ(i)); bits_lp &= ~(1 << i); } bits_ld &= ((1 << IT8152_LD_IRQ_COUNT) - 1); while (bits_ld) { i = __ffs(bits_ld); - generic_handle_irq(IT8152_LD_IRQ(i)); + ipipe_handle_irq_cond(IT8152_LD_IRQ(i)); bits_ld &= ~(1 << i); } } diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 00f46d9..84ea0d3 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -81,6 +81,18 @@ .macro enable_irq_notrace cpsie i .endm + + .macro disable_irq_cond +#ifdef CONFIG_IPIPE + cpsid i +#endif /* CONFIG_IPIPE */ + .endm + + .macro enable_irq_cond +#ifdef CONFIG_IPIPE + cpsie i +#endif /* CONFIG_IPIPE */ + .endm #else .macro disable_irq_notrace msr cpsr_c, #PSR_I_BIT | SVC_MODE @@ -89,6 +101,18 @@ .macro enable_irq_notrace msr cpsr_c, #SVC_MODE .endm + + .macro disable_irq_cond +#ifdef CONFIG_IPIPE + msr cpsr_c, #PSR_I_BIT | SVC_MODE +#endif /* CONFIG_IPIPE */ + .endm + + .macro enable_irq_cond +#ifdef CONFIG_IPIPE + msr cpsr_c, #SVC_MODE +#endif /* CONFIG_IPIPE */ + .endm #endif .macro asm_trace_hardirqs_off diff --git a/arch/arm/include/asm/atomic.h b/arch/arm/include/asm/atomic.h index d0daeab..d4f9618 100644 --- a/arch/arm/include/asm/atomic.h +++ b/arch/arm/include/asm/atomic.h @@ -158,10 +158,10 @@ static inline int atomic_add_return(int i, atomic_t *v) unsigned long flags; int val; - raw_local_irq_save(flags); + local_irq_save_hw(flags); val = v->counter; v->counter = val += i; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return val; } @@ -172,10 +172,10 @@ static inline int atomic_sub_return(int i, atomic_t *v) unsigned long flags; int val; - raw_local_irq_save(flags); + local_irq_save_hw(flags); val = v->counter; v->counter = val -= i; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return val; } @@ -186,11 +186,11 @@ static inline int atomic_cmpxchg(atomic_t *v, int old, int new) int ret; unsigned long flags; - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = v->counter; if (likely(ret == old)) v->counter = new; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return ret; } @@ -199,9 +199,9 @@ static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) { unsigned long flags; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *addr &= ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } #endif /* __LINUX_ARM_ARCH__ */ diff --git a/arch/arm/include/asm/bitops.h b/arch/arm/include/asm/bitops.h index 338ff19..acc35cd 100644 --- a/arch/arm/include/asm/bitops.h +++ b/arch/arm/include/asm/bitops.h @@ -41,9 +41,9 @@ static inline void ____atomic_set_bit(unsigned int bit, volatile unsigned long * p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p |= mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline void ____atomic_clear_bit(unsigned int bit, volatile unsigned long *p) @@ -53,9 +53,9 @@ static inline void ____atomic_clear_bit(unsigned int bit, volatile unsigned long p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p &= ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline void ____atomic_change_bit(unsigned int bit, volatile unsigned long *p) @@ -65,9 +65,9 @@ static inline void ____atomic_change_bit(unsigned int bit, volatile unsigned lon p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); *p ^= mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); } static inline int @@ -79,10 +79,10 @@ ____atomic_test_and_set_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res | mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } @@ -96,10 +96,10 @@ ____atomic_test_and_clear_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res & ~mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } @@ -113,10 +113,10 @@ ____atomic_test_and_change_bit(unsigned int bit, volatile unsigned long *p) p += bit >> 5; - raw_local_irq_save(flags); + local_irq_save_hw(flags); res = *p; *p = res ^ mask; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); return (res & mask) != 0; } diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h index 8113bb5..0e3866a 100644 --- a/arch/arm/include/asm/cacheflush.h +++ b/arch/arm/include/asm/cacheflush.h @@ -15,6 +15,7 @@ #include <asm/glue.h> #include <asm/shmparam.h> #include <asm/cachetype.h> +#include <asm/fcse.h> #define CACHE_COLOUR(vaddr) ((vaddr & (SHMLBA - 1)) >> PAGE_SHIFT) @@ -326,16 +327,38 @@ static inline void outer_flush_range(unsigned long start, unsigned long end) #endif +#ifdef CONFIG_ARM_FCSE +#define FCSE_CACHE_MASK (~(L1_CACHE_BYTES - 1)) +#define FCSE_CACHE_ALIGN(addr) (((addr) + ~FCSE_CACHE_MASK) & FCSE_CACHE_MASK) + +static inline void +fcse_flush_cache_user_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + if (cache_is_vivt() + && fcse_mm_in_cache(vma->vm_mm)) { + start = fcse_va_to_mva(vma->vm_mm, start & FCSE_CACHE_MASK); + end = fcse_va_to_mva(vma->vm_mm, FCSE_CACHE_ALIGN(end)); + __cpuc_flush_user_range(start, end, vma->vm_flags); + } +} +#undef FCSE_CACHE_MASK +#undef FCSE_CACHE_ALIGN +#else /* ! CONFIG_ARM_FCSE */ +#define fcse_flush_cache_user_range(vma, start, end) do { } while (0) +#endif /* ! CONFIG_ARM_FCSE */ + /* * Copy user data from/to a page which is mapped into a different * processes address space. Really, we want to allow our "user * space" model to handle this. */ extern void copy_to_user_page(struct vm_area_struct *, struct page *, unsigned long, void *, const void *, unsigned long); -#define copy_from_user_page(vma, page, vaddr, dst, src, len) \ - do { \ - memcpy(dst, src, len); \ +#define copy_from_user_page(vma, page, vaddr, dst, src, len) \ + do { \ + fcse_flush_cache_user_range(vma, vaddr, vaddr + len); \ + memcpy(dst, src, len); \ } while (0) /* @@ -349,23 +373,32 @@ static inline void outer_flush_range(unsigned long start, unsigned long end) static inline void vivt_flush_cache_mm(struct mm_struct *mm) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) - __cpuc_flush_user_all(); + if (fcse_mm_in_cache(mm)) { + unsigned seq; + + do { + seq = fcse_flush_all_start(mm); + __cpuc_flush_user_all(); + } while (!fcse_flush_all_done(mm, seq, 1)); + } } static inline void vivt_flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) - __cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end), - vma->vm_flags); + if (fcse_mm_in_cache(vma->vm_mm)) { + start = fcse_va_to_mva(vma->vm_mm, start & PAGE_MASK); + end = fcse_va_to_mva(vma->vm_mm, PAGE_ALIGN(end)); + __cpuc_flush_user_range(start, end, vma->vm_flags); + } } static inline void vivt_flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn) { - if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { - unsigned long addr = user_addr & PAGE_MASK; + if (fcse_mm_in_cache(vma->vm_mm)) { + unsigned long addr; + addr = fcse_va_to_mva(vma->vm_mm, user_addr) & PAGE_MASK; __cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags); } } @@ -406,14 +439,22 @@ extern void flush_ptrace_access(struct vm_area_struct *vma, struct page *page, * Harvard caches are synchronised for the user space address range. * This is used for the ARM private sys_cacheflush system call. */ -#define flush_cache_user_range(vma,start,end) \ - __cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end)) +#define flush_cache_user_range(vma, start, end) \ + ({ \ + struct mm_struct *_mm = (vma)->vm_mm; \ + unsigned long _start, _end; \ + _start = fcse_va_to_mva(_mm, start) & PAGE_MASK; \ + _end = PAGE_ALIGN(fcse_va_to_mva(_mm, end)); \ + __cpuc_coherent_user_range(_start, _end); \ + }) /* * Perform necessary cache operations to ensure that data previously * stored within this range of addresses can be executed by the CPU. */ -#define flush_icache_range(s,e) __cpuc_coherent_kern_range(s,e) +#define flush_icache_range(s,e) \ + __cpuc_coherent_kern_range(fcse_va_to_mva(current->mm, (s)), \ + fcse_va_to_mva(current->mm, (e))) /* * Perform necessary cache operations to ensure that the TLB will @@ -455,7 +496,8 @@ static inline void flush_anon_page(struct vm_area_struct *vma, extern void __flush_anon_page(struct vm_area_struct *vma, struct page *, unsigned long); if (PageAnon(page)) - __flush_anon_page(vma, page, vmaddr); + __flush_anon_page(vma, page, + fcse_va_to_mva(vma->vm_mm, vmaddr)); } #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE diff --git a/arch/arm/include/asm/cpu-multi32.h b/arch/arm/include/asm/cpu-multi32.h index e2b5b0b..18ca8a0 100644 --- a/arch/arm/include/asm/cpu-multi32.h +++ b/arch/arm/include/asm/cpu-multi32.h @@ -52,7 +52,8 @@ extern struct processor { /* * Set the page table */ - void (*switch_mm)(unsigned long pgd_phys, struct mm_struct *mm); + void (*switch_mm)(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); /* * Set a possibly extended PTE. Non-extended PTEs should * ignore 'ext'. @@ -66,4 +67,4 @@ extern struct processor { #define cpu_do_idle() processor._do_idle() #define cpu_dcache_clean_area(addr,sz) processor.dcache_clean_area(addr,sz) #define cpu_set_pte_ext(ptep,pte,ext) processor.set_pte_ext(ptep,pte,ext) -#define cpu_do_switch_mm(pgd,mm) processor.switch_mm(pgd,mm) +#define cpu_do_switch_mm(pgd,mm,flush) processor.switch_mm(pgd,mm,flush) diff --git a/arch/arm/include/asm/cpu-single.h b/arch/arm/include/asm/cpu-single.h index f073a6d..ec556e3 100644 --- a/arch/arm/include/asm/cpu-single.h +++ b/arch/arm/include/asm/cpu-single.h @@ -39,6 +39,7 @@ extern void cpu_proc_init(void); extern void cpu_proc_fin(void); extern int cpu_do_idle(void); extern void cpu_dcache_clean_area(void *, int); -extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm); +extern void cpu_do_switch_mm(unsigned long pgd_phys, + struct mm_struct *mm, unsigned flush); extern void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext); extern void cpu_reset(unsigned long addr) __attribute__((noreturn)); diff --git a/arch/arm/include/asm/fcse.h b/arch/arm/include/asm/fcse.h new file mode 100644 index 0000000..cfdf14a --- /dev/null +++ b/arch/arm/include/asm/fcse.h @@ -0,0 +1,178 @@ +/* + * Filename: arch/arm/include/asm/fcse.h + * Description: ARM Process ID (PID) includes for Fast Address Space Switching + * (FASS) in ARM Linux. + * + * Copyright: (C) 2001, 2002 Adam Wiggins <awiggins@cse.unsw.edu.au> + * (C) 2007 Sebastian Smolorz <ssm@emlix.com> + * (C) 2008 Richard Cochran + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of teh GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#ifndef __ASM_ARM_FCSE_H +#define __ASM_ARM_FCSE_H + +#ifdef CONFIG_ARM_FCSE + +#include <linux/mm_types.h> /* For struct mm_struct */ +#include <linux/sched.h> +#include <linux/hardirq.h> + +#include <asm/bitops.h> +#include <asm/cachetype.h> + +#define FCSE_PID_SHIFT 25 + +/* Size of PID relocation area */ +#define FCSE_PID_TASK_SIZE (1UL << FCSE_PID_SHIFT) + +/* Mask to get rid of PID from relocated address */ +#define FCSE_PID_MASK (FCSE_PID_TASK_SIZE - 1) + +extern unsigned long fcse_pids_cache_dirty[]; + +#ifdef CONFIG_ARM_FCSE_DEBUG +#define FCSE_BUG_ON(expr) BUG_ON(expr) +#else /* !CONFIG_ARM_FCSE_DEBUG */ +#define FCSE_BUG_ON(expr) do { } while(0) +#endif /* !CONFIG_ARM_FCSE_DEBUG */ + +#if defined(CONFIG_ARM_FCSE_DYNPID) && !defined(CONFIG_PREEMPT_NONE) +#define fcse_check_context(mm) \ + FCSE_BUG_ON(!in_atomic() \ + && (mm)->context.fcse.active \ + && atomic_read(&(mm)->mm_users) \ + && !(mm)->core_state \ + && !rwsem_is_locked(&(mm)->mmap_sem) \ + && !irqs_disabled_hw()); +#else /* !CONFIG_ARM_FCSE_DYNPID */ +#define fcse_check_context(mm) do { (void)(mm); } while(0) +#endif /* !CONFIG_ARM_FCSE_DYNPID */ + +int fcse_pid_alloc(struct mm_struct *mm); +void fcse_pid_free(struct mm_struct *mm); +unsigned fcse_flush_all_start(struct mm_struct *mm); +int fcse_flush_all_done(struct mm_struct *mm, unsigned seq, unsigned dirty); +unsigned long +fcse_check_mmap_inner(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl); + +/* Sets the CPU's PID Register */ +static inline void fcse_pid_set(unsigned long pid) +{ + __asm__ __volatile__ ("mcr p15, 0, %0, c13, c0, 0" + : /* */: "r" (pid) : "cc", "memory"); +} + +static inline unsigned long +fcse_va_to_mva(struct mm_struct *mm, unsigned long va) +{ + if (cache_is_vivt() && va < FCSE_PID_TASK_SIZE) { + fcse_check_context(mm); + return mm->context.fcse.pid | va; + } + return va; +} + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +struct fcse_user { + struct mm_struct *mm; + unsigned count; +}; +extern struct fcse_user fcse_pids_user[]; +int fcse_switch_mm_inner(struct mm_struct *prev, struct mm_struct *next); +void fcse_pid_reference(unsigned pid); + +static inline int fcse_switch_mm(struct mm_struct *prev, struct mm_struct *next) +{ + if (!cache_is_vivt()) + return 0; + + return fcse_switch_mm_inner(prev, next); +} + +static inline int fcse_mm_in_cache(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + int res; + fcse_check_context(mm); + res = test_bit(fcse_pid, fcse_pids_cache_dirty) + && fcse_pids_user[fcse_pid].mm == mm; + return res; +} + +static inline unsigned long +fcse_check_mmap_addr(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ + const unsigned long stack_base = ALIGN(mm->start_stack, PAGE_SIZE) + - current->signal->rlim[RLIMIT_STACK].rlim_cur; + + if (addr + len <= stack_base) + return addr; + + return fcse_check_mmap_inner(mm, start_addr, addr, len, fl); +} + +#else /* CONFIG_ARM_FCSE_GUARANTEED */ +static inline int +fcse_switch_mm(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned fcse_pid; + + if (!cache_is_vivt()) + return 0; + + fcse_pid = next->context.fcse.pid >> FCSE_PID_SHIFT; + set_bit(fcse_pid, fcse_pids_cache_dirty); + fcse_pid_set(next->context.fcse.pid); + return 0; +} + +static inline int fcse_mm_in_cache(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + return test_bit(fcse_pid, fcse_pids_cache_dirty); +} + +static inline unsigned long +fcse_check_mmap_addr(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ + if (addr + len <= FCSE_TASK_SIZE) + return addr; + + return fcse_check_mmap_inner(mm, start_addr, addr, len, fl); +} +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ + +static inline void fcse_mark_dirty(struct mm_struct *mm) +{ + if (cache_is_vivt()) { + set_bit(mm->context.fcse.pid >> FCSE_PID_SHIFT, + fcse_pids_cache_dirty); + FCSE_BUG_ON(!fcse_mm_in_cache(mm)); + } +} + +#else /* ! CONFIG_ARM_FCSE */ +#define fcse_va_to_mva(mm, x) ({ (void)(mm); (x); }) +#define fcse_mark_dirty(mm) do { (void)(mm); } while(0) +#define fcse_switch_mm(prev, next) (1) +#define fcse_flush_all_start(mm) (0) +#define fcse_flush_all_done(mm, seq, dirty) (1) +#define fcse_notify_flush_all() do { } while (0) +#define fcse_mm_in_cache(mm) \ + (cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm))) +#endif /* ! CONFIG_ARM_FCSE */ + +#ifdef CONFIG_ARM_FCSE_MESSAGES +void fcse_notify_segv(struct mm_struct *mm, + unsigned long addr, struct pt_regs *regs); +#else /* !FCSE_MESSAGES */ +#define fcse_notify_segv(mm, addr, regs) do { } while(0) +#endif /* !FCSE_MESSAGES */ + +#endif /* __ASM_ARM_FCSE_H */ diff --git a/arch/arm/include/asm/ipipe.h b/arch/arm/include/asm/ipipe.h new file mode 100644 index 0000000..8ae0cb3 --- /dev/null +++ b/arch/arm/include/asm/ipipe.h @@ -0,0 +1,274 @@ +/* -*- linux-c -*- + * arch/arm/include/asm/ipipe.h + * + * Copyright (C) 2002-2005 Philippe Gerum. + * Copyright (C) 2005 Stelian Pop. + * Copyright (C) 2006-2008 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __ARM_IPIPE_H +#define __ARM_IPIPE_H + +#ifdef CONFIG_IPIPE + +#include <linux/ipipe_percpu.h> +#include <mach/irqs.h> /* For __IPIPE_FEATURE_PIC_MUTE */ + +#define IPIPE_ARCH_STRING "1.16-01" +#define IPIPE_MAJOR_NUMBER 1 +#define IPIPE_MINOR_NUMBER 16 +#define IPIPE_PATCH_NUMBER 1 + +#ifdef CONFIG_SMP +#error "I-pipe/arm: SMP not yet implemented" +#define ipipe_processor_id() (current_thread_info()->cpu) +#else /* !CONFIG_SMP */ +#define ipipe_processor_id() 0 +#endif /* CONFIG_SMP */ + +#define smp_processor_id_hw() ipipe_processor_id() + +#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + +#define prepare_arch_switch(next) \ + do { \ + local_irq_enable_hw(); \ + ipipe_schedule_notify(current, next); \ + } while(0) + +#define task_hijacked(p) \ + ({ \ + int x = !ipipe_root_domain_p; \ + clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \ + x; \ + }) + +#define ipipe_mm_switch_protect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#define ipipe_mm_switch_unprotect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + +#define prepare_arch_switch(next) \ + do { \ + ipipe_schedule_notify(current ,next); \ + local_irq_disable_hw(); \ + } while(0) + +#define task_hijacked(p) \ + ({ \ + int x = !ipipe_root_domain_p; \ + __clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \ + if (!x) local_irq_enable_hw(); x; \ + }) + +#define ipipe_mm_switch_protect(flags) \ + local_irq_save_hw_cond(flags) + +#define ipipe_mm_switch_unprotect(flags) \ + local_irq_restore_hw_cond(flags) + +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + +extern unsigned long arm_return_addr(int level); + +#define BROKEN_BUILTIN_RETURN_ADDRESS +#define __BUILTIN_RETURN_ADDRESS0 arm_return_addr(0) +#define __BUILTIN_RETURN_ADDRESS1 arm_return_addr(1) + + +struct ipipe_domain; + +#define IPIPE_TSC_TYPE_NONE 0 +#define IPIPE_TSC_TYPE_FREERUNNING 1 +#define IPIPE_TSC_TYPE_DECREMENTER 2 + +struct __ipipe_tscinfo { + unsigned type; + union { + struct { + unsigned *counter; /* Hw counter physical address */ + unsigned mask; /* Significant bits in the hw counter. */ + unsigned long long *tsc; /* 64 bits tsc value. */ + } fr; + struct { + unsigned *counter; /* Hw counter physical address */ + unsigned mask; /* Significant bits in the hw counter. */ + unsigned *last_cnt; /* Counter value when updating + tsc value. */ + unsigned long long *tsc; /* 64 bits tsc value. */ + } dec; + } u; +}; + +struct ipipe_sysinfo { + + int ncpus; /* Number of CPUs on board */ + u64 cpufreq; /* CPU frequency (in Hz) */ + + /* Arch-dependent block */ + + struct { + unsigned tmirq; /* Timer tick IRQ */ + u64 tmfreq; /* Timer frequency */ + struct __ipipe_tscinfo tsc; /* exported data for u.s. tsc */ + } archdep; +}; + +DECLARE_PER_CPU(struct mm_struct *,ipipe_active_mm); +/* arch specific stuff */ +extern void *__ipipe_tsc_area; +extern int __ipipe_mach_timerint; +extern int __ipipe_mach_timerstolen; +extern unsigned int __ipipe_mach_ticks_per_jiffy; +extern void __ipipe_mach_acktimer(void); +extern unsigned long long __ipipe_mach_get_tsc(void); +extern void __ipipe_mach_set_dec(unsigned long); +extern void __ipipe_mach_release_timer(void); +extern unsigned long __ipipe_mach_get_dec(void); +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info); +int __ipipe_check_tickdev(const char *devname); + +#define ipipe_read_tsc(t) do { t = __ipipe_mach_get_tsc(); } while (0) +#define __ipipe_read_timebase() __ipipe_mach_get_tsc() + +#define ipipe_cpu_freq() (HZ * __ipipe_mach_ticks_per_jiffy) +#define ipipe_tsc2ns(t) \ +({ \ + unsigned long long delta = (t)*1000; \ + do_div(delta, ipipe_cpu_freq() / 1000000 + 1); \ + (unsigned long)delta; \ +}) +#define ipipe_tsc2us(t) \ +({ \ + unsigned long long delta = (t); \ + do_div(delta, ipipe_cpu_freq() / 1000000 + 1); \ + (unsigned long)delta; \ +}) + +#define ipipe_handle_irq_cond(irq) \ + __ipipe_handle_irq(irq, (struct pt_regs *)1) + +/* Private interface -- Internal use only */ + +#define __ipipe_check_platform() do { } while(0) + +#define __ipipe_init_platform() do { } while(0) + +#define __ipipe_enable_irq(irq) irq_desc[irq].chip->enable(irq) + +#define __ipipe_disable_irq(irq) irq_desc[irq].chip->disable(irq) + +#define __ipipe_hook_critical_ipi(ipd) do { } while(0) + +void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq); + +#ifndef __IPIPE_FEATURE_PIC_MUTE +#define __ipipe_disable_irqdesc(ipd, irq) do { } while (0) +#else /* __IPIPE_FEATURE_PIC_MUTE */ + +typedef unsigned long +__ipipe_irqbits_t[(NR_IRQS + BITS_PER_LONG - 1) / BITS_PER_LONG]; +extern __ipipe_irqbits_t __ipipe_irqbits; + +void __ipipe_disable_irqdesc(struct ipipe_domain *ipd, unsigned irq); + +void ipipe_mute_pic(void); + +void ipipe_unmute_pic(void); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +void __ipipe_enable_pipeline(void); + +void __ipipe_do_critical_sync(unsigned irq, + void *cookie); + +DECLARE_PER_CPU(struct pt_regs, __ipipe_tick_regs); + +int __ipipe_handle_irq(int irq, + struct pt_regs *regs); + +#define ipipe_update_tick_evtdev(evtdev) do { } while (0) + +#define __ipipe_tick_irq __ipipe_mach_timerint + +static inline unsigned long __ipipe_ffnz(unsigned long ul) +{ + return ffs(ul) - 1; +} + +/* When running handlers, enable hw interrupts for all domains but the + * one heading the pipeline, so that IRQs can never be significantly + * deferred for the latter. */ +#define __ipipe_run_isr(ipd, irq) \ +do { \ + if (!__ipipe_pipeline_head_p(ipd)) \ + local_irq_enable_hw(); \ + if (ipd == ipipe_root_domain) { \ + if (likely(!ipipe_virtual_irq_p(irq))) \ + ((void (*)(unsigned, struct pt_regs *)) \ + ipd->irqs[irq].handler) (irq, \ + &__raw_get_cpu_var(__ipipe_tick_regs)); \ + else { \ + irq_enter(); \ + ipd->irqs[irq].handler(irq,ipd->irqs[irq].cookie); \ + irq_exit(); \ + } \ + } else { \ + __clear_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \ + ipd->irqs[irq].handler(irq,ipd->irqs[irq].cookie); \ + __set_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \ + } \ + local_irq_disable_hw(); \ +} while(0) + +#define __ipipe_syscall_watched_p(p, sc) \ + (((p)->flags & PF_EVNOTIFY) || (unsigned long)sc >= __ARM_NR_BASE + 64) + +#define __ipipe_root_tick_p(regs) (!raw_irqs_disabled_flags(regs->ARM_cpsr)) + +#else /* !CONFIG_IPIPE */ + +#define task_hijacked(p) 0 + +#define ipipe_update_tick_evtdev(evtdev) do { } while (0) + +#define smp_processor_id_hw() smp_processor_id() + +#define ipipe_handle_irq_cond(irq) \ + generic_handle_irq(irq) + +#define ipipe_mm_switch_protect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#define ipipe_mm_switch_unprotect(flags) \ + do { \ + (void) (flags); \ + } while (0) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__ARM_IPIPE_H */ diff --git a/arch/arm/include/asm/ipipe_base.h b/arch/arm/include/asm/ipipe_base.h new file mode 100644 index 0000000..20e97a4 --- /dev/null +++ b/arch/arm/include/asm/ipipe_base.h @@ -0,0 +1,108 @@ +/* -*- linux-c -*- + * arch/arm/include/asm/ipipe_base.h + * + * Copyright (C) 2007 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __ARM_IPIPE_BASE_H +#define __ARM_IPIPE_BASE_H + +#include <linux/threads.h> +#include <asm/ptrace.h> +#include <asm/irq.h> + +#define IPIPE_NR_XIRQS NR_IRQS +#define IPIPE_IRQ_ISHIFT 5 /* 25 for 32bits arch. */ + +/* ARM traps */ +#define IPIPE_TRAP_ACCESS 0 /* Data or instruction access exception */ +#define IPIPE_TRAP_SECTION 1 /* Section fault */ +#define IPIPE_TRAP_DABT 2 /* Generic data abort */ +#define IPIPE_TRAP_UNKNOWN 3 /* Unknown exception */ +#define IPIPE_TRAP_BREAK 4 /* Instruction breakpoint */ +#define IPIPE_TRAP_FPU 5 /* Floating point exception */ +#define IPIPE_TRAP_VFP 6 /* VFP floating point exception */ +#define IPIPE_TRAP_UNDEFINSTR 7 /* Undefined instruction */ +#define IPIPE_TRAP_ALIGNMENT 8 /* Unaligned access exception */ +#define IPIPE_NR_FAULTS 9 + +/* Pseudo-vectors used for kernel events */ +#define IPIPE_FIRST_EVENT IPIPE_NR_FAULTS +#define IPIPE_EVENT_SYSCALL (IPIPE_FIRST_EVENT) +#define IPIPE_EVENT_SCHEDULE (IPIPE_FIRST_EVENT + 1) +#define IPIPE_EVENT_SIGWAKE (IPIPE_FIRST_EVENT + 2) +#define IPIPE_EVENT_SETSCHED (IPIPE_FIRST_EVENT + 3) +#define IPIPE_EVENT_INIT (IPIPE_FIRST_EVENT + 4) +#define IPIPE_EVENT_EXIT (IPIPE_FIRST_EVENT + 5) +#define IPIPE_EVENT_CLEANUP (IPIPE_FIRST_EVENT + 6) +#define IPIPE_LAST_EVENT IPIPE_EVENT_CLEANUP +#define IPIPE_NR_EVENTS (IPIPE_LAST_EVENT + 1) + +#ifndef __ASSEMBLY__ + +#include <asm/irqflags.h> + +#ifdef CONFIG_SMP +#error "SMP not implemented." +#define __ipipe_root_status ipipe_root_cpudom_var(status) + +#else /* !CONFIG_SMP */ + +#ifdef CONFIG_VFP +#define __IPIPE_FEATURE_VFP_SAFE 1 +#endif + +#if __GNUC__ >= 4 +/* Alias to ipipe_root_cpudom_var(status) */ +extern unsigned long __ipipe_root_status; +#else +extern unsigned long *const __ipipe_root_status_addr; +#define __ipipe_root_status (*__ipipe_root_status_addr) +#endif + +static inline void __ipipe_stall_root(void) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_root_status |= 1; + local_irq_restore_hw(flags); +} + +static inline unsigned __ipipe_test_root(void) +{ + return __ipipe_root_status & 1; +} + +static inline unsigned __ipipe_test_and_stall_root(void) +{ + unsigned long flags, res; + + local_irq_save_hw(flags); + res = __ipipe_root_status; + __ipipe_root_status = res | 1; + local_irq_restore_hw(flags); + + return res & 1; +} + +#endif /* CONFIG_SMP */ + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ARM_IPIPE_BASE_H */ diff --git a/arch/arm/include/asm/irqflags.h b/arch/arm/include/asm/irqflags.h index 6d09974..0aa4056 100644 --- a/arch/arm/include/asm/irqflags.h +++ b/arch/arm/include/asm/irqflags.h @@ -10,30 +10,30 @@ */ #if __LINUX_ARM_ARCH__ >= 6 -#define raw_local_irq_save(x) \ +#define local_irq_save_hw_notrace(x) \ ({ \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_save\n" \ + "mrs %0, cpsr @ local_irq_save_hw\n" \ "cpsid i" \ : "=r" (x) : : "memory", "cc"); \ }) -#define raw_local_irq_enable() __asm__("cpsie i @ __sti" : : : "memory", "cc") -#define raw_local_irq_disable() __asm__("cpsid i @ __cli" : : : "memory", "cc") -#define local_fiq_enable() __asm__("cpsie f @ __stf" : : : "memory", "cc") -#define local_fiq_disable() __asm__("cpsid f @ __clf" : : : "memory", "cc") +#define local_irq_enable_hw_notrace() __asm__("cpsie i @ __sti" : : : "memory", "cc") +#define local_irq_disable_hw_notrace() __asm__("cpsid i @ __cli" : : : "memory", "cc") +#define local_fiq_enable_hw_notrace() __asm__("cpsie f @ __stf" : : : "memory", "cc") +#define local_fiq_disable_hw_notrace() __asm__("cpsid f @ __clf" : : : "memory", "cc") #else /* * Save the current interrupt enable state & disable IRQs */ -#define raw_local_irq_save(x) \ +#define local_irq_save_hw_notrace(x) \ ({ \ unsigned long temp; \ (void) (&temp == &x); \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_save\n" \ + "mrs %0, cpsr @ local_irq_save_hw\n" \ " orr %1, %0, #128\n" \ " msr cpsr_c, %1" \ : "=r" (x), "=r" (temp) \ @@ -44,11 +44,11 @@ /* * Enable IRQs */ -#define raw_local_irq_enable() \ +#define local_irq_enable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_enable\n" \ + "mrs %0, cpsr @ local_irq_enable_hw\n" \ " bic %0, %0, #128\n" \ " msr cpsr_c, %0" \ : "=r" (temp) \ @@ -59,11 +59,11 @@ /* * Disable IRQs */ -#define raw_local_irq_disable() \ +#define local_irq_disable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_irq_disable\n" \ + "mrs %0, cpsr @ local_irq_disable_hw\n" \ " orr %0, %0, #128\n" \ " msr cpsr_c, %0" \ : "=r" (temp) \ @@ -74,7 +74,7 @@ /* * Enable FIQs */ -#define local_fiq_enable() \ +#define local_fiq_enable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ @@ -89,7 +89,7 @@ /* * Disable FIQs */ -#define local_fiq_disable() \ +#define local_fiq_disable_hw_notrace() \ ({ \ unsigned long temp; \ __asm__ __volatile__( \ @@ -106,19 +106,19 @@ /* * Save the current interrupt enable state. */ -#define raw_local_save_flags(x) \ +#define local_save_flags_hw(x) \ ({ \ __asm__ __volatile__( \ - "mrs %0, cpsr @ local_save_flags" \ + "mrs %0, cpsr @ local_save_flags_hw" \ : "=r" (x) : : "memory", "cc"); \ }) /* * restore saved IRQ & FIQ state */ -#define raw_local_irq_restore(x) \ +#define local_irq_restore_hw_notrace(x) \ __asm__ __volatile__( \ - "msr cpsr_c, %0 @ local_irq_restore\n" \ + "msr cpsr_c, %0 @ local_irq_restore_hw\n" \ : \ : "r" (x) \ : "memory", "cc") @@ -128,5 +128,99 @@ (int)((flags) & PSR_I_BIT); \ }) +#define irqs_disabled_hw() \ +({ \ + unsigned long flags; \ + local_save_flags_hw(flags); \ + raw_irqs_disabled_flags(flags); \ +}) + +static inline unsigned long raw_mangle_irq_bits(int virt, unsigned long real) +{ + /* Merge virtual and real interrupt mask bits into a single + 32bit word. */ + return (real & ~(1L << 8)) | ((virt != 0) << 8); +} + +static inline int raw_demangle_irq_bits(unsigned long *x) +{ + int virt = (*x & (1 << 8)) != 0; + *x &= ~(1L << 8); + return virt; +} + +#ifdef CONFIG_IPIPE + +void __ipipe_unstall_root(void); +void __ipipe_restore_root(unsigned long flags); + +/* PSR_I_BIT is bit no. 7 and is set if interrupts are _disabled_ */ +#define raw_local_irq_save(flags) ((flags) = __ipipe_test_and_stall_root() << 7) +#define raw_local_irq_enable() __ipipe_unstall_root() +#define raw_local_irq_disable() __ipipe_stall_root() +#define local_fiq_enable() __ipipe_unstall_root() +#define local_fiq_disable() __ipipe_stall_root() +#define raw_local_save_flags(flags) ((flags) = __ipipe_test_root() << 7) +#define raw_local_irq_restore(flags) __ipipe_restore_root(flags & (1 << 7)) + +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF + +#include <linux/ipipe_trace.h> + +#define local_irq_disable_hw() do { \ + if (!irqs_disabled_hw()) { \ + local_irq_disable_hw_notrace(); \ + ipipe_trace_begin(0x80000000); \ + } \ +} while (0) +#define local_irq_enable_hw() do { \ + if (irqs_disabled_hw()) { \ + ipipe_trace_end(0x80000000); \ + local_irq_enable_hw_notrace(); \ + } \ +} while (0) +#define local_irq_save_hw(x) do { \ + local_save_flags_hw(x); \ + if (!raw_irqs_disabled_flags(x)) { \ + local_irq_disable_hw_notrace(); \ + ipipe_trace_begin(0x80000001); \ + } \ +} while (0) +#define local_irq_restore_hw(x) do { \ + if (!raw_irqs_disabled_flags(x)) \ + ipipe_trace_end(0x80000001); \ + local_irq_restore_hw_notrace(x); \ +} while (0) + +#else /* !CONFIG_IPIPE_TRACE_IRQSOFF */ + +#define local_irq_save_hw(flags) local_irq_save_hw_notrace(flags) +#define local_irq_enable_hw() local_irq_enable_hw_notrace() +#define local_irq_disable_hw() local_irq_disable_hw_notrace() +#define local_fiq_enable_hw() local_fiq_enable_hw_notrace() +#define local_fiq_disable_hw() local_fiq_disable_hw_notrace() +#define local_irq_restore_hw(flags) local_irq_restore_hw_notrace(flags) + +#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */ + +#else /* !CONFIG_IPIPE */ + +#define raw_local_irq_save(flags) local_irq_save_hw_notrace(flags) +#define raw_local_irq_enable() local_irq_enable_hw_notrace() +#define raw_local_irq_disable() local_irq_disable_hw_notrace() +#define local_fiq_enable() local_fiq_enable_hw_notrace() +#define local_fiq_disable() local_fiq_disable_hw_notrace() +#define raw_local_save_flags(flags) local_save_flags_hw(flags) +#define raw_local_irq_restore(flags) local_irq_restore_hw_notrace(flags) + +#define local_irq_save_hw(flags) local_irq_save_hw_notrace(flags) +#define local_irq_enable_hw() local_irq_enable_hw_notrace() +#define local_irq_disable_hw() local_irq_disable_hw_notrace() +#define local_fiq_enable_hw() local_fiq_enable_hw_notrace() +#define local_fiq_disable_hw() local_fiq_disable_hw_notrace() +#define local_irq_restore_hw(flags) local_irq_restore_hw_notrace(flags) + +#endif /* CONFIG_IPIPE */ + #endif #endif diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h index 5421d82..578d189 100644 --- a/arch/arm/include/asm/memory.h +++ b/arch/arm/include/asm/memory.h @@ -33,7 +33,12 @@ */ #define PAGE_OFFSET UL(CONFIG_PAGE_OFFSET) #define TASK_SIZE (UL(CONFIG_PAGE_OFFSET) - UL(0x01000000)) +#ifndef CONFIG_ARM_FCSE #define TASK_UNMAPPED_BASE (UL(CONFIG_PAGE_OFFSET) / 3) +#else /* CONFIG_ARM_FCSE */ +#define TASK_UNMAPPED_BASE UL(0x00800000) +#define FCSE_TASK_SIZE UL(0x02000000) +#endif /* CONFIG_ARM_FCSE */ /* * The maximum size of a 26-bit user space task. diff --git a/arch/arm/include/asm/mmu.h b/arch/arm/include/asm/mmu.h index b561584..4087d0b 100644 --- a/arch/arm/include/asm/mmu.h +++ b/arch/arm/include/asm/mmu.h @@ -7,6 +7,17 @@ typedef struct { #ifdef CONFIG_CPU_HAS_ASID unsigned int id; #endif +#ifdef CONFIG_ARM_FCSE + struct { + unsigned long pid; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned shared_dirty_pages; + unsigned active : 1; + unsigned large : 1; + unsigned high_pages; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + } fcse; +#endif /* CONFIG_ARM_FCSE */ unsigned int kvm_seq; } mm_context_t; diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index de6cefb..edb8922 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -19,6 +19,7 @@ #include <asm/cachetype.h> #include <asm/proc-fns.h> #include <asm-generic/mm_hooks.h> +#include <asm/fcse.h> void __check_kvm_seq(struct mm_struct *mm); @@ -68,11 +69,50 @@ static inline void check_context(struct mm_struct *mm) #endif } -#define init_new_context(tsk,mm) 0 +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE + int fcse_pid; + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (!mm->context.fcse.large) { + fcse_pid = fcse_pid_alloc(mm); + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; + } else { + /* We are normally forking a process vith a virtual address + space larger than 32 MB, so its pid should be 0. */ + FCSE_BUG_ON(mm->context.fcse.pid); + fcse_pid_reference(0); + } + /* If we are forking, set_pte_at will restore the correct high pages + count, and shared writable pages are write-protected again. */ + mm->context.fcse.shared_dirty_pages = 0; + mm->context.fcse.high_pages = 0; + mm->context.fcse.active = 0; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + fcse_pid = fcse_pid_alloc(mm); + if (fcse_pid < 0) + return fcse_pid; + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#endif /* CONFIG_ARM_FCSE */ + + return 0; +} #endif -#define destroy_context(mm) do { } while(0) +static inline void destroy_context(struct mm_struct *mm) +{ +#ifdef CONFIG_ARM_FCSE +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + FCSE_BUG_ON(mm->context.fcse.shared_dirty_pages); + FCSE_BUG_ON(mm->context.fcse.high_pages); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + fcse_pid_free(mm); +#endif /* CONFIG_ARM_FCSE */ +} /* * This is called when "tsk" is about to enter lazy TLB mode. @@ -95,11 +135,11 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) * actually changed. */ static inline void -switch_mm(struct mm_struct *prev, struct mm_struct *next, - struct task_struct *tsk) +__switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *tsk) { #ifdef CONFIG_MMU - unsigned int cpu = smp_processor_id(); + unsigned int cpu = smp_processor_id_hw(); #ifdef CONFIG_SMP /* check for possible thread migration */ @@ -109,14 +149,64 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, #endif if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) { check_context(next); - cpu_switch_mm(next->pgd, next); - if (cache_is_vivt()) +#if defined(CONFIG_IPIPE) + if (ipipe_root_domain_p) { + /* mark mm state as undefined. */ + per_cpu(ipipe_active_mm, cpu) = NULL; + barrier(); + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(prev, next)); + barrier(); + per_cpu(ipipe_active_mm, cpu) = next; + while (test_and_clear_thread_flag(TIF_MMSWITCH_INT)) { + /* mark mm state as undefined. */ + per_cpu(ipipe_active_mm, cpu) = NULL; + barrier(); + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(NULL, next)); + barrier(); + per_cpu(ipipe_active_mm, cpu) = next; + } + } else +#endif /* CONFIG_IPIPE */ + cpu_switch_mm(next->pgd, next, + fcse_switch_mm(prev, next)); +#if defined(CONFIG_IPIPE) && defined(CONFIG_ARM_FCSE) + if (tsk) + set_tsk_thread_flag(tsk, TIF_SWITCHED); +#endif /* CONFIG_IPIPE && CONFIG_ARM_FCSE */ + if (cache_is_vivt() && prev) cpumask_clear_cpu(cpu, mm_cpumask(prev)); - } + } else + fcse_mark_dirty(next); #endif } +static inline void +switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *tsk) +{ +#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + unsigned long flags; + local_irq_save_hw(flags); +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ + __switch_mm(prev, next, tsk); +#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH + local_irq_restore_hw(flags); +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ +} + #define deactivate_mm(tsk,mm) do { } while (0) + +#ifndef CONFIG_ARM_FCSE_BEST_EFFORT #define activate_mm(prev,next) switch_mm(prev, next, NULL) +#else +#define activate_mm(prev,next) \ + ({ \ + switch_mm(prev, next, NULL); \ + next->context.fcse.active = 1; \ + FCSE_BUG_ON(current->mm == next && !fcse_mm_in_cache(next)); \ + }) +#endif #endif diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 1139768..3d71ffb 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -112,6 +112,8 @@ #define LIBRARY_TEXT_START 0x0c000000 #ifndef __ASSEMBLY__ +#include <asm/fcse.h> + extern void __pte_error(const char *file, int line, unsigned long val); extern void __pmd_error(const char *file, int line, unsigned long val); extern void __pgd_error(const char *file, int line, unsigned long val); @@ -248,6 +250,40 @@ extern pgprot_t pgprot_kernel; #define __S111 __PAGE_SHARED_EXEC #ifndef __ASSEMBLY__ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#define fcse_account_page_removal(mm, addr, val) do { \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && ((_val) & L_PTE_SHARED)) \ + --_mm->context.fcse.shared_dirty_pages; \ + if (pte_present(_val) && _addr < TASK_SIZE) { \ + if (_addr >= FCSE_TASK_SIZE) \ + --_mm->context.fcse.high_pages; \ + } \ +} while (0) + +#define fcse_account_page_addition(mm, addr, val) ({ \ + struct mm_struct *_mm = (mm); \ + unsigned long _addr = (addr); \ + unsigned long _val = (val); \ + if (pte_present(_val) && (_val & L_PTE_SHARED)) { \ + if ((_val & (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + != (PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) \ + _val &= ~L_PTE_SHARED; \ + else \ + ++_mm->context.fcse.shared_dirty_pages; \ + } \ + if (pte_present(_val) \ + && _addr < TASK_SIZE && _addr >= FCSE_TASK_SIZE) \ + ++_mm->context.fcse.high_pages; \ + _val; \ +}) +#else /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ +#define fcse_account_page_removal(mm, addr, val) do { } while (0) +#define fcse_account_page_addition(mm, addr, val) (val) +#endif /* CONFIG_ARM_FCSE_GUARANTEED || !CONFIG_ARM_FCSE */ + /* * ZERO_PAGE is a global shared page that is always zero: used * for zero-mapped memory areas etc.. @@ -259,7 +295,10 @@ extern struct page *empty_zero_page; #define pfn_pte(pfn,prot) (__pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot))) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte_ext(ptep, __pte(0), 0) +#define pte_clear(mm,addr,ptep) do { \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + set_pte_ext(ptep, __pte(0), 0); \ +} while (0) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) #define pte_offset_kernel(dir,addr) (pmd_page_vaddr(*(dir)) + __pte_index(addr)) @@ -278,9 +317,13 @@ extern struct page *empty_zero_page; #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext) -#define set_pte_at(mm,addr,ptep,pteval) do { \ - set_pte_ext(ptep, pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ - } while (0) +#define set_pte_at(mm,addr,ptep,pteval) do { \ + unsigned long _pteval = (pteval); \ + fcse_account_page_removal(mm, addr, pte_val(*ptep)); \ + pte_val(_pteval) = \ + fcse_account_page_addition(mm, addr, pte_val(_pteval)); \ + set_pte_ext(ptep, _pteval, (addr) >= TASK_SIZE ? 0 : PTE_EXT_NG); \ +} while (0) /* * The following only work if pte_present() is true. @@ -372,10 +415,14 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd) /* to find an entry in a page-table-directory */ #define pgd_index(addr) ((addr) >> PGDIR_SHIFT) -#define pgd_offset(mm, addr) ((mm)->pgd+pgd_index(addr)) +#define pgd_offset(mm, addr) \ + ({ \ + struct mm_struct *_mm = (mm); \ + (_mm->pgd + pgd_index(fcse_va_to_mva(_mm, (addr)))); \ + }) /* to find an entry in a kernel page-table-directory */ -#define pgd_offset_k(addr) pgd_offset(&init_mm, addr) +#define pgd_offset_k(addr) (init_mm.pgd + pgd_index(addr)) /* Find an entry in the second-level page table.. */ #define pmd_offset(dir, addr) ((pmd_t *)(dir)) diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h index 8fdae9b..0c280ca 100644 --- a/arch/arm/include/asm/proc-fns.h +++ b/arch/arm/include/asm/proc-fns.h @@ -261,12 +261,13 @@ #ifdef CONFIG_MMU -#define cpu_switch_mm(pgd,mm) cpu_do_switch_mm(virt_to_phys(pgd),mm) +#define cpu_switch_mm(pgd,mm,flush) \ + cpu_do_switch_mm(virt_to_phys(pgd), (mm), (flush)) #define cpu_get_pgd() \ ({ \ unsigned long pg; \ - __asm__("mrc p15, 0, %0, c2, c0, 0" \ + __asm__ __volatile__ ("mrc p15, 0, %0, c2, c0, 0" \ : "=r" (pg) : : "cc"); \ pg &= ~0x3fff; \ (pgd_t *)phys_to_virt(pg); \ diff --git a/arch/arm/include/asm/processor.h b/arch/arm/include/asm/processor.h index 6a89567..bc33df1 100644 --- a/arch/arm/include/asm/processor.h +++ b/arch/arm/include/asm/processor.h @@ -23,9 +23,14 @@ #include <asm/types.h> #ifdef __KERNEL__ +#ifndef CONFIG_ARM_FCSE #define STACK_TOP ((current->personality & ADDR_LIMIT_32BIT) ? \ TASK_SIZE : TASK_SIZE_26) #define STACK_TOP_MAX TASK_SIZE +#else /* CONFIG_ARM_FCSE */ +#define STACK_TOP FCSE_TASK_SIZE +#define STACK_TOP_MAX FCSE_TASK_SIZE +#endif /* CONFIG_ARM_FCSE */ #endif union debug_insn { diff --git a/arch/arm/include/asm/resource.h b/arch/arm/include/asm/resource.h index 734b581..5ba5042 100644 --- a/arch/arm/include/asm/resource.h +++ b/arch/arm/include/asm/resource.h @@ -1,6 +1,16 @@ #ifndef _ARM_RESOURCE_H #define _ARM_RESOURCE_H +/* + * When FCSE is enabled, reduce the default stack size to 1MB, and maximum + * to 16MB, the address space is only 32MB. + */ +#ifdef CONFIG_ARM_FCSE +#define _STK_LIM (1024*1024) + +#define _STK_LIM_MAX (16*1024*1024) +#endif /* CONFIG_ARM_FCSE */ + #include <asm-generic/resource.h> #endif diff --git a/arch/arm/include/asm/system.h b/arch/arm/include/asm/system.h index 058e7e9..6ba023d 100644 --- a/arch/arm/include/asm/system.h +++ b/arch/arm/include/asm/system.h @@ -218,10 +218,19 @@ static inline void set_copro_access(unsigned int val) */ extern struct task_struct *__switch_to(struct task_struct *, struct thread_info *, struct thread_info *); +#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH #define switch_to(prev,next,last) \ do { \ - last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ + local_irq_disable_hw_cond(); \ + last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ + local_irq_enable_hw_cond(); \ } while (0) +#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ +#define switch_to(prev,next,last) \ +do { \ + last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ +} while (0) +#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */ #if defined(CONFIG_CPU_SA1100) || defined(CONFIG_CPU_SA110) /* @@ -282,17 +291,17 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int size #error SMP is not supported on this platform #endif case 1: - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = *(volatile unsigned char *)ptr; *(volatile unsigned char *)ptr = x; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); break; case 4: - raw_local_irq_save(flags); + local_irq_save_hw(flags); ret = *(volatile unsigned long *)ptr; *(volatile unsigned long *)ptr = x; - raw_local_irq_restore(flags); + local_irq_restore_hw(flags); break; #else case 1: diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 2dfb7d7..e20256d 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -143,6 +143,12 @@ extern void vfp_sync_state(struct thread_info *thread); #define TIF_MEMDIE 18 #define TIF_FREEZE 19 #define TIF_RESTORE_SIGMASK 20 +#ifdef CONFIG_IPIPE +#define TIF_MMSWITCH_INT 21 +#ifdef CONFIG_ARM_FCSE +#define TIF_SWITCHED 22 +#endif /* CONFIG_ARM_FCSE */ +#endif /* CONFIG_IPIPE */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -152,6 +158,12 @@ extern void vfp_sync_state(struct thread_info *thread); #define _TIF_USING_IWMMXT (1 << TIF_USING_IWMMXT) #define _TIF_FREEZE (1 << TIF_FREEZE) #define _TIF_RESTORE_SIGMASK (1 << TIF_RESTORE_SIGMASK) +#ifdef CONFIG_IPIPE +#define _TIF_MMSWITCH_INT (1 << TIF_MMSWITCH_INT) +#ifdef CONFIG_ARM_FCSE +#define _TIF_SWITCHED (1 << TIF_SWITCHED) +#endif /* CONFIG_ARM_FCSE */ +#endif /* CONFIG_IPIPE */ /* * Change these and you break ASM code in entry-common.S diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h index c2f1605..70a81b1 100644 --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -210,6 +210,7 @@ #ifndef __ASSEMBLY__ #include <linux/sched.h> +#include <asm/fcse.h> struct cpu_tlb_fns { void (*flush_user_range)(unsigned long, unsigned long, struct vm_area_struct *); @@ -384,7 +385,8 @@ local_flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) const int zero = 0; const unsigned int __tlb_flag = __cpu_tlb_flags; - uaddr = (uaddr & PAGE_MASK) | ASID(vma->vm_mm); + uaddr = (fcse_va_to_mva(vma->vm_mm, uaddr) & PAGE_MASK) + | ASID(vma->vm_mm); if (tlb_flag(TLB_WB)) dsb(); @@ -505,7 +507,15 @@ static inline void clean_pmd_entry(pmd_t *pmd) /* * Convert calls to our calling convention. */ -#define local_flush_tlb_range(vma,start,end) __cpu_flush_user_tlb_range(start,end,vma) +#define local_flush_tlb_range(vma, start, end) \ + ({ \ + struct mm_struct *_mm = (vma)->vm_mm; \ + unsigned long _start, _end; \ + _start = fcse_va_to_mva(_mm, start); \ + _end = fcse_va_to_mva(_mm, end); \ + __cpu_flush_user_tlb_range(_start, _end, vma); \ + }) + #define local_flush_tlb_kernel_range(s,e) __cpu_flush_kern_tlb_range(s,e) #ifndef CONFIG_SMP diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile index dd00f74..ca557ee 100644 --- a/arch/arm/kernel/Makefile +++ b/arch/arm/kernel/Makefile @@ -20,7 +20,7 @@ obj-y := compat.o elf.o entry-armv.o entry-common.o irq.o \ obj-$(CONFIG_OC_ETM) += etm.o obj-$(CONFIG_ISA_DMA_API) += dma.o -obj-$(CONFIG_ARCH_ACORN) += ecard.o +obj-$(CONFIG_ARCH_ACORN) += ecard.o obj-$(CONFIG_FIQ) += fiq.o obj-$(CONFIG_MODULES) += armksyms.o module.o obj-$(CONFIG_ARTHUR) += arthur.o @@ -55,5 +55,7 @@ endif head-y := head$(MMUEXT).o obj-$(CONFIG_DEBUG_LL) += debug.o obj-$(CONFIG_EARLY_PRINTK) += early_printk.o +obj-$(CONFIG_ARM_FCSE) += fcse.o +obj-$(CONFIG_IPIPE) += ipipe.o extra-y := $(head-y) init_task.o vmlinux.lds diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 6c5cf36..f485f8c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -4,6 +4,7 @@ * Copyright (C) 1996,1997,1998 Russell King. * ARM700 fix by Matthew Godbolt (linux-user@willothewisp.demon.co.uk) * nommu support by Hyok S. Choi (hyok.choi@samsung.com) + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -29,14 +30,22 @@ * Interrupt handling. Preserves r7, r8, r9 */ .macro irq_handler +#ifdef CONFIG_IPIPE + mov r0, #2 +#endif get_irqnr_preamble r5, lr -1: get_irqnr_and_base r0, r6, r5, lr +1: get_irqnr_and_base r1, r6, r5, lr + movne r0, r1 movne r1, sp @ @ routine called with r0 = irq number, r1 = struct pt_regs * @ adrne lr, BSYM(1b) +#ifdef CONFIG_IPIPE + bne __ipipe_grab_irq +#else bne asm_do_IRQ +#endif #ifdef CONFIG_SMP /* @@ -45,18 +54,22 @@ * this macro assumes that irqstat (r6) and base (r5) are * preserved from get_irqnr_and_base above */ - test_for_ipi r0, r6, r5, lr + test_for_ipi r1, r6, r5, lr movne r0, sp adrne lr, BSYM(1b) bne do_IPI #ifdef CONFIG_LOCAL_TIMERS - test_for_ltirq r0, r6, r5, lr + test_for_ltirq r1, r6, r5, lr movne r0, sp adrne lr, BSYM(1b) bne do_local_timer #endif #endif +#ifdef CONFIG_IPIPE + cmp r0, #2 + bleq __ipipe_check_root_interruptible +#endif .endm @@ -78,6 +91,21 @@ mov r1, #\reason .endm + .macro fcse_dabt_mva_to_va +#ifdef CONFIG_ARM_FCSE + @ + @ If FCSE is enabled the data abort fault address must be + @ converted from MVA to VA. This must happen before the irqs are + @ enabled if PREEMPT is enabled, as context switches may + @ change the FCSE pid. + @ + mrc p15, 0, r2, c13, c0, 0 + eor r2, r2, r0 + tst r2, #0xfe000000 + moveq r0, r2 +#endif + .endm + __pabt_invalid: inv_entry BAD_PREFETCH b common_invalid @@ -193,6 +221,7 @@ __dabt_svc: #else bl CPU_DABORT_HANDLER #endif + fcse_dabt_mva_to_va @ @ set desired IRQ state, then call main handler @@ -220,14 +249,26 @@ __irq_svc: #ifdef CONFIG_PREEMPT get_thread_info tsk +#ifndef CONFIG_IPIPE ldr r8, [tsk, #TI_PREEMPT] @ get preempt count add r7, r8, #1 @ increment it str r7, [tsk, #TI_PREEMPT] #endif +#endif irq_handler +#ifdef CONFIG_IPIPE + cmp r0, #0 + ldreq r4, [sp, #S_PSR] @ irqs are already disabled + beq __ipipe_fast_svc_irq_exit +#endif + #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE str r8, [tsk, #TI_PREEMPT] @ restore preempt count +#else + ldr r8, [tsk, #TI_PREEMPT] +#endif ldr r0, [tsk, #TI_FLAGS] @ get flags teq r8, #0 @ if preempt count != 0 movne r0, #0 @ force flags to 0 @@ -239,6 +280,9 @@ __irq_svc: tst r4, #PSR_I_BIT bleq trace_hardirqs_on #endif +#ifdef CONFIG_IPIPE +__ipipe_fast_svc_irq_exit: +#endif svc_exit r4 @ return from exception UNWIND(.fnend ) ENDPROC(__irq_svc) @@ -248,12 +292,16 @@ ENDPROC(__irq_svc) #ifdef CONFIG_PREEMPT svc_preempt: mov r8, lr +#ifndef CONFIG_IPIPE 1: bl preempt_schedule_irq @ irq en/disable is done inside +#else /* CONFIG_IPIPE */ +1: bl __ipipe_preempt_schedule_irq @ irq en/disable is done inside +#endif /* CONFIG_IPIPE */ ldr r0, [tsk, #TI_FLAGS] @ get new tasks TI_FLAGS tst r0, #_TIF_NEED_RESCHED moveq pc, r8 @ go again b 1b -#endif +#endif /* CONFIG_PREEMPT */ .align 5 __und_svc: @@ -266,6 +314,17 @@ __und_svc: svc_entry #endif +#ifdef CONFIG_IPIPE + /* Beware, do_undefinstr in arch/arm/traps.c will not signal the trap + when in svc mode because of this hunk. */ + mov r4, r2 + mov r0, #7 @ r0 = IPIPE_TRAP_UNDEFINSTR + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + bne 1f + mov r2, r4 +#endif /* CONFIG_IPIPE */ @ @ call emulation code, which returns using r9 if it has emulated @ the instruction, or the more conventional lr if we are to treat @@ -433,6 +492,7 @@ __dabt_usr: #else bl CPU_DABORT_HANDLER #endif + fcse_dabt_mva_to_va @ @ IRQs on, then call the main handler @@ -451,13 +511,22 @@ __irq_usr: get_thread_info tsk #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE ldr r8, [tsk, #TI_PREEMPT] @ get preempt count add r7, r8, #1 @ increment it str r7, [tsk, #TI_PREEMPT] #endif +#endif irq_handler +#ifdef CONFIG_IPIPE + cmp r0, #0 + bne __ipipe_usr_irq_continue + slow_restore_user_regs @ Fast exit path over non-root domains +__ipipe_usr_irq_continue: +#endif #ifdef CONFIG_PREEMPT +#ifndef CONFIG_IPIPE ldr r0, [tsk, #TI_PREEMPT] str r8, [tsk, #TI_PREEMPT] teq r0, r7 @@ -465,6 +534,7 @@ __irq_usr: THUMB( movne r0, #0 ) THUMB( strne r0, [r0] ) #endif +#endif #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_on #endif @@ -580,7 +650,7 @@ call_fpe: mov r7, #1 strb r7, [r10, #TI_USED_CP + 10] @ mark CP#10 as used strb r7, [r10, #TI_USED_CP + 11] @ mark CP#11 as used - b do_vfp @ let VFP handler handle this + b _do_vfp @ let VFP handler handle this 1: #endif tst r0, #0x08000000 @ only CDP/CPRT/LDC/STC have bit 27 @@ -626,8 +696,8 @@ call_fpe: movw_pc lr @ CP#8 movw_pc lr @ CP#9 #ifdef CONFIG_VFP - W(b) do_vfp @ CP#10 (VFP) - W(b) do_vfp @ CP#11 (VFP) + W(b) _do_vfp @ CP#10 (VFP) + W(b) _do_vfp @ CP#11 (VFP) #else movw_pc lr @ CP#10 (VFP) movw_pc lr @ CP#11 (VFP) @@ -662,11 +732,43 @@ call_fpe: #endif do_fpe: +#ifdef CONFIG_IPIPE + mov r4, r0 + mov r5, r2 + mov r6, lr + mov r0, #5 @ r0 = IPIPE_TRAP_FPU + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + movne pc, r9 + mov lr, r6 + mov r2, r5 + mov r0, r4 +#endif enable_irq ldr r4, .LCfp add r10, r10, #TI_FPSTATE @ r10 = workspace ldr pc, [r4] @ Call FP module USR entry point +#ifdef CONFIG_VFP +_do_vfp: +#ifdef CONFIG_IPIPE + mov r4, r0 + mov r5, r2 + mov r6, lr + mov r0, #6 @ r0 = IPIPE_TRAP_VFP + mov r1, sp @ r1 = ®s + bl __ipipe_dispatch_event @ branch to trap handler + cmp r0, #0 + movne pc, r9 + mov lr, r6 + mov r2, r5 + mov r0, r4 +#endif + b do_vfp +#endif + + /* * The FP module is called with these registers set: * r0 = instruction @@ -715,6 +817,13 @@ __pabt_usr: ENTRY(ret_from_exception) UNWIND(.fnstart ) UNWIND(.cantunwind ) +#ifdef CONFIG_IPIPE + bl __ipipe_check_root + cmp r0, #0 + bne 1f + slow_restore_user_regs @ Fast exit path over non-root domains +1: +#endif /* CONFIG_IPIPE */ get_thread_info tsk mov why, #0 b ret_to_user @@ -752,7 +861,11 @@ ENTRY(__switch_to) add r4, r2, #TI_CPU_SAVE ldr r0, =thread_notify_head mov r1, #THREAD_NOTIFY_SWITCH +#ifndef CONFIG_IPIPE bl atomic_notifier_call_chain +#else /* !CONFIG_IPIPE */ + bl __ipipe_switch_to_notifier_call_chain +#endif /* !CONFIG_IPIPE */ THUMB( mov ip, r4 ) mov r0, r5 ARM( ldmia r4, {r4 - sl, fp, sp, pc} ) @ Load all regs saved previously @@ -795,6 +908,18 @@ ENDPROC(__switch_to) */ THUMB( .arm ) +#ifdef CONFIG_IPIPE +/* We use the same mechanism as Linux user helpers to store variables related to + TSC emulation, so that they can be used in user-space. */ + .align 5 + .globl __ipipe_tsc_area_start + +__ipipe_tsc_area_start: + .rep 12 + .word 0 + .endr +#endif /* CONFIG_IPIPE */ + .macro usr_ret, reg #ifdef CONFIG_ARM_THUMB bx \reg @@ -836,7 +961,7 @@ __kuser_helper_start: * * #define __kernel_dmb() \ * asm volatile ( "mov r0, #0xffff0fff; mov lr, pc; sub pc, r0, #95" \ - * : : : "r0", "lr","cc" ) + * : : : "r0", "lr","cc" ) */ __kuser_memory_barrier: @ 0xffff0fa0 @@ -1004,7 +1129,7 @@ kuser_cmpxchg_fixup: * #define __kernel_get_tls() \ * ({ register unsigned int __val asm("r0"); \ * asm( "mov r0, #0xffff0fff; mov lr, pc; sub pc, r0, #31" \ - * : "=r" (__val) : : "lr","cc" ); \ + * : "=r" (__val) : : "lr","cc" ); \ * __val; }) */ diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S index 2c1db77..57e3bf1 100644 --- a/arch/arm/kernel/entry-common.S +++ b/arch/arm/kernel/entry-common.S @@ -2,6 +2,7 @@ * linux/arch/arm/kernel/entry-common.S * * Copyright (C) 2000 Russell King + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -22,6 +23,11 @@ * possible here, and this includes saving r0 back into the SVC * stack. */ +#ifdef CONFIG_IPIPE +__ipipe_ret_fast_syscall: + ldr r0, [sp, #S_R0+S_OFF] @ returned r0 + /* fall through */ +#endif ret_fast_syscall: UNWIND(.fnstart ) UNWIND(.cantunwind ) @@ -30,10 +36,19 @@ ret_fast_syscall: tst r1, #_TIF_WORK_MASK bne fast_work_pending +fast_restore_user_regs: /* perform architecture specific actions before user return */ arch_ret_to_user r1, lr restore_user_regs fast = 1, offset = S_OFF + +#ifdef CONFIG_IPIPE +__ipipe_fast_exit_syscall: + ldr r0, [sp, #S_R0+S_OFF] @ returned r0 + disable_irq @ disable interrupts + b fast_restore_user_regs +#endif /* CONFIG_IPIPE */ + UNWIND(.fnend ) /* @@ -46,13 +61,16 @@ work_pending: bne work_resched tst r1, #_TIF_SIGPENDING|_TIF_NOTIFY_RESUME beq no_work_pending + enable_irq_cond mov r0, sp @ 'regs' mov r2, why @ 'syscall' bl do_notify_resume b ret_slow_syscall @ Check work again work_resched: + enable_irq_cond bl schedule + /* * "slow" syscall return path. "why" tells us if this was a real syscall. */ @@ -63,16 +81,14 @@ ret_slow_syscall: tst r1, #_TIF_WORK_MASK bne work_pending no_work_pending: - /* perform architecture specific actions before user return */ - arch_ret_to_user r1, lr - - restore_user_regs fast = 0, offset = 0 + slow_restore_user_regs ENDPROC(ret_to_user) /* * This is how we return from a fork. */ ENTRY(ret_from_fork) + enable_irq_cond bl schedule_tail get_thread_info tsk ldr r1, [tsk, #TI_FLAGS] @ check for syscall tracing @@ -169,8 +185,8 @@ ftrace_stub: *----------------------------------------------------------------------------- */ - /* If we're optimising for StrongARM the resulting code won't - run on an ARM7 and we can save a couple of instructions. + /* If we're optimising for StrongARM the resulting code won't + run on an ARM7 and we can save a couple of instructions. --pb */ #ifdef CONFIG_CPU_ARM710 #define A710(code...) code @@ -275,6 +291,18 @@ ENTRY(vector_swi) #endif stmdb sp!, {r4, r5} @ push fifth and sixth args +#ifdef CONFIG_IPIPE + stmfd sp!, {r0-r3, ip} + sub sp, sp, #4 + add r1, sp, #S_OFF+24 + mov r0, scno + bl __ipipe_syscall_root + cmp r0, #0 + add sp, sp, #4 + ldmfd sp!, {r0-r3, ip} + blt __ipipe_ret_fast_syscall + bgt __ipipe_fast_exit_syscall +#endif /* CONFIG_IPIPE */ tst ip, #_TIF_SYSCALL_TRACE @ are we tracing syscalls? bne __sys_trace @@ -286,7 +314,7 @@ ENTRY(vector_swi) 2: mov why, #0 @ no longer a real syscall cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE) eor r0, scno, #__NR_SYSCALL_BASE @ put OS number back - bcs arm_syscall + bcs arm_syscall b sys_ni_syscall @ not private func ENDPROC(vector_swi) @@ -322,6 +350,9 @@ __sys_trace_return: __cr_alignment: .word cr_alignment #endif +#ifdef CONFIG_IPIPE + .word __ipipe_syscall_root +#endif .ltorg /* @@ -475,3 +506,28 @@ ENTRY(sys_oabi_call_table) #endif + +#ifdef CONFIG_FRAME_POINTER + + .text + .align 0 + .type arm_return_addr %function + .global arm_return_addr + +arm_return_addr: + mov ip, r0 + mov r0, fp +3: + cmp r0, #0 + beq 1f @ frame list hit end, bail + cmp ip, #0 + beq 2f @ reached desired frame + ldr r0, [r0, #-12] @ else continue, get next fp + sub ip, ip, #1 + b 3b +2: + ldr r0, [r0, #-4] @ get target return address +1: + mov pc, lr + +#endif diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 7e9ed1e..f465901 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -22,7 +22,7 @@ @ #define S_OFF 8 -/* +/* * The SWI code relies on the fact that R0 is at the bottom of the stack * (due to slow/fast restore user regs). */ @@ -163,6 +163,12 @@ .endm #endif /* !CONFIG_THUMB2_KERNEL */ + .macro slow_restore_user_regs + /* perform architecture specific actions before user return */ + arch_ret_to_user r1, lr + restore_user_regs fast = 0, offset = 0 + .endm + /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6. diff --git a/arch/arm/kernel/fcse.c b/arch/arm/kernel/fcse.c new file mode 100644 index 0000000..1b9550c --- /dev/null +++ b/arch/arm/kernel/fcse.c @@ -0,0 +1,465 @@ +#include <linux/bitops.h> +#include <linux/memory.h> +#include <linux/spinlock.h> +#include <linux/mm.h> +#include <linux/kernel_stat.h> +#include <linux/mman.h> +#include <linux/dcache.h> +#include <linux/fs.h> + +#include <asm/fcse.h> +#include <asm/cacheflush.h> +#include <asm/tlbflush.h> + +#define NR_PIDS (TASK_SIZE / FCSE_PID_TASK_SIZE) +#define PIDS_LONGS ((NR_PIDS + BITS_PER_LONG - 1) / BITS_PER_LONG) + +static IPIPE_DEFINE_SPINLOCK(fcse_lock); +static unsigned long fcse_pids_bits[PIDS_LONGS]; +unsigned long fcse_pids_cache_dirty[PIDS_LONGS]; +EXPORT_SYMBOL(fcse_pids_cache_dirty); + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static unsigned random_pid; +struct fcse_user fcse_pids_user[NR_PIDS]; +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + +static inline void fcse_pid_reference_inner(unsigned pid) +{ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if (++fcse_pids_user[pid].count == 1) +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __set_bit(pid, fcse_pids_bits); +} + +static inline void fcse_pid_dereference(struct mm_struct *mm) +{ + unsigned fcse_pid = mm->context.fcse.pid >> FCSE_PID_SHIFT; + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +#ifndef CONFIG_IPIPE + /* FIXME: Why ? */ + FCSE_BUG_ON(fcse_mm_in_cache(mm)); +#endif /* CONFIG_IPIPE */ + if (--fcse_pids_user[fcse_pid].count == 0) + __clear_bit(fcse_pid, fcse_pids_bits); + + /* + * The following means we suppose that by the time this + * function is called, this mm is out of cache: + * - when the caller is destroy_context, exit_mmap is called + * by mmput before, which flushes the cache; + * - when the caller is fcse_relocate_mm_to_pid from + * fcse_switch_mm_inner, we only relocate when the mm is out + * of cache; + * - when the caller is fcse_relocate_mm_to_pid from + * fcse_relocate_mm_to_null_pid, we flush the cache in this + * function. + */ + if (fcse_pids_user[fcse_pid].mm == mm) { + fcse_pids_user[fcse_pid].mm = NULL; + __clear_bit(fcse_pid, fcse_pids_cache_dirty); + } +#else /* CONFIG_ARM_FCSE_BEST_EFFORT */ + __clear_bit(fcse_pid, fcse_pids_bits); + __clear_bit(fcse_pid, fcse_pids_cache_dirty); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +} + +static inline unsigned find_free_pid(unsigned long bits[]) +{ + unsigned fcse_pid; + + fcse_pid = find_next_zero_bit(bits, NR_PIDS, 1); + if (fcse_pid == NR_PIDS) + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ + if (!test_bit(0, bits)) + fcse_pid = 0; + + return fcse_pid; +} + +void fcse_pid_free(struct mm_struct *mm) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_dereference(mm); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +int fcse_pid_alloc(struct mm_struct *mm) +{ + unsigned long flags; + unsigned fcse_pid; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid = find_free_pid(fcse_pids_bits); + if (fcse_pid == NR_PIDS) { + /* Allocate zero pid last, since zero pid is also used by + processes with address space larger than 32MB in + best-effort mode. */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + if(++random_pid == NR_PIDS) + random_pid = 0; + fcse_pid = random_pid; +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + spin_unlock_irqrestore(&fcse_lock, flags); +#ifdef CONFIG_ARM_FCSE_MESSAGES + printk(KERN_WARNING "FCSE: system would exceed the %lu processes limit.\n", + NR_PIDS); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + return -EAGAIN; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ + } + fcse_pid_reference_inner(fcse_pid); +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + FCSE_BUG_ON(fcse_mm_in_cache(mm)); +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + spin_unlock_irqrestore(&fcse_lock, flags); + + return fcse_pid; +} + +static inline void fcse_clear_dirty_all(void) +{ + switch(ARRAY_SIZE(fcse_pids_cache_dirty)) { + case 4: + fcse_pids_cache_dirty[3] = 0UL; + case 3: + fcse_pids_cache_dirty[2] = 0UL; + case 2: + fcse_pids_cache_dirty[1] = 0UL; + case 1: + fcse_pids_cache_dirty[0] = 0UL; + } +} + +unsigned fcse_flush_all_start(struct mm_struct *mm) +{ + if (!cache_is_vivt()) + return 0; + +#ifndef CONFIG_ARM_FCSE_PREEMPT_FLUSH + preempt_disable(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + +#if defined(CONFIG_IPIPE) + clear_ti_thread_flag(current_thread_info(), TIF_SWITCHED); +#elif defined(CONFIG_ARM_FCSE_PREEMPT_FLUSH) + return nr_context_switches(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + + return 0; +} + +noinline int +fcse_flush_all_done(struct mm_struct *mm, unsigned seq, unsigned dirty) +{ + unsigned long flags; + int done = 1; + + if (!cache_is_vivt()) + return 1; + + spin_lock_irqsave(&fcse_lock, flags); +#if defined(CONFIG_IPIPE) + if (test_ti_thread_flag(current_thread_info(), TIF_SWITCHED)) { + if (fcse_mm_in_cache(mm)) { + done = 0; + goto ret; + } + } else +#elif defined(CONFIG_ARM_FCSE_PREEMPT_FLUSH) + if (seq != nr_context_switches()) { + if (fcse_mm_in_cache(mm)) { + done = 0; + goto ret; + } + } else +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + fcse_clear_dirty_all(); + + if (dirty && current->mm != &init_mm && current->mm) { + unsigned fcse_pid = + current->mm->context.fcse.pid >> FCSE_PID_SHIFT; + __set_bit(fcse_pid, fcse_pids_cache_dirty); + } + + ret: + spin_unlock_irqrestore(&fcse_lock, flags); +#ifndef CONFIG_ARM_FCSE_PREEMPT_FLUSH + preempt_enable(); +#endif /* CONFIG_ARM_FCSE_PREEMPT_FLUSH */ + return done; +} + +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +/* Called with preemption disabled, mm->mmap_sem being held for writing. */ +static noinline int fcse_relocate_mm_to_pid(struct mm_struct *mm, int fcse_pid) +{ + const unsigned len = pgd_index(FCSE_TASK_SIZE) * sizeof(pgd_t); + unsigned long flags; + pgd_t *from, *to; + + spin_lock_irqsave(&fcse_lock, flags); +#if defined(CONFIG_ARM_FCSE_DYNPID) + /* pid == -1 means find a free pid. */ + if (fcse_pid == -1) { + fcse_pid = find_free_pid(fcse_pids_bits); + if (fcse_pid == NR_PIDS) { + fcse_pid = find_free_pid(fcse_pids_cache_dirty); + if (unlikely(fcse_pid == NR_PIDS)) { + spin_unlock_irqrestore(&fcse_lock, flags); + return -ENOENT; + } + } + } +#endif /* CONFIG_ARM_FCSE_DYNPID */ + fcse_pid_dereference(mm); + fcse_pid_reference_inner(fcse_pid); + fcse_pids_user[fcse_pid].mm = mm; + __set_bit(fcse_pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); + + from = pgd_offset(mm, 0); + mm->context.fcse.pid = fcse_pid << FCSE_PID_SHIFT; + to = pgd_offset(mm, 0); + + memcpy(to, from, len); + memset(from, '\0', len); + barrier(); + clean_dcache_area(from, len); + clean_dcache_area(to, len); + + return fcse_pid; +} + +int fcse_switch_mm_inner(struct mm_struct *prev, struct mm_struct *next) +{ + unsigned fcse_pid = next->context.fcse.pid >> FCSE_PID_SHIFT; + unsigned flush_needed, reused_pid = 0; + unsigned long flags; + + if (unlikely(next == &init_mm)) { + spin_lock_irqsave(&fcse_lock, flags); + goto is_flush_needed; + } + +#ifdef CONFIG_ARM_FCSE_DYNPID + /* + * If the next mm's pid is currently in use, and not by that + * mm, try and find a new, free, pid. + */ + if (unlikely(fcse_pids_user[fcse_pid].mm != next) + && test_bit(fcse_pid, fcse_pids_cache_dirty) + && !rwsem_is_locked(&next->mmap_sem) + && fcse_pids_user[fcse_pid].mm + && !next->context.fcse.large + && !next->core_state) { + int new_fcse_pid = fcse_relocate_mm_to_pid(next, -1); + if (new_fcse_pid >= 0) + fcse_pid = new_fcse_pid; + } +#endif /* CONFIG_ARM_FCSE_DYNPID */ + + spin_lock_irqsave(&fcse_lock, flags); + if (fcse_pids_user[fcse_pid].mm != next) { + if (fcse_pids_user[fcse_pid].mm) + reused_pid = test_bit(fcse_pid, fcse_pids_cache_dirty); + fcse_pids_user[fcse_pid].mm = next; + } + + is_flush_needed: + flush_needed = reused_pid + || next->context.fcse.high_pages + || !prev + || prev->context.fcse.shared_dirty_pages + || prev->context.fcse.high_pages; + + fcse_pid_set(fcse_pid << FCSE_PID_SHIFT); + if (flush_needed) + fcse_clear_dirty_all(); + if (next != &init_mm) + __set_bit(fcse_pid, fcse_pids_cache_dirty); + spin_unlock_irqrestore(&fcse_lock, flags); + + return flush_needed; +} +EXPORT_SYMBOL_GPL(fcse_switch_mm_inner); + +void fcse_pid_reference(unsigned fcse_pid) +{ + unsigned long flags; + + spin_lock_irqsave(&fcse_lock, flags); + fcse_pid_reference_inner(fcse_pid); + spin_unlock_irqrestore(&fcse_lock, flags); +} + +/* Called with mm->mmap_sem write-locked. */ +static noinline void fcse_relocate_mm_to_null_pid(struct mm_struct *mm) +{ + unsigned long seq; + + if (!cache_is_vivt()) + return; + + preempt_disable(); + if (fcse_mm_in_cache(mm)) + do { + preempt_enable(); + + seq = fcse_flush_all_start(mm); + flush_cache_all(); + + preempt_disable(); + } while (!fcse_flush_all_done(mm, seq, 0)); + + fcse_relocate_mm_to_pid(mm, 0); + barrier(); + flush_tlb_mm(mm); + fcse_pid_set(0); + + preempt_enable(); +} +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ + +unsigned long +fcse_check_mmap_inner(struct mm_struct *mm, unsigned long start_addr, + unsigned long addr, unsigned long len, unsigned long fl) +{ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + unsigned long stack_reserved = + current->signal->rlim[RLIMIT_STACK].rlim_cur; + unsigned long stack_base = PAGE_ALIGN(mm->start_stack) - stack_reserved; + + /* We enfore the RLIMIT_STACK stack size, and here, the return + address would fall in that reserved stack area */ + if ((unsigned long)(addr + len - stack_base) < stack_reserved) { + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED) + && !mm->context.fcse.large) + return TASK_UNMAPPED_BASE; + + /* Forcibly restart from above the stack */ + if (!(fl & MAP_FIXED)) + return PAGE_ALIGN(mm->start_stack); + + /* If MAP_FIXED is set, we encroach upon the reserved + stack area. No choice. */ + } + + /* Address above 32MB */ + if (addr + len > FCSE_TASK_SIZE && !mm->context.fcse.high_pages) { + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED)) + return TASK_UNMAPPED_BASE; + + if (!mm->context.fcse.large) { + /* Ok, the process is going to be larger than 32MB */ +#ifdef CONFIG_ARM_FCSE_MSSAGES + printk(KERN_INFO "FCSE: process %u(%s) VM exceeds 32MB.\n", + current->pid, current->comm); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + mm->context.fcse.large = 1; + } + if (mm->context.fcse.pid) + fcse_relocate_mm_to_null_pid(mm); + } + + return addr; + +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + /* Address above 32MB */ + /* Restart to try and find a hole, once. */ + if (start_addr != TASK_UNMAPPED_BASE && !(fl & MAP_FIXED)) + return TASK_UNMAPPED_BASE; + + /* Fail, no 32MB processes in guaranteed mode. */ +#ifdef CONFIG_ARM_FCSE_MESSAGES + printk(KERN_WARNING "FCSE: process %u(%s) VM would exceed the 32MB limit.\n", + current->pid, current->comm); +#endif /* CONFIG_ARM_FCSE_MESSAGES */ + return -ENOMEM; +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +} + +#ifdef CONFIG_ARM_FCSE_MESSAGES +#define addr_in_vma(vma, addr) \ + ({ \ + struct vm_area_struct *_vma = (vma); \ + ((unsigned long)((addr) - _vma->vm_start) \ + < (unsigned long)((_vma->vm_end - _vma->vm_start))); \ + }) + +#ifdef CONFIG_DEBUG_USER +static noinline void +dump_vmas(struct mm_struct *mm, unsigned long addr, struct pt_regs *regs) +{ + struct vm_area_struct *vma; + char path[128]; + + printk("mappings:\n"); + down_read(&mm->mmap_sem); + for(vma = mm->mmap; vma; vma = vma->vm_next) { + struct file *file = vma->vm_file; + int flags = vma->vm_flags; + const char *name; + + printk("0x%08lx-0x%08lx %c%c%c%c 0x%08llx ", + vma->vm_start, + vma->vm_end, + flags & VM_READ ? 'r' : '-', + flags & VM_WRITE ? 'w' : '-', + flags & VM_EXEC ? 'x' : '-', + flags & VM_MAYSHARE ? 's' : 'p', + ((loff_t)vma->vm_pgoff) << PAGE_SHIFT); + + if (file) + name = d_path(&file->f_path, path, sizeof(path)); + else if ((name = arch_vma_name(vma))) + ; + else if (!vma->vm_mm) + name = "[vdso]"; + else if (vma->vm_start <= mm->start_brk + && vma->vm_end >= mm->brk) + name = "[heap]"; + else if (vma->vm_start <= mm->start_stack && + vma->vm_end >= mm->start_stack) + name = "[stack]"; + else + name = ""; + printk("%s", name); + if (addr_in_vma(vma, regs->ARM_pc)) + printk(" <- PC"); + if (addr_in_vma(vma, regs->ARM_sp)) + printk(" <- SP"); + if (addr_in_vma(vma, addr)) + printk("%s fault", + (addr_in_vma(vma, regs->ARM_pc) + || addr_in_vma(vma, regs->ARM_sp) + ? "," : " <-")); + printk("\n"); + } + up_read(&mm->mmap_sem); +} +#endif /* CONFIG_DEBUG_USER */ + +void fcse_notify_segv(struct mm_struct *mm, + unsigned long addr, struct pt_regs *regs) +{ +#if defined(CONFIG_DEBUG_USER) + if (user_debug & UDBG_SEGV) + dump_vmas(mm, addr, regs); +#endif /* CONFIG_DEBUG_USER */ + + down_read(&mm->mmap_sem); + if (find_vma(mm, addr) == find_vma(mm, regs->ARM_sp)) + printk(KERN_INFO "FCSE: process %u(%s) probably overflowed stack at 0x%08lx.\n", + current->pid, current->comm, regs->ARM_pc); + up_read(&mm->mmap_sem); +} +#endif /* CONFIG_ARM_FCSE_MESSAGES */ diff --git a/arch/arm/kernel/ipipe.c b/arch/arm/kernel/ipipe.c new file mode 100644 index 0000000..41b1013 --- /dev/null +++ b/arch/arm/kernel/ipipe.c @@ -0,0 +1,549 @@ +/* -*- linux-c -*- + * linux/arch/arm/kernel/ipipe.c + * + * Copyright (C) 2002-2005 Philippe Gerum. + * Copyright (C) 2004 Wolfgang Grandegger (Adeos/arm port over 2.4). + * Copyright (C) 2005 Heikki Lindholm (PowerPC 970 fixes). + * Copyright (C) 2005 Stelian Pop. + * Copyright (C) 2006-2008 Gilles Chanteperdrix. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Architecture-dependent I-PIPE support for ARM. + */ + +#include <linux/kernel.h> +#include <linux/smp.h> +#include <linux/sched.h> +#include <linux/slab.h> +#include <linux/bitops.h> +#include <linux/interrupt.h> +#include <linux/module.h> +#include <linux/errno.h> +#include <linux/kallsyms.h> +#include <linux/kprobes.h> +#include <linux/ipipe_trace.h> +#include <asm/system.h> +#include <asm/atomic.h> +#include <asm/hardirq.h> +#include <asm/io.h> +#include <asm/irq.h> +#include <asm/unistd.h> +#include <asm/mach/irq.h> +#include <asm/mmu_context.h> + +/* Next tick date (timebase value). */ +DEFINE_PER_CPU(struct pt_regs, __ipipe_tick_regs); +DEFINE_PER_CPU(struct mm_struct *,ipipe_active_mm); +EXPORT_PER_CPU_SYMBOL(ipipe_active_mm); +#ifdef __IPIPE_FEATURE_PIC_MUTE +__ipipe_irqbits_t __ipipe_irqbits; +IPIPE_DEFINE_SPINLOCK(__ipipe_irqbits_lock); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +extern struct irq_desc irq_desc[]; +asmlinkage void asm_do_IRQ(unsigned int irq, struct pt_regs *regs); + +#ifdef CONFIG_SMP + +static cpumask_t __ipipe_cpu_sync_map; + +static cpumask_t __ipipe_cpu_lock_map; + +static IPIPE_DEFINE_SPINLOCK(__ipipe_cpu_barrier); + +static atomic_t __ipipe_critical_count = ATOMIC_INIT(0); + +static void (*__ipipe_cpu_sync) (void); + +/* Always called with hw interrupts off. */ + + +void __ipipe_do_critical_sync(unsigned irq) +{ + int cpu = ipipe_processor_id(); + + cpu_set(cpu, __ipipe_cpu_sync_map); + + /* + * Now we are in sync with the lock requestor running on another + * CPU. Enter a spinning wait until he releases the global + * lock. + */ + spin_lock(&__ipipe_cpu_barrier); + + /* Got it. Now get out. */ + + if (__ipipe_cpu_sync) + /* Call the sync routine if any. */ + __ipipe_cpu_sync(); + + spin_unlock(&__ipipe_cpu_barrier); + + cpu_clear(cpu, __ipipe_cpu_sync_map); +} + +#endif /* CONFIG_SMP */ + +/* + * ipipe_trigger_irq() -- Push the interrupt at front of the pipeline + * just like if it has been actually received from a hw source. Also + * works for virtual interrupts. + */ +int ipipe_trigger_irq(unsigned irq) +{ + unsigned long flags; + +/* FIXME */ +#ifdef CONFIG_IPIPE_DEBUG + if (irq >= IPIPE_NR_IRQS || + (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))) + return -EINVAL; +#else + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + if (ipipe_virtual_irq_p(irq)) { + if (!test_bit(irq - IPIPE_VIRQ_BASE, + &__ipipe_virtual_irq_map)) + return -EINVAL; + } else if (irq_to_desc(irq) == NULL) + return -EINVAL; +#endif + + local_irq_save_hw(flags); + __ipipe_handle_irq(irq, NULL); + local_irq_restore_hw(flags); + + return 1; +} + +int ipipe_get_sysinfo(struct ipipe_sysinfo *info) +{ + info->ncpus = num_online_cpus(); + info->cpufreq = ipipe_cpu_freq(); + info->archdep.tmirq = __ipipe_mach_timerint; + info->archdep.tmfreq = info->cpufreq; + __ipipe_mach_get_tscinfo(&info->archdep.tsc); + + return 0; +} + +static void __ipipe_ack_irq(unsigned irq, struct irq_desc *desc) +{ + desc->ipipe_ack(irq, desc); +} + +static void __ipipe_ack_timerirq(unsigned irq, struct irq_desc *desc) +{ + desc->ipipe_ack(irq, desc); + __ipipe_mach_acktimer(); + desc->ipipe_end(irq, desc); +} + +void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq) +{ + unsigned long flags; + irq_desc[irq].status &= ~IRQ_DISABLED; +#ifdef __IPIPE_FEATURE_PIC_MUTE + if (ipd == &ipipe_root) + return; + + spin_lock_irqsave(&__ipipe_irqbits_lock, flags); + __ipipe_irqbits[irq / BITS_PER_LONG] &= ~(1 << (irq % BITS_PER_LONG)); + spin_unlock_irqrestore(&__ipipe_irqbits_lock, flags); +#else + (void) flags; +#endif /* __IPIPE_FEATURE_PIC_MUTE */ +} +EXPORT_SYMBOL(__ipipe_enable_irqdesc); + +#ifdef __IPIPE_FEATURE_PIC_MUTE +void __ipipe_disable_irqdesc(struct ipipe_domain *ipd, unsigned irq) +{ + unsigned long flags; + + if (ipd == &ipipe_root) + return; + + spin_lock_irqsave(&__ipipe_irqbits_lock, flags); + __ipipe_irqbits[irq / BITS_PER_LONG] |= 1 << (irq % BITS_PER_LONG); + spin_unlock_irqrestore(&__ipipe_irqbits_lock, flags); +} +EXPORT_SYMBOL(__ipipe_disable_irqdesc); +EXPORT_SYMBOL(ipipe_mute_pic); +EXPORT_SYMBOL(ipipe_unmute_pic); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ + +/* + * __ipipe_enable_pipeline() -- We are running on the boot CPU, hw + * interrupts are off, and secondary CPUs are still lost in space. + */ +void __ipipe_enable_pipeline(void) +{ + unsigned long flags; + unsigned irq; + +/* We do not want "wfi" to be called in arm926ejs based processor, as + this causes the I-cache to be disabled when idle. */ +#ifdef CONFIG_CPU_ARM926T + disable_hlt(); +#endif + + flags = ipipe_critical_enter(NULL); + + /* First, virtualize all interrupts from the root domain. */ + + for (irq = 0; irq < NR_IRQS; irq++) + ipipe_virtualize_irq(ipipe_root_domain, + irq, + (ipipe_irq_handler_t)&asm_do_IRQ, NULL, + ((irq == __ipipe_mach_timerint) + ? &__ipipe_ack_timerirq + : &__ipipe_ack_irq), + IPIPE_HANDLE_MASK | IPIPE_PASS_MASK); + + ipipe_critical_exit(flags); +} + +/* + * ipipe_critical_enter() -- Grab the superlock excluding all CPUs + * but the current one from a critical section. This lock is used when + * we must enforce a global critical section for a single CPU in a + * possibly SMP system whichever context the CPUs are running. + */ +unsigned long ipipe_critical_enter(void (*syncfn) (void)) +{ + unsigned long flags; + + local_irq_save_hw(flags); + +#ifdef CONFIG_SMP + if (likely(num_online_cpus() > 1)) { /* We might be running a SMP-kernel on a UP box... */ + int cpu = ipipe_processor_id(); + cpumask_t lock_map; + + if (!cpu_test_and_set(cpu, __ipipe_cpu_lock_map)) { + while (cpu_test_and_set(BITS_PER_LONG - 1, + __ipipe_cpu_lock_map)) { + int n = 0; + do { + cpu_relax(); + } while (++n < cpu); + } + + spin_lock(&__ipipe_cpu_barrier); + + __ipipe_cpu_sync = syncfn; + + /* Send the sync IPI to all processors but the current one. */ + send_IPI_allbutself(IPIPE_CRITICAL_VECTOR); + + cpus_andnot(lock_map, cpu_online_map, + __ipipe_cpu_lock_map); + + while (!cpus_equal(__ipipe_cpu_sync_map, lock_map)) + cpu_relax(); + } + + atomic_inc(&__ipipe_critical_count); + } +#endif /* CONFIG_SMP */ + + return flags; +} + +/* ipipe_critical_exit() -- Release the superlock. */ + +void ipipe_critical_exit(unsigned long flags) +{ +#ifdef CONFIG_SMP + if (likely(num_online_cpus() > 1)) { /* We might be running a SMP-kernel on a UP box... */ + if (atomic_dec_and_test(&__ipipe_critical_count)) { + spin_unlock(&__ipipe_cpu_barrier); + + while (!cpus_empty(__ipipe_cpu_sync_map)) + cpu_relax(); + + cpu_clear(ipipe_processor_id(), __ipipe_cpu_lock_map); + cpu_clear(BITS_PER_LONG - 1, __ipipe_cpu_lock_map); + } + } +#endif /* CONFIG_SMP */ + + local_irq_restore_hw(flags); +} + +#ifdef CONFIG_PREEMPT + +asmlinkage void __sched __ipipe_preempt_schedule_irq(void) +{ + extern asmlinkage void __sched preempt_schedule_irq(void); + struct ipipe_percpu_domain_data *p; + unsigned long flags; + /* + * preempt_schedule_irq expects the root stage to be stalled. + */ +#ifdef CONFIG_IPIPE_DEBUG + BUG_ON(!irqs_disabled_hw()); +#endif + local_irq_save(flags); + local_irq_enable_hw(); + preempt_schedule_irq(); /* Ok, may reschedule now. */ + local_irq_disable_hw(); + + /* + * Flush any pending interrupt that may have been logged after + * preempt_schedule_irq() stalled the root stage before + * returning to us, and now. + */ + p = ipipe_root_cpudom_ptr(); + if (unlikely(__ipipe_ipending_p(p))) { + add_preempt_count(PREEMPT_ACTIVE); + clear_bit(IPIPE_STALL_FLAG, &p->status); + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + sub_preempt_count(PREEMPT_ACTIVE); + } + + __local_irq_restore_nosync(flags); +} + +#endif /* CONFIG_PREEMPT */ + +asmlinkage int __ipipe_check_root(void) +{ + return ipipe_root_domain_p; +} + +asmlinkage int __ipipe_check_root_interruptible(void) +{ + return ipipe_root_domain_p && !__ipipe_test_root(); +} + +__kprobes int +__ipipe_switch_to_notifier_call_chain(struct atomic_notifier_head *nh, + unsigned long val, void *v) +{ + unsigned long flags; + int rc; + + local_irq_save(flags); + rc = atomic_notifier_call_chain(nh, val, v); + __local_irq_restore_nosync(flags); + + return rc; +} + +asmlinkage int __ipipe_syscall_root(unsigned long scno, struct pt_regs *regs) +{ + struct ipipe_percpu_domain_data *p; + unsigned long flags, origr7; + + /* We use r7 to pass the syscall number to the other domains */ + origr7 = regs->ARM_r7; + regs->ARM_r7 = __NR_SYSCALL_BASE + scno; + /* + * This routine either returns: + * 0 -- if the syscall is to be passed to Linux; + * >0 -- if the syscall should not be passed to Linux, and no + * tail work should be performed; + * <0 -- if the syscall should not be passed to Linux but the + * tail work has to be performed (for handling signals etc). + */ + + WARN_ON_ONCE(irqs_disabled_hw()); + + if (!__ipipe_syscall_watched_p(current, regs->ARM_r7) || + !__ipipe_event_monitored_p(IPIPE_EVENT_SYSCALL)) + goto done; + + if (__ipipe_dispatch_event(IPIPE_EVENT_SYSCALL,regs) > 0){ + if (ipipe_root_domain_p && !in_atomic()) { + /* + * Sync pending VIRQs before _TIF_NEED_RESCHED + * is tested. + */ + local_irq_save_hw(flags); + p = ipipe_root_cpudom_ptr(); + if (__ipipe_ipending_p(p)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOVIRT); + local_irq_restore_hw(flags); + regs->ARM_r7 = origr7; + return -1; + } + regs->ARM_r7 = origr7; + return 1; + } +done: + regs->ARM_r7 = origr7; + return 0; +} + +/* + * __ipipe_handle_irq() -- IPIPE's generic IRQ handler. An optimistic + * interrupt protection log is maintained here for each domain. Hw + * interrupts are off on entry. + */ +int __ipipe_handle_irq(int irq, struct pt_regs *regs) +{ + struct ipipe_domain *this_domain, *next_domain; + struct list_head *head, *pos; + int m_ack; + + m_ack = (regs == NULL); + +#ifdef CONFIG_IPIPE_DEBUG + if (irq >= IPIPE_NR_IRQS) { + printk(KERN_ERR "I-pipe: spurious interrupt %d\n", irq); + goto finalize_nosync; + } +#endif + + this_domain = ipipe_current_domain; + + if (test_bit(IPIPE_STICKY_FLAG, &this_domain->irqs[irq].control)) + head = &this_domain->p_link; + else { + head = __ipipe_pipeline.next; + next_domain = list_entry(head, struct ipipe_domain, p_link); + if (likely(test_bit(IPIPE_WIRED_FLAG, &next_domain->irqs[irq].control))) { + if (!m_ack && next_domain->irqs[irq].acknowledge != NULL) + next_domain->irqs[irq].acknowledge(irq, irq_desc + irq); + __ipipe_dispatch_wired(next_domain, irq); + goto finalize_nosync; + } + } + + /* Ack the interrupt. */ + + pos = head; + + while (pos != &__ipipe_pipeline) { + next_domain = list_entry(pos, struct ipipe_domain, p_link); + prefetch(next_domain); + + /* + * For each domain handling the incoming IRQ, mark it + * as pending in its log. + */ + if (test_bit(IPIPE_HANDLE_FLAG, &next_domain->irqs[irq].control)) { + /* + * Domains that handle this IRQ are polled for + * acknowledging it by decreasing priority + * order. The interrupt must be made pending + * _first_ in the domain's status flags before + * the PIC is unlocked. + */ + __ipipe_set_irq_pending(next_domain, irq); + + if (!m_ack && next_domain->irqs[irq].acknowledge) { + next_domain->irqs[irq].acknowledge(irq, irq_desc + irq); + m_ack = 1; + } + } + + /* + * If the domain does not want the IRQ to be passed + * down the interrupt pipe, exit the loop now. + */ + + if (!test_bit(IPIPE_PASS_FLAG, &next_domain->irqs[irq].control)) + break; + + pos = next_domain->p_link.next; + } + + /* + * If the interrupt preempted the head domain, then do not + * even try to walk the pipeline, unless an interrupt is + * pending for it. + */ + if (test_bit(IPIPE_AHEAD_FLAG, &this_domain->flags) && + ipipe_head_cpudom_var(irqpend_himap) == 0) + goto finalize_nosync; + + /* + * Now walk the pipeline, yielding control to the highest + * priority domain that has pending interrupt(s) or + * immediately to the current domain if the interrupt has been + * marked as 'sticky'. This search does not go beyond the + * current domain in the pipeline. + */ + + __ipipe_walk_pipeline(head); + +finalize_nosync: + if (!ipipe_root_domain_p || __ipipe_test_root()) + return 0; + +#ifdef CONFIG_SMP + /* + * Prevent a spurious rescheduling from being triggered on + * preemptible kernels along the way out through + * ret_from_intr. + */ + if (!regs) + __set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); +#endif /* CONFIG_SMP */ + + return 1; +} + +asmlinkage int __ipipe_grab_irq(int irq, struct pt_regs *regs) +{ + int status; + +#ifdef irq_finish + /* AT91 specific workaround */ + irq_finish(irq); +#endif /* irq_finish */ + + if (irq == __ipipe_mach_timerint) { + /* + * Given our deferred dispatching model for regular IRQs, we + * only record CPU regs for the last timer interrupt, so that + * the timer handler charges CPU times properly. It is assumed + * that other interrupt handlers don't actually care for such + * information. + */ + __raw_get_cpu_var(__ipipe_tick_regs).ARM_cpsr = + (ipipe_root_domain_p + ? regs->ARM_cpsr + : regs->ARM_cpsr | PSR_I_BIT); + __raw_get_cpu_var(__ipipe_tick_regs).ARM_pc = regs->ARM_pc; + } + + ipipe_trace_irq_entry(irq); + status = __ipipe_handle_irq(irq, regs); + ipipe_trace_irq_exit(irq); + + return status; +} + +EXPORT_SYMBOL_GPL(show_stack); +EXPORT_SYMBOL_GPL(init_mm); +#ifndef MULTI_CPU +EXPORT_SYMBOL_GPL(cpu_do_switch_mm); +#endif +EXPORT_SYMBOL_GPL(__check_kvm_seq); +#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) +EXPORT_SYMBOL_GPL(tasklist_lock); +#endif /* CONFIG_SMP || CONFIG_DEBUG_SPINLOCK */ + +#ifdef CONFIG_CPU_HAS_ASID +EXPORT_SYMBOL_GPL(__new_context); +EXPORT_SYMBOL_GPL(cpu_last_asid); +#endif /* CONFIG_CPU_HAS_ASID */ diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index b7cb45b..b759267 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -121,8 +121,10 @@ asmlinkage void __exception asm_do_IRQ(unsigned int irq, struct pt_regs *regs) generic_handle_irq(irq); } +#ifndef CONFIG_IPIPE /* AT91 specific workaround */ irq_finish(irq); +#endif /* !CONFIG_IPIPE */ irq_exit(); set_irq_regs(old_regs); diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index ba2adef..f612624 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -127,8 +127,15 @@ EXPORT_SYMBOL_GPL(arm_pm_restart); */ static void default_idle(void) { - if (!need_resched()) + if (!need_resched()) { +#ifdef CONFIG_IPIPE + __ipipe_unstall_root(); +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF + ipipe_trace_end(0x8000000E); +#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */ +#endif /* CONFIG_IPIPE */ arch_idle(); + } local_irq_enable(); } @@ -148,6 +155,7 @@ void cpu_idle(void) /* endless idle loop with no priority at all */ while (1) { tick_nohz_stop_sched_tick(1); + ipipe_suspend_domain(); leds_event(led_idle_start); while (!need_resched()) { #ifdef CONFIG_HOTPLUG_CPU diff --git a/arch/arm/kernel/ptrace.c b/arch/arm/kernel/ptrace.c index a2ea385..487ab4c 100644 --- a/arch/arm/kernel/ptrace.c +++ b/arch/arm/kernel/ptrace.c @@ -479,6 +479,10 @@ void ptrace_break(struct task_struct *tsk, struct pt_regs *regs) static int break_trap(struct pt_regs *regs, unsigned int instr) { + + if (ipipe_trap_notify(IPIPE_TRAP_BREAK,regs)) + return 0; + ptrace_break(current, regs); return 0; } diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 57162af..b891257 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -37,6 +37,7 @@ #include <asm/ptrace.h> #include <asm/localtimer.h> #include <asm/smp_plat.h> +#include <asm/fcse.h> /* * as from 2.5, kernels no longer have an init_tasks structure @@ -638,7 +639,7 @@ void flush_tlb_all(void) void flush_tlb_mm(struct mm_struct *mm) { if (tlb_ops_need_broadcast()) - on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, mm_cpumask(mm)); + on_each_cpu_mask(ipi_flush_tlb_mm, mm, 1, fcse_tlb_mask(mm)); else local_flush_tlb_mm(mm); } @@ -649,7 +650,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) struct tlb_args ta; ta.ta_vma = vma; ta.ta_start = uaddr; - on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_page, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_page(vma, uaddr); } @@ -672,7 +673,7 @@ void flush_tlb_range(struct vm_area_struct *vma, ta.ta_vma = vma; ta.ta_start = start; ta.ta_end = end; - on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, mm_cpumask(vma->vm_mm)); + on_each_cpu_mask(ipi_flush_tlb_range, &ta, 1, fcse_tlb_mask(vma->vm_mm)); } else local_flush_tlb_range(vma, start, end); } diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 3f361a7..efe92a5 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -331,6 +331,11 @@ asmlinkage void __exception do_undefinstr(struct pt_regs *regs) siginfo_t info; void __user *pc; + /* In kernel mode, the trap was already signaled at the beginning of + * __und_svc in arch/arm/kernel/entry-armv.S */ + if (user_mode(regs) && ipipe_trap_notify(IPIPE_TRAP_UNDEFINSTR, regs)) + return; + /* * According to the ARM ARM, PC is 2 or 4 bytes ahead, * depending whether we're in Thumb mode or not. @@ -728,13 +733,22 @@ void __init trap_init(void) return; } +#ifdef CONFIG_IPIPE +void *__ipipe_tsc_area; +#endif /* CONFIG_IPIPE */ + void __init early_trap_init(void) { unsigned long vectors = CONFIG_VECTORS_BASE; extern char __stubs_start[], __stubs_end[]; extern char __vectors_start[], __vectors_end[]; +#ifndef CONFIG_IPIPE extern char __kuser_helper_start[], __kuser_helper_end[]; int kuser_sz = __kuser_helper_end - __kuser_helper_start; +#else /* !CONFIG_IPIPE */ + extern char __ipipe_tsc_area_start[], __kuser_helper_end[]; + int kuser_sz = __kuser_helper_end - __ipipe_tsc_area_start; +#endif /* !CONFIG_IPIPE */ /* * Copy the vectors, stubs and kuser helpers (in entry-armv.S) @@ -743,7 +757,13 @@ void __init early_trap_init(void) */ memcpy((void *)vectors, __vectors_start, __vectors_end - __vectors_start); memcpy((void *)vectors + 0x200, __stubs_start, __stubs_end - __stubs_start); +#ifndef CONFIG_IPIPE memcpy((void *)vectors + 0x1000 - kuser_sz, __kuser_helper_start, kuser_sz); +#else /* !CONFIG_IPIPE */ + BUG_ON(0x1000 - kuser_sz < 0x200 + __stubs_end - __stubs_start); + memcpy((void *)vectors + 0x1000 - kuser_sz, __ipipe_tsc_area_start, kuser_sz); + __ipipe_tsc_area = (void *)vectors + 0x1000 - kuser_sz; +#endif /* !CONFIG_IPIPE */ /* * Copy signal return handlers into the vector page, and diff --git a/arch/arm/mach-at91/Kconfig b/arch/arm/mach-at91/Kconfig index 0b2ee95..2308048 100644 --- a/arch/arm/mach-at91/Kconfig +++ b/arch/arm/mach-at91/Kconfig @@ -406,6 +406,19 @@ endif # ---------------------------------------------------------- +comment "Adeos I-pipe Options" + +config IPIPE_AT91_TC + depends on IPIPE + int "AT91 TC used as time base by Adeos I-pipe" + default 0 + help + When Adeos interrupt pipeline is enabled, TC0 is used by default + as time base, but you can use TC1 or TC2 by setting this variable to 1 + or 2. This should only be needed to avoid conflicts with other drivers. + +# ---------------------------------------------------------- + comment "AT91 Board Options" config MTD_AT91_DATAFLASH_CARD diff --git a/arch/arm/mach-at91/Makefile b/arch/arm/mach-at91/Makefile index 709fbad..09cfdd0 100644 --- a/arch/arm/mach-at91/Makefile +++ b/arch/arm/mach-at91/Makefile @@ -83,3 +83,14 @@ obj-$(CONFIG_CPU_IDLE) += cpuidle.o ifeq ($(CONFIG_PM_DEBUG),y) CFLAGS_pm.o += -DDEBUG endif + +ifeq ($(CONFIG_IPIPE),y) +obj-y := $(filter-out at91rm9200_time.o at91sam926x_time.o at91x40_time.o, $(obj-y)) +obj-$(CONFIG_ARCH_AT91RM9200) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9260) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9261) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9263) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9RL) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91SAM9G20) += at91_ipipe_time.o +obj-$(CONFIG_ARCH_AT91X40) += at91_ipipe_time.o +endif diff --git a/arch/arm/mach-at91/at91_ipipe_time.c b/arch/arm/mach-at91/at91_ipipe_time.c new file mode 100644 index 0000000..a31515c --- /dev/null +++ b/arch/arm/mach-at91/at91_ipipe_time.c @@ -0,0 +1,419 @@ +/* + * linux/arch/arm/mach-at91/at91_ipipe_time.c + * + * Copyright (C) 2007 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> + * + * Adaptation to AT91SAM926x: + * Copyright (C) 2007 Gregory CLEMENT, Adeneo + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include <linux/interrupt.h> +#include <linux/irq.h> +#include <linux/kernel.h> +#include <linux/clockchips.h> +#include <linux/clk.h> +#include <linux/stringify.h> +#include <linux/err.h> +#include <linux/console.h> +#include <linux/module.h> +#include <linux/ipipe.h> + +#include <asm/io.h> +#include <asm/mach/time.h> + +#include <mach/hardware.h> +#include <mach/at91_st.h> +#include <mach/at91_tc.h> +#include <mach/at91_pit.h> +#include "clock.h" + +#if defined(CONFIG_ARCH_AT91RM9200) +#define AT91_ID_TC0 AT91RM9200_ID_TC0 +#define AT91_ID_TC1 AT91RM9200_ID_TC1 +#define AT91_ID_TC2 AT91RM9200_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9G20) +#define AT91_ID_TC0 AT91SAM9260_ID_TC0 +#define AT91_ID_TC1 AT91SAM9260_ID_TC1 +#define AT91_ID_TC2 AT91SAM9260_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9261) +#define AT91_ID_TC0 AT91SAM9261_ID_TC0 +#define AT91_ID_TC1 AT91SAM9261_ID_TC1 +#define AT91_ID_TC2 AT91SAM9261_ID_TC2 +#elif defined(CONFIG_ARCH_AT91SAM9263) +#define AT91_ID_TC0 AT91SAM9263_ID_TCB +#define AT91_ID_TC1 AT91SAM9263_ID_TCB +#define AT91_ID_TC2 AT91SAM9263_ID_TCB +#elif defined(CONFIG_ARCH_AT91SAM9RL) +#define AT91_ID_TC0 AT91SAM9RL_ID_TC0 +#define AT91_ID_TC1 AT91SAM9RL_ID_TC1 +#define AT91_ID_TC2 AT91SAM9RL_ID_TC2 +#elif defined(CONFIG_ARCH_AT91X40) +#define AT91_ID_TC0 AT91X40_ID_TC0 +#define AT91_ID_TC1 AT91X40_ID_TC1 +#define AT91_ID_TC2 AT91X40_ID_TC2 +#else +#error "AT91 processor unsupported by Adeos" +#endif + +#if (CONFIG_IPIPE_AT91_TC==0) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC0 +#elif (CONFIG_IPIPE_AT91_TC==1) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC1 +#elif (CONFIG_IPIPE_AT91_TC==2) +# define KERNEL_TIMER_IRQ_NUM AT91_ID_TC2 +#else +#error IPIPE_AT91_TC must be 0, 1 or 2. +#endif + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ + +#define TCNXCNS(timer,v) ((v) << ((timer)<<1)) +#define AT91_TC_REG_MASK (0xffff) + +static unsigned long next_match; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + union { + struct { + unsigned short mid; + unsigned short low; + }; + unsigned long mid_low; + }; + }; +#else /* __LITTLE_ENDIAN */ + struct { + union { + struct { + unsigned short low; + unsigned short mid; + }; + unsigned long mid_low; + }; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; +static unsigned max_delta_ticks, min_delta_ticks; +static struct clock_event_device clkevt; +static int tc_timer_clock; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = + (unsigned *) + (AT91_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC + AT91_TC_CV); + info->u.fr.mask = AT91_TC_REG_MASK; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static inline unsigned int at91_tc_read(unsigned int reg_offset) +{ + unsigned long addr = + (AT91_VA_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC); + + return readl((void __iomem *)(addr + reg_offset)); +} + +static inline void at91_tc_write(unsigned int reg_offset, unsigned long value) +{ + unsigned long addr = + (AT91_VA_BASE_TCB0 + 0x40 * CONFIG_IPIPE_AT91_TC); + + writel(value, (void __iomem *)(addr + reg_offset)); +} + +#define read_CV() at91_tc_read(AT91_TC_CV) +#define read_RC() at91_tc_read(AT91_TC_RC) +#define write_RC(value) at91_tc_write(AT91_TC_RC, value) + +int __ipipe_mach_timerint = KERNEL_TIMER_IRQ_NUM; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static void ipipe_mach_update_tsc(unsigned short stamp); + +static int at91_timer_initialized; + +/* + * IRQ handler for the timer. + */ +static irqreturn_t at91_timer_interrupt(int irq, void *dev_id) +{ + clkevt.event_handler(&clkevt); + return IRQ_HANDLED; +} + +static struct irqaction at91_timer_irq = { + .name = "at91_tick", + .flags = IRQF_DISABLED | IRQF_TIMER, + .handler = &at91_timer_interrupt +}; + +static void ipipe_mach_update_tsc(unsigned short stamp) +{ + union tsc_reg *local_tsc; + + local_tsc = &tsc[ipipe_processor_id()]; + if (unlikely(stamp < local_tsc->low)) + local_tsc->full += AT91_TC_REG_MASK + 1; + local_tsc->low = stamp; +} + +void __ipipe_mach_acktimer(void) +{ + at91_tc_read(AT91_TC_SR); + + if (unlikely(!__ipipe_mach_timerstolen)) { + ipipe_mach_update_tsc(read_CV()); + next_match = (next_match + __ipipe_mach_ticks_per_jiffy) + & AT91_TC_REG_MASK; + write_RC(next_match); + } +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(at91_timer_initialized)) { + union tsc_reg *local_tsc, before, after; + unsigned short stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(after.full): "r"(local_tsc), "m"(*local_tsc)); + do { + before = after; + stamp = read_CV(); + barrier(); + __asm__ ("ldmia %1, %M0\n": + "=r"(after.full): + "r"(local_tsc), "m"(*local_tsc)); + } while (after.mid_low != before.mid_low); + if (unlikely(stamp < before.low)) + before.full += AT91_TC_REG_MASK + 1; + before.low = stamp; + return before.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +cycle_t at91_ipipe_get_clocksource(struct clocksource *cs) +{ + return (cycle_t)__ipipe_mach_get_tsc(); +} + +static struct clocksource clksrc = { + .name = "at91_tc" __stringify(CONFIG_IPIPE_AT91_TC), + .rating = 250, + .read = at91_ipipe_get_clocksource, + .mask = CLOCKSOURCE_MASK(64), + .shift = 20, + .flags = CLOCK_SOURCE_IS_CONTINUOUS, +}; + +static void +at91_tc_set_mode(enum clock_event_mode mode, struct clock_event_device *dev) +{ + /* Disable the channel */ + at91_tc_write(AT91_TC_CCR, AT91_TC_CLKDIS); + + /* Disable all interrupts. */ + at91_tc_write(AT91_TC_IDR, ~0ul); + + if (mode == CLOCK_EVT_MODE_PERIODIC) { + unsigned long v; + +#ifndef CONFIG_ARCH_AT91SAM9263 + clk_enable(clk_get(NULL, "tc" + __stringify(CONFIG_IPIPE_AT91_TC) "_clk")); +#else /* AT91SAM9263 */ + clk_enable(clk_get(NULL, "tcb_clk")); +#endif /* AT91SAM9263 */ + + /* No Sync. */ + at91_tc_write(AT91_TC_BCR, 0); + + /* program NO signal on XCN */ + v = readl((void __iomem *) (AT91_VA_BASE_TCB0 + AT91_TC_BMR)); + v &= ~TCNXCNS(CONFIG_IPIPE_AT91_TC, 3); + v |= TCNXCNS(CONFIG_IPIPE_AT91_TC, 1); /* AT91_TC_TCNXCNS_NONE */ + writel(v, (void __iomem *) (AT91_VA_BASE_TCB0 + AT91_TC_BMR)); + + /* Use the clock selected by at91_timer_init as input clock. */ + at91_tc_write(AT91_TC_CMR, tc_timer_clock); + + /* Load the TC register C. */ + next_match = __ipipe_mach_ticks_per_jiffy; + write_RC(next_match); + + /* Enable CPCS interrupt. */ + at91_tc_write(AT91_TC_IER, AT91_TC_CPCS); + + /* Enable the channel. */ + at91_tc_write(AT91_TC_CCR, AT91_TC_CLKEN | AT91_TC_SWTRG); + + at91_timer_initialized = 1; + } +} + +/* + * Reprogram the timer + */ +void __ipipe_mach_set_dec(unsigned long delay) +{ + unsigned short stamp; + unsigned long flags; + + if (delay > max_delta_ticks) + delay = max_delta_ticks; + + if (likely(delay > min_delta_ticks)) { + local_irq_save_hw(flags); + stamp = read_CV(); + ipipe_mach_update_tsc(stamp); + write_RC((stamp + delay) & AT91_TC_REG_MASK); + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(KERNEL_TIMER_IRQ_NUM); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clkevt.name); +} + +static struct clock_event_device clkevt = { + .name = "at91_tc" __stringify(CONFIG_IPIPE_AT91_TC), + .features = CLOCK_EVT_FEAT_PERIODIC, + .shift = 32, + .rating = 250, + .set_mode = at91_tc_set_mode, +}; + +void __ipipe_mach_release_timer(void) +{ + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return (read_RC() - read_CV()) & AT91_TC_REG_MASK; +} + +void __init at91_timer_init(void) +{ + unsigned char tc_divisors[] = { 2, 8, 32, 128, 0, }; + unsigned master_freq, divisor = 0, divided_freq = 0; + unsigned long long wrap_ns; + + /* Disable (boot loader) timer interrupts. */ +#if defined(CONFIG_ARCH_AT91RM9200) + at91_sys_write(AT91_ST_IDR, AT91_ST_PITS | AT91_ST_WDOVF | AT91_ST_RTTINC | AT91_ST_ALMS); + (void) at91_sys_read(AT91_ST_SR); /* Clear any pending interrupts */ +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9261) \ + || defined(CONFIG_ARCH_AT91SAM9263) || defined(CONFIG_ARCH_AT91SAM9RL) \ + || defined(CONFIG_ARCH_AT91SAM9G20) + at91_sys_write(AT91_PIT_MR, 0); + + /* Clear any pending interrupts */ + (void) at91_sys_read(AT91_PIT_PIVR); +#endif /* CONFIG_ARCH_AT91SAM926x */ + + master_freq = clk_get_rate(clk_get(NULL, "mck")); + /* Find the first frequency above 1 MHz */ + for (tc_timer_clock = ARRAY_SIZE(tc_divisors) - 1; + tc_timer_clock >= 0; tc_timer_clock--) { + divisor = tc_divisors[tc_timer_clock]; + divided_freq = (divisor + ? master_freq / divisor : AT91_SLOW_CLOCK); + if (divided_freq > 1000000) + break; + } + + wrap_ns = (unsigned long long) (AT91_TC_REG_MASK + 1) * NSEC_PER_SEC; + do_div(wrap_ns, divided_freq); + + if (divided_freq < 1000000) + printk(KERN_INFO "AT91 I-pipe warning: could not find a" + " frequency greater than 1MHz\n"); + + printk(KERN_INFO "AT91 I-pipe timer: div: %u, freq: %u.%06u MHz, wrap: " + "%u.%06u ms\n", divisor, + divided_freq / 1000000, divided_freq % 1000000, + (unsigned) wrap_ns / 1000000, (unsigned) wrap_ns % 1000000); + + /* Add a 1ms margin. It means that when an interrupt occurs, update_tsc + must be called within 1ms. update_tsc is called by acktimer when no + higher domain handles the timer, and called through set_dec when a + higher domain handles the timer. */ + wrap_ns -= 1000000; + /* Set up the interrupt. */ + setup_irq(KERNEL_TIMER_IRQ_NUM, &at91_timer_irq); + +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; +#endif /* CONFIG_SMP */ + + clkevt.mult = div_sc(divided_freq, NSEC_PER_SEC, clkevt.shift); + clkevt.max_delta_ns = wrap_ns; + clkevt.min_delta_ns = 2000; + clkevt.cpumask = cpumask_of(0); + clockevents_register_device(&clkevt); + + clksrc.mult = clocksource_hz2mult(divided_freq, clksrc.shift); + clocksource_register(&clksrc); + + __ipipe_mach_ticks_per_jiffy = (divided_freq + HZ/2) / HZ; + max_delta_ticks = (wrap_ns * clkevt.mult) >> clkevt.shift; + min_delta_ticks = ((unsigned long long) clkevt.min_delta_ns + * clkevt.mult) >> clkevt.shift; +} + +#ifdef CONFIG_ARCH_AT91RM9200 +struct sys_timer at91rm9200_timer = { +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9261) \ + || defined(CONFIG_ARCH_AT91SAM9263) || defined(CONFIG_ARCH_AT91SAM9RL) \ + || defined(CONFIG_ARCH_AT91SAM9G20) +struct sys_timer at91sam926x_timer = { +#elif defined(CONFIG_ARCH_AT91X40) +struct sys_timer at91x40_timer = { +#else +#error "Unknown machine" +#endif + .init = at91_timer_init, + .suspend = NULL, + .resume = NULL, +}; diff --git a/arch/arm/mach-at91/at91rm9200.c b/arch/arm/mach-at91/at91rm9200.c index 2e9ecad..2dea6d7 100644 --- a/arch/arm/mach-at91/at91rm9200.c +++ b/arch/arm/mach-at91/at91rm9200.c @@ -33,7 +33,14 @@ static struct map_desc at91rm9200_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91RM9200_BASE_EMAC), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ + }, { .virtual = AT91_IO_VIRT_BASE - AT91RM9200_SRAM_SIZE, .pfn = __phys_to_pfn(AT91RM9200_SRAM_BASE), .length = AT91RM9200_SRAM_SIZE, @@ -300,6 +307,7 @@ void __init at91rm9200_initialize(unsigned long main_clock, unsigned short banks * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91rm9200_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller (FIQ) */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -332,6 +340,42 @@ static unsigned int at91rm9200_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller (IRQ4) */ 0, /* Advanced Interrupt Controller (IRQ5) */ 0 /* Advanced Interrupt Controller (IRQ6) */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 6, /* System Peripheral */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, /* Parallel IO Controller D */ + 5, /* USART 0 */ + 5, /* USART 1 */ + 5, /* USART 2 */ + 5, /* USART 3 */ + 0, /* Multimedia Card Interface */ + 3, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 5, /* Serial Peripheral Interface */ + 4, /* Serial Synchronous Controller */ + 4, /* Serial Synchronous Controller */ + 4, /* Serial Synchronous Controller */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 0, /* Timer Counter 3 */ + 0, /* Timer Counter 4 */ + 0, /* Timer Counter 5 */ + 2, /* USB Host port */ + 2, /* Ethernet MAC */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0 /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91rm9200_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9260.c b/arch/arm/mach-at91/at91sam9260.c index 0894f10..63f05ad 100644 --- a/arch/arm/mach-at91/at91sam9260.c +++ b/arch/arm/mach-at91/at91sam9260.c @@ -31,6 +31,13 @@ static struct map_desc at91sam9260_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ } }; @@ -350,6 +357,7 @@ void __init at91sam9260_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9260_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -382,6 +390,42 @@ static unsigned int at91sam9260_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 7, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, /* Analog-to-Digital Converter */ + 6, /* USART 0 */ + 6, /* USART 1 */ + 6, /* USART 2 */ + 0, /* Multimedia Card Interface */ + 4, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 6, /* Serial Peripheral Interface 0 */ + 6, /* Serial Peripheral Interface 1 */ + 5, /* Serial Synchronous Controller */ + 0, + 0, + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 3, /* USB Host port */ + 3, /* Ethernet */ + 0, /* Image Sensor Interface */ + 6, /* USART 3 */ + 6, /* USART 4 */ + 6, /* USART 5 */ + 7, /* Timer Counter 3 */ + 7, /* Timer Counter 4 */ + 7, /* Timer Counter 5 */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9260_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9261.c b/arch/arm/mach-at91/at91sam9261.c index 4ecf379..8a13d5f 100644 --- a/arch/arm/mach-at91/at91sam9261.c +++ b/arch/arm/mach-at91/at91sam9261.c @@ -31,6 +31,13 @@ static struct map_desc at91sam9261_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, }; @@ -306,6 +313,7 @@ void __init at91sam9261_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9261_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -338,6 +346,42 @@ static unsigned int at91sam9261_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 7, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C */ + 0, + 6, /* USART 0 */ + 6, /* USART 1 */ + 6, /* USART 2 */ + 0, /* Multimedia Card Interface */ + 4, /* USB Device Port */ + 0, /* Two-Wire Interface */ + 6, /* Serial Peripheral Interface 0 */ + 6, /* Serial Peripheral Interface 1 */ + 5, /* Serial Synchronous Controller 0 */ + 5, /* Serial Synchronous Controller 1 */ + 5, /* Serial Synchronous Controller 2 */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 3, /* USB Host port */ + 3, /* LCD Controller */ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9261_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9263.c b/arch/arm/mach-at91/at91sam9263.c index 942792d..c043f3a 100644 --- a/arch/arm/mach-at91/at91sam9263.c +++ b/arch/arm/mach-at91/at91sam9263.c @@ -30,6 +30,13 @@ static struct map_desc at91sam9263_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, { .virtual = AT91_IO_VIRT_BASE - AT91SAM9263_SRAM0_SIZE, .pfn = __phys_to_pfn(AT91SAM9263_SRAM0_BASE), @@ -311,6 +318,7 @@ void __init at91sam9263_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9263_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller (FIQ) */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -343,6 +351,42 @@ static unsigned int at91sam9263_default_irq_priority[NR_AIC_IRQS] __initdata = { 2, /* USB Host port */ 0, /* Advanced Interrupt Controller (IRQ0) */ 0, /* Advanced Interrupt Controller (IRQ1) */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller (FIQ) */ + 6, /* System Peripherals */ + 0, /* Parallel IO Controller A */ + 0, /* Parallel IO Controller B */ + 0, /* Parallel IO Controller C, D and E */ + 0, + 0, + 5, /* USART 0 */ + 5, /* USART 1 */ + 5, /* USART 2 */ + 0, /* Multimedia Card Interface 0 */ + 0, /* Multimedia Card Interface 1 */ + 3, /* CAN */ + 0, /* Two-Wire Interface */ + 5, /* Serial Peripheral Interface 0 */ + 5, /* Serial Peripheral Interface 1 */ + 4, /* Serial Synchronous Controller 0 */ + 4, /* Serial Synchronous Controller 1 */ + 5, /* AC97 Controller */ + 7, /* Timer Counter 0, 1 and 2 */ + 0, /* Pulse Width Modulation Controller */ + 2, /* Ethernet */ + 0, + 0, /* 2D Graphic Engine */ + 2, /* USB Device Port */ + 0, /* Image Sensor Interface */ + 2, /* LDC Controller */ + 0, /* DMA Controller */ + 0, + 2, /* USB Host port */ + 0, /* Advanced Interrupt Controller (IRQ0) */ + 0, /* Advanced Interrupt Controller (IRQ1) */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9263_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/at91sam9rl.c b/arch/arm/mach-at91/at91sam9rl.c index 211c5c1..9cdad31 100644 --- a/arch/arm/mach-at91/at91sam9rl.c +++ b/arch/arm/mach-at91/at91sam9rl.c @@ -30,6 +30,13 @@ static struct map_desc at91sam9rl_io_desc[] __initdata = { .pfn = __phys_to_pfn(AT91_BASE_SYS), .length = SZ_16K, .type = MT_DEVICE, +#ifdef CONFIG_IPIPE + }, { + .virtual = AT91_VA_BASE_TCB0, + .pfn = __phys_to_pfn(AT91_BASE_TCB0), + .length = SZ_16K, + .type = MT_DEVICE, +#endif /* CONFIG_IPIPE */ }, }; @@ -303,6 +310,7 @@ void __init at91sam9rl_initialize(unsigned long main_clock) * The default interrupt priority levels (0 = lowest, 7 = highest). */ static unsigned int at91sam9rl_default_irq_priority[NR_AIC_IRQS] __initdata = { +#ifndef CONFIG_IPIPE 7, /* Advanced Interrupt Controller */ 7, /* System Peripherals */ 1, /* Parallel IO Controller A */ @@ -335,6 +343,42 @@ static unsigned int at91sam9rl_default_irq_priority[NR_AIC_IRQS] __initdata = { 0, 0, 0, /* Advanced Interrupt Controller */ +#else /* CONFIG_IPIPE */ +/* Give the highest priority to TC, since they are used as timer interrupt by + I-pipe. */ + 7, /* Advanced Interrupt Controller */ + 6, /* System Peripherals */ + 1, /* Parallel IO Controller A */ + 1, /* Parallel IO Controller B */ + 1, /* Parallel IO Controller C */ + 1, /* Parallel IO Controller D */ + 4, /* USART 0 */ + 4, /* USART 1 */ + 4, /* USART 2 */ + 4, /* USART 3 */ + 0, /* Multimedia Card Interface */ + 5, /* Two-Wire Interface 0 */ + 5, /* Two-Wire Interface 1 */ + 4, /* Serial Peripheral Interface */ + 3, /* Serial Synchronous Controller 0 */ + 3, /* Serial Synchronous Controller 1 */ + 7, /* Timer Counter 0 */ + 7, /* Timer Counter 1 */ + 7, /* Timer Counter 2 */ + 0, + 0, /* Touch Screen Controller */ + 0, /* DMA Controller */ + 2, /* USB Device High speed port */ + 2, /* LCD Controller */ + 5, /* AC97 Controller */ + 0, + 0, + 0, + 0, + 0, + 0, + 0, /* Advanced Interrupt Controller */ +#endif /*CONFIG_IPIPE */ }; void __init at91sam9rl_init_interrupts(unsigned int priority[NR_AIC_IRQS]) diff --git a/arch/arm/mach-at91/gpio.c b/arch/arm/mach-at91/gpio.c index ae4772e..9b3c914 100644 --- a/arch/arm/mach-at91/gpio.c +++ b/arch/arm/mach-at91/gpio.c @@ -19,12 +19,20 @@ #include <linux/list.h> #include <linux/module.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <mach/at91_pio.h> #include <mach/gpio.h> #include <asm/gpio.h> +#ifdef CONFIG_IPIPE +#include <asm/irq.h> + +#ifdef __IPIPE_FEATURE_PIC_MUTE +DEFINE_PER_CPU(__ipipe_irqbits_t, __ipipe_muted_irqs); +#endif /* __IPIPE_FEATURE_PIC_MUTE */ +#endif /* CONFIG_IPIPE */ #include "generic.h" @@ -375,6 +383,10 @@ static int gpio_irq_type(unsigned pin, unsigned type) static struct irq_chip gpio_irqchip = { .name = "GPIO", +#ifdef CONFIG_IPIPE + .ack = gpio_irq_mask, + .mask_ack = gpio_irq_mask, +#endif /* CONFIG_IPIPE */ .mask = gpio_irq_mask, .unmask = gpio_irq_unmask, .set_type = gpio_irq_type, @@ -422,7 +434,7 @@ static void gpio_irq_handler(unsigned irq, struct irq_desc *desc) gpio_irq_mask(pin); } else - generic_handle_irq(pin); + ipipe_handle_irq_cond(pin); } pin++; gpio++; @@ -433,6 +445,38 @@ static void gpio_irq_handler(unsigned irq, struct irq_desc *desc) /* now it may re-trigger */ } +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) +void ipipe_mute_pic(void) +{ + unsigned long unmasked, muted; + unsigned i; + + for (i = 0; i < gpio_banks; i++) { + unmasked = __raw_readl(gpio_chip[i].regbase + PIO_IMR); + muted = unmasked & __ipipe_irqbits[i + 1]; + __raw_get_cpu_var(__ipipe_muted_irqs)[i + 1] = muted; + __raw_writel(muted, gpio_chip[i].regbase + PIO_IDR); + } + + unmasked = at91_sys_read(AT91_AIC_IMR); + muted = unmasked & __ipipe_irqbits[0]; + __raw_get_cpu_var(__ipipe_muted_irqs)[0] = muted; + at91_sys_write(AT91_AIC_IDCR, muted); +} + +void ipipe_unmute_pic(void) +{ + unsigned i; + + at91_sys_write(AT91_AIC_IECR, __raw_get_cpu_var(__ipipe_muted_irqs)[0]); + for (i = 0; i < gpio_banks; i++) { + unsigned long muted; + muted = __raw_get_cpu_var(__ipipe_muted_irqs)[i + 1]; + __raw_writel(muted, gpio_chip[i].regbase + PIO_IER); + } +} +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ + /*--------------------------------------------------------------------------*/ #ifdef CONFIG_DEBUG_FS @@ -509,6 +553,10 @@ void __init at91_gpio_irq_setup(void) unsigned pioc, pin; struct at91_gpio_chip *this, *prev; +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) + memset(__ipipe_irqbits, ~0, sizeof(__ipipe_irqbits)); +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ + for (pioc = 0, pin = PIN_BASE, this = gpio_chip, prev = NULL; pioc++ < gpio_banks; prev = this, this++) { @@ -538,6 +586,9 @@ void __init at91_gpio_irq_setup(void) set_irq_chip_data(id, this); set_irq_chained_handler(id, gpio_irq_handler); +#if defined(CONFIG_IPIPE) && defined(__IPIPE_FEATURE_PIC_MUTE) + __ipipe_irqbits[0] &= ~(1 << id); +#endif /* CONFIG_IPIPE && __IPIPE_FEATURE_PIC_MUTE */ } pr_info("AT91: %d gpio irqs in %d banks\n", pin - PIN_BASE, gpio_banks); } diff --git a/arch/arm/mach-at91/include/mach/hardware.h b/arch/arm/mach-at91/include/mach/hardware.h index a0df8b0..05ff930 100644 --- a/arch/arm/mach-at91/include/mach/hardware.h +++ b/arch/arm/mach-at91/include/mach/hardware.h @@ -63,6 +63,25 @@ #define AT91_VA_BASE_SYS AT91_IO_P2V(AT91_BASE_SYS) #define AT91_VA_BASE_EMAC AT91_IO_P2V(AT91RM9200_BASE_EMAC) +#ifdef CONFIG_IPIPE +#if defined(CONFIG_ARCH_AT91RM9200) +#define AT91_BASE_TCB0 AT91RM9200_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9260) || defined(CONFIG_ARCH_AT91SAM9G20) +#define AT91_BASE_TCB0 AT91SAM9260_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9261) +#define AT91_BASE_TCB0 AT91SAM9261_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9263) +#define AT91_BASE_TCB0 AT91SAM9263_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91SAM9RL) +#define AT91_BASE_TCB0 AT91SAM9RL_BASE_TCB0 +#elif defined(CONFIG_ARCH_AT91X40) +#define AT91_BASE_TCB0 (AT91_BASE_SYS + AT91_TC) +#else +#error "AT91 processor unsupported by Adeos" +#endif +#define AT91_VA_BASE_TCB0 AT91_IO_P2V(AT91_BASE_TCB0) +#endif + /* Internal SRAM is mapped below the IO devices */ #define AT91_SRAM_MAX SZ_1M #define AT91_VIRT_BASE (AT91_IO_VIRT_BASE - AT91_SRAM_MAX) diff --git a/arch/arm/mach-at91/include/mach/irqs.h b/arch/arm/mach-at91/include/mach/irqs.h index 36bd55f..cedaab2 100644 --- a/arch/arm/mach-at91/include/mach/irqs.h +++ b/arch/arm/mach-at91/include/mach/irqs.h @@ -45,4 +45,6 @@ /* FIQ is AIC source 0. */ #define FIQ_START AT91_ID_FIQ +#define __IPIPE_FEATURE_PIC_MUTE + #endif diff --git a/arch/arm/mach-at91/include/mach/timex.h b/arch/arm/mach-at91/include/mach/timex.h index 31ac2d9..29a70e2 100644 --- a/arch/arm/mach-at91/include/mach/timex.h +++ b/arch/arm/mach-at91/include/mach/timex.h @@ -82,6 +82,6 @@ #define AT91X40_MASTER_CLOCK 40000000 #define CLOCK_TICK_RATE (AT91X40_MASTER_CLOCK) -#endif +#endif /* arch specific */ #endif diff --git a/arch/arm/mach-at91/irq.c b/arch/arm/mach-at91/irq.c index da3494a..d375427 100644 --- a/arch/arm/mach-at91/irq.c +++ b/arch/arm/mach-at91/irq.c @@ -121,6 +121,9 @@ static struct irq_chip at91_aic_chip = { .name = "AIC", .ack = at91_aic_mask_irq, .mask = at91_aic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = at91_aic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = at91_aic_unmask_irq, .set_type = at91_aic_set_type, .set_wake = at91_aic_set_wake, diff --git a/arch/arm/mach-integrator/core.c b/arch/arm/mach-integrator/core.c index a0f60e5..84c99a7 100644 --- a/arch/arm/mach-integrator/core.c +++ b/arch/arm/mach-integrator/core.c @@ -2,6 +2,7 @@ * linux/arch/arm/mach-integrator/core.c * * Copyright (C) 2000-2003 Deep Blue Solutions Ltd + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2, as @@ -230,53 +231,62 @@ EXPORT_SYMBOL(cm_control); /* * How long is the timer interval? */ -#define TIMER_INTERVAL (TICKS_PER_uSEC * mSEC_10) -#if TIMER_INTERVAL >= 0x100000 -#define TICKS2USECS(x) (256 * (x) / TICKS_PER_uSEC) -#elif TIMER_INTERVAL >= 0x10000 -#define TICKS2USECS(x) (16 * (x) / TICKS_PER_uSEC) -#else #define TICKS2USECS(x) ((x) / TICKS_PER_uSEC) -#endif static unsigned long timer_reload; +static unsigned long timer_interval; +static unsigned long timer_lxlost; +static int tscok; + +#ifdef CONFIG_IPIPE +int __ipipe_mach_timerint = IRQ_TIMERINT1; +static unsigned long long __ipipe_mach_tsc; +static IPIPE_DEFINE_SPINLOCK(timer_lock); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} +#endif /* - * Returns number of ms since last clock interrupt. Note that interrupts - * will have been disabled by do_gettimeoffset() + * Called with IRQ disabled from do_gettimeofday(). */ -unsigned long integrator_gettimeoffset(void) +static inline unsigned long integrator_getticksoffset(void) { - unsigned long ticks1, ticks2, status; + unsigned long ticks; - /* - * Get the current number of ticks. Note that there is a race - * condition between us reading the timer and checking for - * an interrupt. We get around this by ensuring that the - * counter has not reloaded between our two reads. - */ - ticks2 = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - do { - ticks1 = ticks2; - status = __raw_readl(VA_IC_BASE + IRQ_RAW_STATUS); - ticks2 = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - } while (ticks2 > ticks1); + if (!tscok) + return 0; - /* - * Number of ticks since last interrupt. - */ - ticks1 = timer_reload - ticks2; + ticks = readl(TIMER1_VA_BASE + TIMER_VALUE) & 0xffff; - /* - * Interrupt pending? If so, we've reloaded once already. - */ - if (status & (1 << IRQ_TIMERINT1)) - ticks1 += timer_reload; + if (ticks > timer_reload) + ticks = 0xffff + timer_reload - ticks; + else + ticks = timer_reload - ticks; + if (timer_interval < 0x10000) + return ticks; + else if (timer_interval < 0x100000) + return ticks * 16; + else + return ticks * 256; +} + +/* + * Returns number of ms since last clock interrupt. Note that interrupts + * will have been disabled by do_gettimeoffset() + */ +unsigned long integrator_gettimeoffset(void) +{ /* * Convert the ticks to usecs */ - return TICKS2USECS(ticks1); + return TICKS2USECS(timer_lxlost + integrator_getticksoffset()); } /* @@ -285,11 +295,22 @@ unsigned long integrator_gettimeoffset(void) static irqreturn_t integrator_timer_interrupt(int irq, void *dev_id) { + timer_lxlost = 0; + +#ifdef CONFIG_IPIPE /* - * clear the interrupt + * If Linux is the only domain, ack the timer and reprogram it */ - writel(1, TIMER1_VA_BASE + TIMER_INTCLR); + if (!__ipipe_mach_timerstolen) { + __ipipe_mach_tsc += integrator_getticksoffset(); +#else + writel(1, TIMER1_VA_BASE + TIMER_INTCLR); +#endif + writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); +#ifdef CONFIG_IPIPE + } +#endif timer_tick(); return IRQ_HANDLED; @@ -301,24 +322,30 @@ static struct irqaction integrator_timer_irq = { .handler = integrator_timer_interrupt, }; -/* - * Set up timer interrupt, and return the current time in seconds. - */ -void __init integrator_time_init(unsigned long reload, unsigned int ctrl) +static inline void set_dec(unsigned long reload) { - unsigned int timer_ctrl = TIMER_CTRL_ENABLE | TIMER_CTRL_PERIODIC; + unsigned int ctrl = TIMER_CTRL_ENABLE | TIMER_CTRL_IE; timer_reload = reload; - timer_ctrl |= ctrl; + timer_interval = reload; - if (timer_reload > 0x100000) { + if (timer_reload >= 0x100000) { timer_reload >>= 8; - timer_ctrl |= TIMER_CTRL_DIV256; - } else if (timer_reload > 0x010000) { + ctrl |= TIMER_CTRL_DIV256; + } else if (timer_reload >= 0x010000) { timer_reload >>= 4; - timer_ctrl |= TIMER_CTRL_DIV16; + ctrl |= TIMER_CTRL_DIV16; } + writel(ctrl, TIMER1_VA_BASE + TIMER_CTRL); + writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); +} + +/* + * Set up timer interrupt, and return the current time in seconds. + */ +void __init integrator_time_init(unsigned long reload, unsigned int ctrl) +{ /* * Initialise to a known state (all timers off) */ @@ -326,12 +353,60 @@ void __init integrator_time_init(unsigned long reload, unsigned int ctrl) writel(0, TIMER1_VA_BASE + TIMER_CTRL); writel(0, TIMER2_VA_BASE + TIMER_CTRL); - writel(timer_reload, TIMER1_VA_BASE + TIMER_LOAD); - writel(timer_reload, TIMER1_VA_BASE + TIMER_VALUE); - writel(timer_ctrl, TIMER1_VA_BASE + TIMER_CTRL); + set_dec(reload); /* * Make irqs happen for the system timer */ setup_irq(IRQ_TIMERINT1, &integrator_timer_irq); + + tscok = 1; +} + +#ifdef CONFIG_IPIPE + +void __ipipe_mach_acktimer(void) +{ + writel(1, TIMER1_VA_BASE + TIMER_INTCLR); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + unsigned long long result; + unsigned long flags; + + local_irq_save_hw_notrace(flags); + spin_lock(&timer_lock); + result = __ipipe_mach_tsc + integrator_getticksoffset(); + spin_unlock(&timer_lock); + local_irq_restore_hw_notrace(flags); + return result; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +void __ipipe_mach_set_dec(unsigned long reload) +{ + unsigned long ticks; + unsigned long flags; + + spin_lock_irqsave(&timer_lock, flags); + ticks = integrator_getticksoffset(); + __ipipe_mach_tsc += ticks; + timer_lxlost += ticks; + + set_dec(reload); + spin_unlock_irqrestore(&timer_lock, flags); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return readl(TIMER1_VA_BASE + TIMER_VALUE); } +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-integrator/include/mach/entry-macro.S b/arch/arm/mach-integrator/include/mach/entry-macro.S index 7649c57..ab3623d 100644 --- a/arch/arm/mach-integrator/include/mach/entry-macro.S +++ b/arch/arm/mach-integrator/include/mach/entry-macro.S @@ -28,7 +28,11 @@ teq \irqstat, #0 ldreq \irqstat, [\base, #(INTEGRATOR_HDR_IC_OFFSET+IRQ_STATUS)] moveq \irqnr, #IRQ_CIC_START - +#ifdef CONFIG_IPIPE + tst \irqstat, #0x00000040 @ check IRQ_TIMERINT1 first + movne \irqnr, #6 + bne 1003f +#endif /* CONFIG_IPIPE */ 1001: tst \irqstat, #15 bne 1002f add \irqnr, \irqnr, #4 diff --git a/arch/arm/mach-integrator/include/mach/irqs.h b/arch/arm/mach-integrator/include/mach/irqs.h index 1fbe6d1..dea6390 100644 --- a/arch/arm/mach-integrator/include/mach/irqs.h +++ b/arch/arm/mach-integrator/include/mach/irqs.h @@ -79,4 +79,3 @@ #define IRQ_SIC_END 46 #define NR_IRQS 47 - diff --git a/arch/arm/mach-integrator/include/mach/platform.h b/arch/arm/mach-integrator/include/mach/platform.h index e00a262..3b39ce3 100644 --- a/arch/arm/mach-integrator/include/mach/platform.h +++ b/arch/arm/mach-integrator/include/mach/platform.h @@ -417,7 +417,7 @@ * Timer definitions * * Only use timer 1 & 2 - * (both run at 24MHz and will need the clock divider set to 16). + * (both run at 1MHZ on /CP and at 24MHz on /AP) * * Timer 0 runs at bus frequency and therefore could vary and currently * uHAL can't handle that. @@ -430,7 +430,12 @@ #define MAX_TIMER 2 #define MAX_PERIOD 699050 + +#ifdef CONFIG_ARCH_INTEGRATOR_CP +#define TICKS_PER_uSEC 1 +#else #define TICKS_PER_uSEC 24 +#endif /* * These are useconds NOT ticks. diff --git a/arch/arm/mach-integrator/include/mach/timex.h b/arch/arm/mach-integrator/include/mach/timex.h index 1dcb420..a04653a 100644 --- a/arch/arm/mach-integrator/include/mach/timex.h +++ b/arch/arm/mach-integrator/include/mach/timex.h @@ -21,6 +21,6 @@ */ /* - * ?? + * Timer rate */ -#define CLOCK_TICK_RATE (50000000 / 16) +#define CLOCK_TICK_RATE (1000000) diff --git a/arch/arm/mach-integrator/integrator_cp.c b/arch/arm/mach-integrator/integrator_cp.c index 3f35293..91d2eaa 100644 --- a/arch/arm/mach-integrator/integrator_cp.c +++ b/arch/arm/mach-integrator/integrator_cp.c @@ -2,6 +2,7 @@ * linux/arch/arm/mach-integrator/integrator_cp.c * * Copyright (C) 2003 Deep Blue Solutions Ltd + * Copyright (C) 2005 Stelian Pop. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -21,6 +22,7 @@ #include <linux/amba/clcd.h> #include <linux/amba/mmci.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <asm/clkdev.h> #include <mach/clkdev.h> @@ -161,6 +163,9 @@ static struct irq_chip cic_chip = { .name = "CIC", .ack = cic_mask_irq, .mask = cic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = cic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = cic_unmask_irq, }; @@ -180,6 +185,9 @@ static struct irq_chip pic_chip = { .name = "PIC", .ack = pic_mask_irq, .mask = pic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = pic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = pic_unmask_irq, }; @@ -199,6 +207,9 @@ static struct irq_chip sic_chip = { .name = "SIC", .ack = sic_mask_irq, .mask = sic_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = sic_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = sic_unmask_irq, }; @@ -218,7 +229,7 @@ sic_handle_irq(unsigned int irq, struct irq_desc *desc) irq += IRQ_SIC_START; - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } while (status); } @@ -571,9 +582,14 @@ static void __init intcp_init(void) #define TIMER_CTRL_IE (1 << 5) /* Interrupt Enable */ +#ifdef CONFIG_IPIPE +unsigned int __ipipe_mach_ticks_per_jiffy = 1000000 * TICKS_PER_uSEC / HZ; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); +#endif + static void __init intcp_timer_init(void) { - integrator_time_init(1000000 / HZ, TIMER_CTRL_IE); + integrator_time_init(1000000 * TICKS_PER_uSEC / HZ, TIMER_CTRL_IE); } static struct sys_timer cp_timer = { diff --git a/arch/arm/mach-ixp4xx/common.c b/arch/arm/mach-ixp4xx/common.c index 3bbf40f..cfb1f44 100644 --- a/arch/arm/mach-ixp4xx/common.c +++ b/arch/arm/mach-ixp4xx/common.c @@ -45,6 +45,65 @@ static void __init ixp4xx_clocksource_init(void); static void __init ixp4xx_clockevent_init(void); static struct clock_event_device clockevent_ixp4xx; +#ifdef CONFIG_IPIPE +#include <linux/ipipe.h> + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif + +int __ipipe_mach_timerint = IRQ_IXP4XX_TIMER1; +int __ipipe_mach_timerstolen = 0; +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_timerint); +EXPORT_SYMBOL(__ipipe_mach_timerstolen); +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int ixp4xx_timer_initialized; + +#define ONE_SHOT_ENABLE (IXP4XX_OST_ENABLE|IXP4XX_OST_ONE_SHOT) + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = + (unsigned *) + (IXP4XX_TIMER_BASE_PHYS + IXP4XX_OSTS_OFFSET); + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + /************************************************************************* * IXP4xx chipset I/O mapping *************************************************************************/ @@ -269,8 +328,12 @@ static irqreturn_t ixp4xx_timer_interrupt(int irq, void *dev_id) { struct clock_event_device *evt = dev_id; +#ifndef CONFIG_IPIPE /* Clear Pending Interrupt by writing '1' to it */ *IXP4XX_OSST = IXP4XX_OSST_TIMER_1_PEND; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ evt->event_handler(evt); @@ -295,6 +358,14 @@ void __init ixp4xx_timer_init(void) /* Reset time-stamp counter */ *IXP4XX_OSTS = 0; +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + ixp4xx_timer_initialized = 1; +#endif /* Connect the interrupt handler and enable the interrupt */ setup_irq(IRQ_IXP4XX_TIMER1, &ixp4xx_timer_irq); @@ -470,6 +541,13 @@ static void ixp4xx_set_mode(enum clock_event_mode mode, *IXP4XX_OSRT1 = osrt | opts; } +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_ixp4xx.name); +} +#endif /* CONFIG_IPIPE */ + static struct clock_event_device clockevent_ixp4xx = { .name = "ixp4xx timer1", .features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT, @@ -491,3 +569,98 @@ static void __init ixp4xx_clockevent_init(void) clockevents_register_device(&clockevent_ixp4xx); } + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + /* Clear Pending Interrupt by writing '1' to it */ + *IXP4XX_OSST = IXP4XX_OSST_TIMER_1_PEND; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = *IXP4XX_OSTS; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(ixp4xx_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = *IXP4XX_OSTS; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + return result.full; + } + + return 0; +} + +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + * + * The timer is aperiodic (most of the time) when running Xenomai, so + * __ipipe_mach_set_dec is called for each timer tick and programs the + * timer hardware for the next tick. + * + */ +#define MIN_DELAY 333 /* 5 usec with the 66.66 MHz system clock */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + unsigned long flags; + if (delay > MIN_DELAY) { + local_irq_save_hw(flags); + *IXP4XX_OSRT1 = delay | ONE_SHOT_ENABLE; + local_irq_restore_hw(flags); + } else { + ipipe_trigger_irq(IRQ_IXP4XX_TIMER1); + } +} + +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +/* + * This returns the number of clock ticks remaining. + */ +unsigned long __ipipe_mach_get_dec(void) +{ + return(*IXP4XX_OST1); /* remaining */ +} + +EXPORT_SYMBOL(__ipipe_mach_get_dec); + +void __ipipe_mach_release_timer(void) +{ + unsigned long flags; + + local_irq_save_hw(flags); + ixp4xx_set_mode(clockevent_ixp4xx.mode, &clockevent_ixp4xx); + if (clockevent_ixp4xx.mode == CLOCK_EVT_MODE_ONESHOT) + ixp4xx_set_next_event(LATCH, &clockevent_ixp4xx); + local_irq_restore_hw(flags); +} + +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx25/devices.c b/arch/arm/mach-mx25/devices.c index 9fdeea1..89f74e8 100644 --- a/arch/arm/mach-mx25/devices.c +++ b/arch/arm/mach-mx25/devices.c @@ -19,6 +19,7 @@ #include <linux/platform_device.h> #include <linux/dma-mapping.h> #include <linux/gpio.h> +#include <mach/common.h> #include <mach/mx25.h> #include <mach/irqs.h> @@ -438,3 +439,14 @@ struct platform_device mx25_fec_device = { .num_resources = ARRAY_SIZE(mx25_fec_resources), .resource = mx25_fec_resources, }; + +#ifdef CONFIG_IPIPE +static int post_cpu_init(void) +{ + ipipe_mach_allow_hwtimer_uaccess(MX25_AIPS1_BASE_ADDR_VIRT, + MX25_AIPS2_BASE_ADDR_VIRT); + return 0; +} + +postcore_initcall(post_cpu_init); +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx3/devices.c b/arch/arm/mach-mx3/devices.c index 6adb586..9040c13 100644 --- a/arch/arm/mach-mx3/devices.c +++ b/arch/arm/mach-mx3/devices.c @@ -22,6 +22,7 @@ #include <linux/platform_device.h> #include <linux/serial.h> #include <linux/gpio.h> +#include <mach/common.h> #include <mach/hardware.h> #include <mach/irqs.h> #include <mach/common.h> @@ -603,3 +604,15 @@ static int mx3_devices_init(void) } subsys_initcall(mx3_devices_init); + +#ifdef CONFIG_IPIPE +static int post_cpu_init(void) +{ + if (cpu_is_mx31()) + ipipe_mach_allow_hwtimer_uaccess(AIPS1_BASE_ADDR_VIRT, + AIPS2_BASE_ADDR_VIRT); + return 0; +} + +postcore_initcall(post_cpu_init); +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-mx3/mx31ads.c b/arch/arm/mach-mx3/mx31ads.c index 938c549..b7c2139 100644 --- a/arch/arm/mach-mx3/mx31ads.c +++ b/arch/arm/mach-mx3/mx31ads.c @@ -25,6 +25,7 @@ #include <linux/gpio.h> #include <linux/i2c.h> #include <linux/irq.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach-types.h> @@ -134,7 +135,7 @@ static void mx31ads_expio_irq_handler(u32 irq, struct irq_desc *desc) if ((int_valid & 1) == 0) continue; - generic_handle_irq(expio_irq); + ipipe_handle_irq_cond(expio_irq); } } diff --git a/arch/arm/mach-mx3/mx31pdk.c b/arch/arm/mach-mx3/mx31pdk.c index 18715f1..aa47303 100644 --- a/arch/arm/mach-mx3/mx31pdk.c +++ b/arch/arm/mach-mx3/mx31pdk.c @@ -23,6 +23,7 @@ #include <linux/gpio.h> #include <linux/smsc911x.h> #include <linux/platform_device.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach-types.h> @@ -108,7 +109,7 @@ static void mx31pdk_expio_irq_handler(uint32_t irq, struct irq_desc *desc) for (; int_valid != 0; int_valid >>= 1, expio_irq++) { if ((int_valid & 1) == 0) continue; - generic_handle_irq(expio_irq); + ipipe_handle_irq_cond(expio_irq); } } diff --git a/arch/arm/mach-omap2/irq.c b/arch/arm/mach-omap2/irq.c index 26aeef5..5ed22fa 100644 --- a/arch/arm/mach-omap2/irq.c +++ b/arch/arm/mach-omap2/irq.c @@ -148,6 +148,7 @@ static struct irq_chip omap_irq_chip = { .name = "INTC", .ack = omap_mask_ack_irq, .mask = omap_mask_irq, + .mask_ack = omap_mask_ack_irq, .unmask = omap_unmask_irq, }; @@ -167,8 +168,14 @@ static void __init omap_irq_bank_init_one(struct omap_irq_bank *bank) while (!(intc_bank_read_reg(bank, INTC_SYSSTATUS) & 0x1)) /* Wait for reset to complete */; +#ifndef CONFIG_IPIPE /* Enable autoidle */ intc_bank_write_reg(1 << 0, bank, INTC_SYSCONFIG); +#else /* !CONFIG_IPIPE */ + /* Disable autoidle */ + intc_bank_write_reg(0, bank, INTC_SYSCONFIG); + intc_bank_write_reg(0x2, bank, INTC_IDLE); +#endif /* CONFIG_IPIPE */ } int omap_irq_pending(void) diff --git a/arch/arm/mach-omap2/timer-gp.c b/arch/arm/mach-omap2/timer-gp.c index cd04dea..389df09 100644 --- a/arch/arm/mach-omap2/timer-gp.c +++ b/arch/arm/mach-omap2/timer-gp.c @@ -35,6 +35,7 @@ #include <linux/irq.h> #include <linux/clocksource.h> #include <linux/clockchips.h> +#include <linux/ipipe.h> #include <asm/mach/time.h> #include <plat/dmtimer.h> @@ -48,13 +49,74 @@ static struct clock_event_device clockevent_gpt; static u8 __initdata gptimer_id = 1; static u8 __initdata inited; struct omap_dm_timer *gptimer_wakeup; +#ifndef CONFIG_OMAP_32K_TIMER +static struct omap_dm_timer *gpt_clocksource; +#endif /* !CONFIG_OMAP_32K_TIMER */ + +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int omap2_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *) + omap_dm_timer_get_phys_counter_addr(gpt_clocksource); + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ static irqreturn_t omap2_gp_timer_interrupt(int irq, void *dev_id) { - struct omap_dm_timer *gpt = (struct omap_dm_timer *)dev_id; struct clock_event_device *evt = &clockevent_gpt; +#ifndef CONFIG_IPIPE + struct omap_dm_timer *gpt = (struct omap_dm_timer *)dev_id; omap_dm_timer_write_status(gpt, OMAP_TIMER_INT_OVERFLOW); +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ evt->event_handler(evt); return IRQ_HANDLED; @@ -126,6 +188,13 @@ int __init omap2_gp_clockevent_set_gptimer(u8 id) return 0; } +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_gpt.name); +} +#endif /* CONFIG_IPIPE */ + static void __init omap2_gp_clockevent_init(void) { u32 tick_rate; @@ -158,6 +227,12 @@ static void __init omap2_gp_clockevent_init(void) gptimer_id, tick_rate); omap2_gp_timer_irq.dev_id = (void *)gptimer; + +#ifdef CONFIG_IPIPE + __ipipe_mach_timerint = omap_dm_timer_get_irq(gptimer); + __ipipe_mach_ticks_per_jiffy = (tick_rate + HZ / 2) / HZ; +#endif /* CONFIG_IPIPE */ + setup_irq(omap_dm_timer_get_irq(gptimer), &omap2_gp_timer_irq); omap_dm_timer_set_int_enable(gptimer, OMAP_TIMER_INT_OVERFLOW); @@ -168,7 +243,6 @@ static void __init omap2_gp_clockevent_init(void) clockevent_gpt.min_delta_ns = clockevent_delta2ns(3, &clockevent_gpt); /* Timer internal resynch latency. */ - clockevent_gpt.cpumask = cpumask_of(0); clockevents_register_device(&clockevent_gpt); } @@ -176,10 +250,10 @@ static void __init omap2_gp_clockevent_init(void) /* Clocksource code */ #ifdef CONFIG_OMAP_32K_TIMER -/* +/* * When 32k-timer is enabled, don't use GPTimer for clocksource * instead, just leave default clocksource which uses the 32k - * sync counter. See clocksource setup in see plat-omap/common.c. + * sync counter. See clocksource setup in see plat-omap/common.c. */ static inline void __init omap2_gp_clocksource_init(void) {} @@ -187,7 +261,6 @@ static inline void __init omap2_gp_clocksource_init(void) {} /* * clocksource */ -static struct omap_dm_timer *gpt_clocksource; static cycle_t clocksource_read_cycles(struct clocksource *cs) { return (cycle_t)omap_dm_timer_read_counter(gpt_clocksource); @@ -225,6 +298,16 @@ static void __init omap2_gp_clocksource_init(void) clocksource_gpt.mult = clocksource_khz2mult(tick_rate/1000, clocksource_gpt.shift); + +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + omap2_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + if (clocksource_register(&clocksource_gpt)) printk(err2, clocksource_gpt.name); } @@ -245,3 +328,79 @@ static void __init omap2_gp_timer_init(void) struct sys_timer omap_timer = { .init = omap2_gp_timer_init, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + omap_dm_timer_write_status(gptimer, OMAP_TIMER_INT_OVERFLOW); + omap_dm_timer_read_status(gptimer); +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + if (likely(omap2_timer_initialized)) { + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = omap_dm_timer_read_counter(gpt_clocksource); + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); + } +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(omap2_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n" : + "=r"(result.full) : "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = omap_dm_timer_read_counter(gpt_clocksource); + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > 3) + omap_dm_timer_set_load_start(gptimer, 0, 0xffffffff - delay); + else + ipipe_trigger_irq(__ipipe_mach_timerint); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + struct clock_event_device *ckdev = &clockevent_gpt; + ckdev->set_mode(ckdev->mode, ckdev); + if (ckdev->mode == CLOCK_EVT_MODE_ONESHOT) + ckdev->set_next_event(__ipipe_mach_ticks_per_jiffy, ckdev); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return 0xffffffff - omap_dm_timer_read_counter(gptimer); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-pxa/irq.c b/arch/arm/mach-pxa/irq.c index 1beb40f..3b27f7b 100644 --- a/arch/arm/mach-pxa/irq.c +++ b/arch/arm/mach-pxa/irq.c @@ -51,6 +51,9 @@ static struct irq_chip pxa_internal_irq_chip = { .name = "SC", .ack = pxa_mask_irq, .mask = pxa_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = pxa_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = pxa_unmask_irq, }; diff --git a/arch/arm/mach-pxa/leds-idp.c b/arch/arm/mach-pxa/leds-idp.c index 8b9c171..144cdc1 100644 --- a/arch/arm/mach-pxa/leds-idp.c +++ b/arch/arm/mach-pxa/leds-idp.c @@ -13,6 +13,7 @@ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/leds-lubbock.c b/arch/arm/mach-pxa/leds-lubbock.c index e26d5ef..706cced 100644 --- a/arch/arm/mach-pxa/leds-lubbock.c +++ b/arch/arm/mach-pxa/leds-lubbock.c @@ -12,6 +12,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/leds-mainstone.c b/arch/arm/mach-pxa/leds-mainstone.c index db4af5e..c204cd4 100644 --- a/arch/arm/mach-pxa/leds-mainstone.c +++ b/arch/arm/mach-pxa/leds-mainstone.c @@ -11,6 +11,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-pxa/lpd270.c b/arch/arm/mach-pxa/lpd270.c index 1373c22..f931637 100644 --- a/arch/arm/mach-pxa/lpd270.c +++ b/arch/arm/mach-pxa/lpd270.c @@ -24,6 +24,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/pwm_backlight.h> +#include <linux/ipipe.h> #include <asm/types.h> #include <asm/setup.h> @@ -124,7 +125,7 @@ static void lpd270_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear useless edge notification */ if (likely(pending)) { irq = LPD270_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); pending = __raw_readw(LPD270_INT_STATUS) & lpd270_irq_enabled; diff --git a/arch/arm/mach-pxa/lubbock.c b/arch/arm/mach-pxa/lubbock.c index 98ee7e5..5dbef64 100644 --- a/arch/arm/mach-pxa/lubbock.c +++ b/arch/arm/mach-pxa/lubbock.c @@ -22,6 +22,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/smc91x.h> +#include <linux/ipipe.h> #include <linux/spi/spi.h> #include <linux/spi/ads7846.h> @@ -161,7 +162,7 @@ static void lubbock_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear our parent irq */ if (likely(pending)) { irq = LUBBOCK_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = LUB_IRQ_SET_CLR & lubbock_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/mainstone.c b/arch/arm/mach-pxa/mainstone.c index 851ee0f..9c1428a 100644 --- a/arch/arm/mach-pxa/mainstone.c +++ b/arch/arm/mach-pxa/mainstone.c @@ -27,6 +27,7 @@ #include <linux/gpio_keys.h> #include <linux/pwm_backlight.h> #include <linux/smc91x.h> +#include <linux/ipipe.h> #include <asm/types.h> #include <asm/setup.h> @@ -165,7 +166,7 @@ static void mainstone_irq_handler(unsigned int irq, struct irq_desc *desc) GEDR(0) = GPIO_bit(0); /* clear useless edge notification */ if (likely(pending)) { irq = MAINSTONE_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = MST_INTSETCLR & mainstone_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/pcm990-baseboard.c b/arch/arm/mach-pxa/pcm990-baseboard.c index d5255ae..cf0ec2a 100644 --- a/arch/arm/mach-pxa/pcm990-baseboard.c +++ b/arch/arm/mach-pxa/pcm990-baseboard.c @@ -24,6 +24,7 @@ #include <linux/platform_device.h> #include <linux/i2c.h> #include <linux/pwm_backlight.h> +#include <linux/ipipe.h> #include <media/soc_camera.h> @@ -263,7 +264,7 @@ static void pcm990_irq_handler(unsigned int irq, struct irq_desc *desc) GPIO_bit(PCM990_CTRL_INT_IRQ_GPIO); if (likely(pending)) { irq = PCM027_IRQ(0) + __ffs(pending); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = (~PCM990_INTSETCLR) & pcm990_irq_enabled; } while (pending); diff --git a/arch/arm/mach-pxa/time.c b/arch/arm/mach-pxa/time.c index 750c448..2b7e7f8 100644 --- a/arch/arm/mach-pxa/time.c +++ b/arch/arm/mach-pxa/time.c @@ -24,6 +24,60 @@ #include <asm/mach/time.h> #include <mach/regs-ost.h> +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint = IRQ_OST0; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int pxa_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *) 0x40A00010; + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + /* * This is PXA's sched_clock implementation. This has a resolution * of at least 308 ns and a maximum value of 208 days. @@ -66,8 +120,12 @@ pxa_ost0_interrupt(int irq, void *dev_id) struct clock_event_device *c = dev_id; /* Disarm the compare/match, signal the event. */ +#ifndef CONFIG_IPIPE OIER &= ~OIER_E0; OSSR = OSSR_M0; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ c->event_handler(c); return IRQ_HANDLED; @@ -79,8 +137,8 @@ pxa_osmr0_set_next_event(unsigned long delta, struct clock_event_device *dev) unsigned long flags, next, oscr; raw_local_irq_save(flags); - OIER |= OIER_E0; next = OSCR + delta; + OIER |= OIER_E0; OSMR0 = next; oscr = OSCR; raw_local_irq_restore(flags); @@ -125,6 +183,13 @@ static struct clock_event_device ckevt_pxa_osmr0 = { .set_mode = pxa_osmr0_set_mode, }; +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, ckevt_pxa_osmr0.name); +} +#endif /* CONFIG_IPIPE */ + static cycle_t pxa_read_oscr(struct clocksource *cs) { return OSCR; @@ -168,6 +233,15 @@ static void __init pxa_timer_init(void) setup_irq(IRQ_OST0, &pxa_ost0_irq); +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + pxa_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + clocksource_register(&cksrc_pxa_oscr0); clockevents_register_device(&ckevt_pxa_osmr0); } @@ -213,3 +287,81 @@ struct sys_timer pxa_timer = { .suspend = pxa_timer_suspend, .resume = pxa_timer_resume, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + OSSR = OSSR_M0; /* Clear match on timer 0 */ + OIER &= ~OIER_E0; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = OSCR; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(pxa_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = OSCR; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > MIN_OSCR_DELTA) { + unsigned long flags; + + local_irq_save_hw(flags); + OSMR0 = delay + OSCR; + OIER |= OIER_E0; + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(IRQ_OST0); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + pxa_osmr0_set_mode(ckevt_pxa_osmr0.mode, &ckevt_pxa_osmr0); + if (ckevt_pxa_osmr0.mode == CLOCK_EVT_MODE_ONESHOT) + pxa_osmr0_set_next_event(LATCH, &ckevt_pxa_osmr0); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return OSMR0 - OSCR; +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mach-pxa/viper.c b/arch/arm/mach-pxa/viper.c index 89f258c..51e1d64 100644 --- a/arch/arm/mach-pxa/viper.c +++ b/arch/arm/mach-pxa/viper.c @@ -41,6 +41,7 @@ #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/mtd/physmap.h> +#include <linux/ipipe.h> #include <mach/pxa25x.h> #include <mach/audio.h> @@ -285,7 +286,7 @@ static void viper_irq_handler(unsigned int irq, struct irq_desc *desc) if (likely(pending)) { irq = viper_bit_to_irq(__ffs(pending)); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } pending = viper_irq_pending(); } while (pending); diff --git a/arch/arm/mach-s3c2410/include/mach/irqs.h b/arch/arm/mach-s3c2410/include/mach/irqs.h index 6c12c63..06ccc4e 100644 --- a/arch/arm/mach-s3c2410/include/mach/irqs.h +++ b/arch/arm/mach-s3c2410/include/mach/irqs.h @@ -3,6 +3,8 @@ * Copyright (c) 2003-2005 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. diff --git a/arch/arm/mach-s3c2440/irq.c b/arch/arm/mach-s3c2440/irq.c index 0c049b9..23a5c90 100644 --- a/arch/arm/mach-s3c2440/irq.c +++ b/arch/arm/mach-s3c2440/irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -25,6 +27,7 @@ #include <linux/ioport.h> #include <linux/sysdev.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -57,10 +60,10 @@ static void s3c_irq_demux_wdtac97(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_S3C2440_WDT); + ipipe_handle_irq_cond(IRQ_S3C2440_WDT); } if (subsrc & 2) { - generic_handle_irq(IRQ_S3C2440_AC97); + ipipe_handle_irq_cond(IRQ_S3C2440_AC97); } } } diff --git a/arch/arm/mach-sa1100/irq.c b/arch/arm/mach-sa1100/irq.c index 3093d46..04a56d2 100644 --- a/arch/arm/mach-sa1100/irq.c +++ b/arch/arm/mach-sa1100/irq.c @@ -15,6 +15,7 @@ #include <linux/irq.h> #include <linux/ioport.h> #include <linux/sysdev.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach/irq.h> @@ -125,7 +126,7 @@ sa1100_high_gpio_handler(unsigned int irq, struct irq_desc *desc) mask >>= 11; do { if (mask & 1) - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); mask >>= 1; irq++; } while (mask); @@ -217,6 +218,9 @@ static struct irq_chip sa1100_normal_chip = { .name = "SC", .ack = sa1100_mask_irq, .mask = sa1100_mask_irq, +#ifdef CONFIG_IPIPE + .mask_ack = sa1100_mask_irq, +#endif /* CONFIG_IPIPE */ .unmask = sa1100_unmask_irq, .set_wake = sa1100_set_wake, }; diff --git a/arch/arm/mach-sa1100/leds-assabet.c b/arch/arm/mach-sa1100/leds-assabet.c index 64e9b4b..0d4e5a4 100644 --- a/arch/arm/mach-sa1100/leds-assabet.c +++ b/arch/arm/mach-sa1100/leds-assabet.c @@ -10,6 +10,7 @@ * - Red - on if system is not idle */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-badge4.c b/arch/arm/mach-sa1100/leds-badge4.c index cf1e384..e41140f 100644 --- a/arch/arm/mach-sa1100/leds-badge4.c +++ b/arch/arm/mach-sa1100/leds-badge4.c @@ -11,6 +11,7 @@ */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-cerf.c b/arch/arm/mach-sa1100/leds-cerf.c index 259b48e..3ab9eff 100644 --- a/arch/arm/mach-sa1100/leds-cerf.c +++ b/arch/arm/mach-sa1100/leds-cerf.c @@ -4,6 +4,7 @@ * Author: ??? */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-hackkit.c b/arch/arm/mach-sa1100/leds-hackkit.c index 2bce137..0a4fe91 100644 --- a/arch/arm/mach-sa1100/leds-hackkit.c +++ b/arch/arm/mach-sa1100/leds-hackkit.c @@ -10,6 +10,7 @@ * as cpu led, the green one is used as timer led. */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-lart.c b/arch/arm/mach-sa1100/leds-lart.c index 0505a1f..f03128d 100644 --- a/arch/arm/mach-sa1100/leds-lart.c +++ b/arch/arm/mach-sa1100/leds-lart.c @@ -10,6 +10,7 @@ * pace of the LED. */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/leds-simpad.c b/arch/arm/mach-sa1100/leds-simpad.c index d50f4ee..2c154e3 100644 --- a/arch/arm/mach-sa1100/leds-simpad.c +++ b/arch/arm/mach-sa1100/leds-simpad.c @@ -4,6 +4,7 @@ * Author: Juergen Messerer <juergen.messerer@siemens.ch> */ #include <linux/init.h> +#include <linux/ipipe_base.h> #include <mach/hardware.h> #include <asm/leds.h> diff --git a/arch/arm/mach-sa1100/time.c b/arch/arm/mach-sa1100/time.c index b9cbb56..90c62f1 100644 --- a/arch/arm/mach-sa1100/time.c +++ b/arch/arm/mach-sa1100/time.c @@ -14,19 +14,78 @@ #include <linux/irq.h> #include <linux/timex.h> #include <linux/clockchips.h> +#include <linux/ipipe.h> #include <asm/mach/time.h> #include <mach/hardware.h> #define MIN_OSCR_DELTA 2 +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint = IRQ_OST0; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int sa1100_timer_initialized; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + info->u.fr.counter = (unsigned *)0x90000010; + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); + +#endif /* CONFIG_IPIPE */ + static irqreturn_t sa1100_ost0_interrupt(int irq, void *dev_id) { struct clock_event_device *c = dev_id; /* Disarm the compare/match, signal the event. */ +#ifndef CONFIG_IPIPE OIER &= ~OIER_E0; OSSR = OSSR_M0; +#else /* CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* CONFIG_IPIPE */ c->event_handler(c); return IRQ_HANDLED; @@ -77,7 +136,14 @@ static struct clock_event_device ckevt_sa1100_osmr0 = { .set_mode = sa1100_osmr0_set_mode, }; -static cycle_t sa1100_read_oscr(struct clocksource *s) +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, ckevt_sa1100_osmr0.name); +} +#endif /* CONFIG_IPIPE */ + +static cycle_t sa1100_read_oscr(struct clocksource *cs) { return OSCR; } @@ -116,6 +182,15 @@ static void __init sa1100_timer_init(void) setup_irq(IRQ_OST0, &sa1100_timer_irq); +#ifdef CONFIG_IPIPE +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + sa1100_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ + clocksource_register(&cksrc_sa1100_oscr); clockevents_register_device(&ckevt_sa1100_osmr0); } @@ -156,3 +231,80 @@ struct sys_timer sa1100_timer = { .suspend = sa1100_timer_suspend, .resume = sa1100_timer_resume, }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + OSSR = OSSR_M0; /* Clear match on timer 0 */ + OIER &= ~OIER_E0; +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + stamp = OSCR; + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(sa1100_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + __asm__ ("ldmia %1, %M0\n": + "=r"(result.full): "r"(local_tsc), "m"(*local_tsc)); + barrier(); + stamp = OSCR; + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > MIN_OSCR_DELTA) { + unsigned long flags; + + local_irq_save_hw(flags); + OSMR0 = delay + OSCR; + OIER |= OIER_E0; + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(IRQ_OST0); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + sa1100_osmr0_set_mode(ckevt_sa1100_osmr0.mode, &ckevt_sa1100_osmr0); + if (ckevt_sa1100_osmr0.mode == CLOCK_EVT_MODE_ONESHOT) + sa1100_osmr0_set_next_event(LATCH, &ckevt_sa1100_osmr0); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return OSMR0 - OSCR; +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index baf6384..9c6421c 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -781,3 +781,83 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARM_FCSE + bool "Fast Context Switch Extension (EXPERIMENTAL)" + depends on EXPERIMENTAL && !SMP && (CPU_32v4 || CPU_32v4T || CPU_32v5) + help + The Fast Context Switch Extension (FCSE for short) is an extension of + some ARM processors which allows to switch contexts between processes + without flushing cache and saves a few tens of microseconds in the + worst case. + + Enabling this option makes linux use the FCSE. + + We propose two modes: + - the guaranteed mode: we guarantee that there will never be any cache + flush when switching context, but this means that there can not be + more than 95 running processes in the system, each with a virtual + memory space smaller than 32MB, and that the shared memory + mappings do not use cache; + - the best effort mode: we allow some cache flushes to happen from + time to time, but do not limit the number of processes or the + virtual memory space available for each process, and the shared + memory mappings use cache. + +if ARM_FCSE + +choice + prompt "FCSE mode" + default ARM_FCSE_BEST_EFFORT + help + This option allow setting which FCSE mode will be used. + +config ARM_FCSE_GUARANTEED + bool "guaranteed" + help + Select guaranteed mode. + +config ARM_FCSE_BEST_EFFORT + bool "best effort" + help + Select best-effort mode. + +endchoice + +config ARM_FCSE_DYNPID + bool "Dynamic FCSE pid allocation" + depends on ARM_FCSE_BEST_EFFORT && !IPIPE + help + When this option is enabled, the kernel may decide to change the FCSE + pid of a process when switching context. This helps reducing the + number of cache flushes, but adds some overhead to the context + switches. + +config ARM_FCSE_PREEMPT_FLUSH + bool "Preemptible cache flushes" + help + When FCSE is enabled, some cache flushes happen with preemption + disabled, with this option, we make them preemptible, this results in + better response time, but may result in the need to restart a cache + flush if it was preempted, so in a global higher cpu consumption. + +config ARM_FCSE_MESSAGES + bool "help messages" + help + When FCSE is enabled in best-effort mode, due to the VM space + reduction, a too large stack size limit may result in processes + exceeding the 32MB limit too easily. A too small stack size may result + in stack overflows. Enabling this option will print messages in these + situations to assist you in tuning the stack size limit. + + In guaranteed mode, this option will cause message to be printed if + one of the hard limits (95 proceses, 32 MB VM space) is exceeded. + +config ARM_FCSE_DEBUG + bool "FCSE debug" + select ARM_FCSE_MESSAGES + help + This option enables some internal debug checks. It has a high + overhead, and is only useful for debugging the FCSE code. + +endif diff --git a/arch/arm/mm/alignment.c b/arch/arm/mm/alignment.c index 62820ed..510a94f 100644 --- a/arch/arm/mm/alignment.c +++ b/arch/arm/mm/alignment.c @@ -726,6 +726,9 @@ do_alignment(unsigned long addr, unsigned int fsr, struct pt_regs *regs) int isize = 4; int thumb2_32b = 0; + if (ipipe_trap_notify(IPIPE_TRAP_ALIGNMENT,regs)) + return 0; + instrptr = instruction_pointer(regs); fs = get_fs(); diff --git a/arch/arm/mm/context.c b/arch/arm/mm/context.c index a9e22e3..24e2999 100644 --- a/arch/arm/mm/context.c +++ b/arch/arm/mm/context.c @@ -14,7 +14,7 @@ #include <asm/mmu_context.h> #include <asm/tlbflush.h> -static DEFINE_SPINLOCK(cpu_asid_lock); +static IPIPE_DEFINE_SPINLOCK(cpu_asid_lock); unsigned int cpu_last_asid = ASID_FIRST_VERSION; /* @@ -31,8 +31,9 @@ void __init_new_context(struct task_struct *tsk, struct mm_struct *mm) void __new_context(struct mm_struct *mm) { unsigned int asid; + unsigned long flags; - spin_lock(&cpu_asid_lock); + spin_lock_irqsave_cond(&cpu_asid_lock, flags); #ifdef CONFIG_SMP /* * Check the ASID again, in case the change was broadcast from @@ -54,5 +55,5 @@ void __new_context(struct mm_struct *mm) } set_mm_context(mm, asid); - spin_unlock(&cpu_asid_lock); + spin_unlock_irqrestore_cond(&cpu_asid_lock, flags); } diff --git a/arch/arm/mm/copypage-v4mc.c b/arch/arm/mm/copypage-v4mc.c index 7370a71..3a7928a 100644 --- a/arch/arm/mm/copypage-v4mc.c +++ b/arch/arm/mm/copypage-v4mc.c @@ -44,7 +44,7 @@ static DEFINE_SPINLOCK(minicache_lock); * instruction. If your processor does not supply this, you have to write your * own copy_user_highpage that does the right thing. */ -static void __naked +static void notrace __naked mc_copy_user_page(void *from, void *to) { asm volatile( diff --git a/arch/arm/mm/copypage-xscale.c b/arch/arm/mm/copypage-xscale.c index 76824d3..db642ce 100644 --- a/arch/arm/mm/copypage-xscale.c +++ b/arch/arm/mm/copypage-xscale.c @@ -42,7 +42,7 @@ static DEFINE_SPINLOCK(minicache_lock); * Dcache aliasing issue. The writes will be forwarded to the write buffer, * and merged as appropriate. */ -static void __naked +static void notrace __naked mc_copy_user_page(void *from, void *to) { /* diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 56ee153..88f5956 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -27,6 +27,57 @@ static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE; +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +static void fcse_set_pte_shared(struct vm_area_struct *vma, + unsigned long address) +{ + pgd_t *pgd; + pmd_t *pmd; + pte_t *pte, entry; + + if (!(vma->vm_flags & VM_MAYSHARE) || address >= TASK_SIZE) + return; + + pgd = pgd_offset(vma->vm_mm, address); + if (pgd_none(*pgd)) + goto no_pgd; + if (pgd_bad(*pgd)) + goto bad_pgd; + + pmd = pmd_offset(pgd, address); + if (pmd_none(*pmd)) + goto no_pmd; + if (pmd_bad(*pmd)) + goto bad_pmd; + + pte = pte_offset_map(pmd, address); + entry = *pte; + + if ((pte_val(entry) + & (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) + == (L_PTE_PRESENT | PTE_CACHEABLE | L_PTE_WRITE | L_PTE_DIRTY)) { + pte_val(entry) |= L_PTE_SHARED; + set_pte_at(vma->vm_mm, address, pte, entry); + } + pte_unmap(pte); + return; + +bad_pgd: + pgd_ERROR(*pgd); + pgd_clear(pgd); +no_pgd: + return; + +bad_pmd: + pmd_ERROR(*pmd); + pmd_clear(pmd); +no_pmd: + return; +} +#else /* !CONFIG_ARM_FCSE_BEST_EFFORT */ +#define fcse_set_pte_shared(vma, addr) do { } while (0) +#endif /* !CONFIG_ARM_FCSE_BEST_EFFORT */ + /* * We take the easy way out of this problem - we make the * PTE uncacheable. However, we leave the write buffer on. @@ -96,7 +147,8 @@ no_pmd: static void make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned long pfn) { +#ifndef CONFIG_ARM_FCSE_GUARANTEED struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *mpnt; struct prio_tree_iter iter; @@ -127,8 +179,14 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne flush_dcache_mmap_unlock(mapping); if (aliases) do_adjust_pte(vma, addr, pfn, ptep); - else + else { + fcse_set_pte_shared(vma, addr); flush_cache_page(vma, addr, pfn); + } +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + if (vma->vm_flags & VM_MAYSHARE) + adjust_pte(vma, addr); +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ } /* diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index 10e0680..4413c0e 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -73,6 +73,10 @@ void show_pte(struct mm_struct *mm, unsigned long addr) if (!mm) mm = &init_mm; +#ifdef CONFIG_ARM_FCSE + printk(KERN_ALERT "fcse pid: %ld, 0x%08lx\n", + mm->context.fcse.pid >> FCSE_PID_SHIFT, mm->context.fcse.pid); +#endif /* CONFIG_ARM_FCSE */ printk(KERN_ALERT "pgd = %p\n", mm->pgd); pgd = pgd_offset(mm, addr); printk(KERN_ALERT "[%08lx] *pgd=%08lx", addr, pgd_val(*pgd)); @@ -161,11 +165,21 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr, if (user_debug & UDBG_SEGV) { printk(KERN_DEBUG "%s: unhandled page fault (%d) at 0x%08lx, code 0x%03x\n", tsk->comm, sig, addr, fsr); +#ifdef CONFIG_ARM_FCSE_DYNPID + /* Disable preemption to avoid page tables changing under our + feet */ + preempt_disable(); +#endif /* CONFIG_ARM_FCSE_DYNPID */ show_pte(tsk->mm, addr); +#ifdef CONFIG_ARM_FCSE_DYNPID + preempt_enable(); +#endif /* CONFIG_ARM_FCSE_DYNPID */ show_regs(regs); } #endif + fcse_notify_segv(tsk->mm, addr, regs); + tsk->thread.address = addr; tsk->thread.error_code = fsr; tsk->thread.trap_no = 14; @@ -266,6 +280,9 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (notify_page_fault(regs, fsr)) return 0; + if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs)) + return 0; + tsk = current; mm = tsk->mm; @@ -386,6 +403,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr, if (addr < TASK_SIZE) return do_page_fault(addr, fsr, regs); + if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs)) + return 0; + index = pgd_index(addr); /* @@ -429,6 +449,10 @@ do_translation_fault(unsigned long addr, unsigned int fsr, static int do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + + if (ipipe_trap_notify(IPIPE_TRAP_SECTION,regs)) + return 0; + do_bad_area(addr, fsr, regs); return 0; } @@ -439,6 +463,9 @@ do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) static int do_bad(unsigned long addr, unsigned int fsr, struct pt_regs *regs) { + if (ipipe_trap_notify(IPIPE_TRAP_DABT,regs)) + return 0; + return 1; } @@ -514,6 +541,9 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (!inf->fn(addr, fsr & ~FSR_LNX_PF, regs)) return; + if (ipipe_trap_notify(IPIPE_TRAP_UNKNOWN,regs)) + return; + printk(KERN_ALERT "Unhandled fault: %s (0x%03x) at 0x%08lx\n", inf->name, fsr, addr); @@ -579,3 +609,42 @@ do_PrefetchAbort(unsigned long addr, unsigned int ifsr, struct pt_regs *regs) arm_notify_die("", regs, &info, ifsr, 0); } +#ifdef CONFIG_IPIPE +extern spinlock_t pgd_lock; +extern struct page *pgd_list; + +static void vmalloc_sync_one(pgd_t *pgd, unsigned long addr) +{ + unsigned int index = pgd_index(addr); + pgd_t *pgd_k; + pmd_t *pmd, *pmd_k; + + pgd += index; + pgd_k = init_mm.pgd + index; + + if (!pgd_present(*pgd)) + set_pgd(pgd, *pgd_k); + + pmd_k = pmd_offset(pgd_k, addr); + pmd = pmd_offset(pgd, addr); + + copy_pmd(pmd, pmd_k); +} + +void __ipipe_pin_range_globally(unsigned long start, unsigned long end) +{ + unsigned long next, addr = start; + + do { + unsigned long flags; + struct page *page; + + next = pgd_addr_end(addr, end); + spin_lock_irqsave(&pgd_lock, flags); + for (page = pgd_list; page; page = (struct page *)page->index) + vmalloc_sync_one(page_address(page), addr); + spin_unlock_irqrestore(&pgd_lock, flags); + + } while (addr = next, addr != end); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/mm/ioremap.c b/arch/arm/mm/ioremap.c index 0ab75c6..32fdbe6 100644 --- a/arch/arm/mm/ioremap.c +++ b/arch/arm/mm/ioremap.c @@ -316,6 +316,7 @@ __arm_ioremap_pfn(unsigned long pfn, unsigned long offset, size_t size, } flush_cache_vmap(addr, addr + size); + __ipipe_pin_range_globally(addr, addr + size); return (void __iomem *) (offset + addr); } EXPORT_SYMBOL(__arm_ioremap_pfn); diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index f5abc51..fb22056 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -29,7 +29,8 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - unsigned long start_addr; + unsigned long start_addr = addr; + #ifdef CONFIG_CPU_V6 unsigned int cache_type; int do_align = 0, aliasing = 0; @@ -50,6 +51,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, #define aliasing 0 #endif +#ifdef CONFIG_ARM_FCSE + start_addr = addr; +#endif /* CONFIG_ARM_FCSE */ /* * We enforce the MAP_FIXED case. */ @@ -57,7 +61,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, if (aliasing && flags & MAP_SHARED && (addr - (pgoff << PAGE_SHIFT)) & (SHMLBA - 1)) return -EINVAL; - return addr; + goto found_addr; } if (len > TASK_SIZE) @@ -72,13 +76,13 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, vma = find_vma(mm, addr); if (TASK_SIZE - len >= addr && (!vma || addr + len <= vma->vm_start)) - return addr; + goto found_addr; } if (len > mm->cached_hole_size) { - start_addr = addr = mm->free_area_cache; + start_addr = addr = mm->free_area_cache; } else { - start_addr = addr = TASK_UNMAPPED_BASE; - mm->cached_hole_size = 0; + start_addr = addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; } full_search: @@ -106,14 +110,31 @@ full_search: * Remember the place where we stopped the search: */ mm->free_area_cache = addr + len; - return addr; + goto found_addr; } if (addr + mm->cached_hole_size < vma->vm_start) - mm->cached_hole_size = vma->vm_start - addr; + mm->cached_hole_size = vma->vm_start - addr; addr = vma->vm_end; if (do_align) addr = COLOUR_ALIGN(addr, pgoff); } + + found_addr: +#ifdef CONFIG_ARM_FCSE + { + unsigned long new_addr = fcse_check_mmap_addr(mm, start_addr, + addr, len, flags); + if (new_addr != addr) { + addr = new_addr; + if (!(addr & ~PAGE_MASK)) { + start_addr = TASK_UNMAPPED_BASE; + mm->cached_hole_size = 0; + goto full_search; + } + } + } +#endif /* CONFIG_ARM_FCSE */ + return addr; } diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 2690146..16c48a3 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -18,6 +18,41 @@ #define FIRST_KERNEL_PGD_NR (FIRST_USER_PGD_NR + USER_PTRS_PER_PGD) +#ifdef CONFIG_IPIPE +/* Copied from arch/i386/mm/pgdtable.c, maintains the list of pgds for the + implementation of ipipe_pin_range_globally in arch/arm/mm/fault.c. */ +DEFINE_SPINLOCK(pgd_lock); +struct page *pgd_list; + +#define pgd_list_lock(flags) spin_lock_irqsave(&pgd_lock, flags) +#define pgd_list_unlock(flags) spin_unlock_irqrestore(&pgd_lock, flags) + +static inline void pgd_list_add(pgd_t *pgd) +{ + struct page *page = virt_to_page(pgd); + page->index = (unsigned long)pgd_list; + if (pgd_list) + set_page_private(pgd_list, (unsigned long)&page->index); + pgd_list = page; + set_page_private(page, (unsigned long)&pgd_list); +} + +static inline void pgd_list_del(pgd_t *pgd) +{ + struct page *next, **pprev, *page = virt_to_page(pgd); + next = (struct page *)page->index; + pprev = (struct page **)page_private(page); + *pprev = next; + if (next) + set_page_private(next, (unsigned long)pprev); +} +#else /* !CONFIG_IPIPE */ +#define pgd_list_lock(flags) ((void) (flags)) +#define pgd_list_unlock(flags) ((void) (flags)) +#define pgd_list_add(pgd) do { } while (0) +#define pgd_list_del(pgd) do { } while (0) +#endif /* !CONFIG_IPIPE */ + /* * need to get a 16k page for level 1 */ @@ -26,6 +61,7 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) pgd_t *new_pgd, *init_pgd; pmd_t *new_pmd, *init_pmd; pte_t *new_pte, *init_pte; + unsigned long flags; new_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, 2); if (!new_pgd) @@ -37,12 +73,20 @@ pgd_t *get_pgd_slow(struct mm_struct *mm) * Copy over the kernel and IO PGD entries */ init_pgd = pgd_offset_k(0); + pgd_list_lock(flags); memcpy(new_pgd + FIRST_KERNEL_PGD_NR, init_pgd + FIRST_KERNEL_PGD_NR, (PTRS_PER_PGD - FIRST_KERNEL_PGD_NR) * sizeof(pgd_t)); + pgd_list_add(new_pgd); + pgd_list_unlock(flags); clean_dcache_area(new_pgd, PTRS_PER_PGD * sizeof(pgd_t)); if (!vectors_high()) { +#ifdef CONFIG_ARM_FCSE + /* FCSE does not work without high vectors. */ + BUG(); +#endif /* CONFIG_ARM_FCSE */ + /* * On ARM, first page must always be allocated since it * contains the machine vectors. @@ -74,6 +118,7 @@ no_pgd: void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) { + unsigned long flags; pmd_t *pmd; pgtable_t pte; @@ -81,7 +126,7 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) return; /* pgd is always present and good */ - pmd = pmd_off(pgd, 0); + pmd = pmd_off(pgd + pgd_index(fcse_va_to_mva(mm, 0)), 0); if (pmd_none(*pmd)) goto free; if (pmd_bad(*pmd)) { @@ -95,5 +140,8 @@ void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd) pte_free(mm, pte); pmd_free(mm, pmd); free: + pgd_list_lock(flags); + pgd_list_del(pgd); + pgd_list_unlock(flags); free_pages((unsigned long) pgd, 2); } diff --git a/arch/arm/mm/proc-arm920.S b/arch/arm/mm/proc-arm920.S index 471669e..e474ecc 100644 --- a/arch/arm/mm/proc-arm920.S +++ b/arch/arm/mm/proc-arm920.S @@ -322,6 +322,11 @@ ENTRY(cpu_arm920_dcache_clean_area) ENTRY(cpu_arm920_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 3f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -339,6 +344,10 @@ ENTRY(cpu_arm920_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* !CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +3: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-arm926.S b/arch/arm/mm/proc-arm926.S index 64db6e2..ce7aa93 100644 --- a/arch/arm/mm/proc-arm926.S +++ b/arch/arm/mm/proc-arm926.S @@ -338,6 +338,11 @@ ENTRY(cpu_arm926_dcache_clean_area) ENTRY(cpu_arm926_switch_mm) #ifdef CONFIG_MMU mov ip, #0 +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED #ifdef CONFIG_CPU_DCACHE_WRITETHROUGH mcr p15, 0, ip, c7, c6, 0 @ invalidate D cache #else @@ -347,6 +352,10 @@ ENTRY(cpu_arm926_switch_mm) #endif mcr p15, 0, ip, c7, c5, 0 @ invalidate I cache mcr p15, 0, ip, c7, c10, 4 @ drain WB +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs #endif diff --git a/arch/arm/mm/proc-feroceon.S b/arch/arm/mm/proc-feroceon.S index dbc3938..d8da8d6 100644 --- a/arch/arm/mm/proc-feroceon.S +++ b/arch/arm/mm/proc-feroceon.S @@ -422,6 +422,12 @@ ENTRY(cpu_feroceon_dcache_clean_area) .align 5 ENTRY(cpu_feroceon_switch_mm) #ifdef CONFIG_MMU +#ifndef CONFIG_ARM_FCSE_GUARANTEED +#ifdef CONFIG_ARM_FCSE + cmp r2, #0 + mov r2, lr + beq 2f +#else /* !CONFIG_ARM_FCSE */ /* * Note: we wish to call __flush_whole_cache but we need to preserve * lr to do so. The only way without touching main memory is to @@ -429,12 +435,19 @@ ENTRY(cpu_feroceon_switch_mm) * compensate locally for the skipped ops if it is not set. */ mov r2, lr @ abuse r2 to preserve lr +#endif /* !CONFIG_ARM_FCSE */ bl __flush_whole_cache @ if r2 contains the VM_EXEC bit then the next 2 ops are done already tst r2, #VM_EXEC mcreq p15, 0, ip, c7, c5, 0 @ invalidate I cache mcreq p15, 0, ip, c7, c10, 4 @ drain WB +#ifdef CONFIG_ARM_FCSE +2: +#endif +#else /* CONFIG_ARM_FCSE_GUARANTEED */ + mov r2, lr +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs mov pc, r2 diff --git a/arch/arm/mm/proc-xscale.S b/arch/arm/mm/proc-xscale.S index 93df472..f794a86 100644 --- a/arch/arm/mm/proc-xscale.S +++ b/arch/arm/mm/proc-xscale.S @@ -416,9 +416,18 @@ ENTRY(cpu_xscale_dcache_clean_area) */ .align 5 ENTRY(cpu_xscale_switch_mm) +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT + cmp r2, #0 + beq 2f +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ +#ifndef CONFIG_ARM_FCSE_GUARANTEED clean_d_cache r1, r2 mcr p15, 0, ip, c7, c5, 0 @ Invalidate I cache & BTB mcr p15, 0, ip, c7, c10, 4 @ Drain Write (& Fill) Buffer +#endif /* CONFIG_ARM_FCSE_GUARANTEED */ +#ifdef CONFIG_ARM_FCSE_BEST_EFFORT +2: +#endif /* CONFIG_ARM_FCSE_BEST_EFFORT */ mcr p15, 0, r0, c2, c0, 0 @ load page table pointer mcr p15, 0, ip, c8, c7, 0 @ invalidate I & D TLBs cpwait_ret lr, ip diff --git a/arch/arm/plat-mxc/cpu.c b/arch/arm/plat-mxc/cpu.c index 386e0d5..0bb140e 100644 --- a/arch/arm/plat-mxc/cpu.c +++ b/arch/arm/plat-mxc/cpu.c @@ -1,5 +1,5 @@ - #include <linux/module.h> +#include <linux/io.h> unsigned int __mxc_cpu_type; EXPORT_SYMBOL(__mxc_cpu_type); @@ -9,3 +9,29 @@ void mxc_set_cpu_type(unsigned int type) __mxc_cpu_type = type; } +#ifdef CONFIG_IPIPE +void ipipe_mach_allow_hwtimer_uaccess(unsigned long aips1, unsigned long aips2) +{ + volatile unsigned long aips_reg; + + /* + * S/W workaround: Clear the off platform peripheral modules + * Supervisor Protect bit for SDMA to access them. + */ + __raw_writel(0x0, aips1 + 0x40); + __raw_writel(0x0, aips1 + 0x44); + __raw_writel(0x0, aips1 + 0x48); + __raw_writel(0x0, aips1 + 0x4C); + aips_reg = __raw_readl(aips1 + 0x50); + aips_reg &= 0x00FFFFFF; + __raw_writel(aips_reg, aips1 + 0x50); + + __raw_writel(0x0, aips2 + 0x40); + __raw_writel(0x0, aips2 + 0x44); + __raw_writel(0x0, aips2 + 0x48); + __raw_writel(0x0, aips2 + 0x4C); + aips_reg = __raw_readl(aips2 + 0x50); + aips_reg &= 0x00FFFFFF; + __raw_writel(aips_reg, aips2 + 0x50); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-mxc/gpio.c b/arch/arm/plat-mxc/gpio.c index d65ebe3..2eedd79 100644 --- a/arch/arm/plat-mxc/gpio.c +++ b/arch/arm/plat-mxc/gpio.c @@ -23,6 +23,7 @@ #include <linux/io.h> #include <linux/irq.h> #include <linux/gpio.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm-generic/bug.h> @@ -174,8 +175,7 @@ static void mxc_gpio_irq_handler(struct mxc_gpio_port *port, u32 irq_stat) if (port->both_edges & (1 << (gpio & 31))) mxc_flip_edge(port, gpio); - irq_desc[gpio_irq_no].handle_irq(gpio_irq_no, - &irq_desc[gpio_irq_no]); + ipipe_handle_irq_cond(gpio_irq_no); } } diff --git a/arch/arm/plat-mxc/include/mach/common.h b/arch/arm/plat-mxc/include/mach/common.h index 4bf1068..1cf801f 100644 --- a/arch/arm/plat-mxc/include/mach/common.h +++ b/arch/arm/plat-mxc/include/mach/common.h @@ -45,4 +45,8 @@ extern void mxc91231_power_off(void); extern void mxc91231_arch_reset(int, const char *); extern void mxc91231_prepare_idle(void); +#ifdef CONFIG_IPIPE +void ipipe_mach_allow_hwtimer_uaccess(unsigned long aips1, unsigned long aips2); +#endif + #endif diff --git a/arch/arm/plat-mxc/time.c b/arch/arm/plat-mxc/time.c index 844567e..bc60cac 100644 --- a/arch/arm/plat-mxc/time.c +++ b/arch/arm/plat-mxc/time.c @@ -25,6 +25,7 @@ #include <linux/irq.h> #include <linux/clockchips.h> #include <linux/clk.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/mach/time.h> @@ -57,6 +58,72 @@ #define MX3_TCN 0x24 #define MX3_TCMP 0x10 +#ifdef CONFIG_IPIPE +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ +int __ipipe_mach_timerint; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +unsigned int __ipipe_mach_ticks_per_jiffy = LATCH; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +static int mxc_timer_initialized; +static unsigned mxc_min_delay; + +union tsc_reg { +#ifdef __BIG_ENDIAN + struct { + unsigned long high; + unsigned long low; + }; +#else /* __LITTLE_ENDIAN */ + struct { + unsigned long low; + unsigned long high; + }; +#endif /* __LITTLE_ENDIAN */ + unsigned long long full; +}; + +#ifdef CONFIG_SMP +static union tsc_reg tsc[NR_CPUS]; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_NONE; +} + +#else /* !CONFIG_SMP */ +static union tsc_reg *tsc; + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_FREERUNNING; + if (cpu_is_mx1()) { +#ifdef CONFIG_ARCH_MX1 + info->u.fr.counter = (unsigned *) (TIM1_BASE_ADDR + MX1_2_TCN); +#endif + } else if (cpu_is_mx2()) { +#ifdef CONFIG_ARCH_MX2 + info->u.fr.counter = (unsigned *) (GPT1_BASE_ADDR + MX1_2_TCN); +#endif + } else if (cpu_is_mx3()) { +#ifdef CONFIG_ARCH_MX3 + info->u.fr.counter = (unsigned *) (GPT1_BASE_ADDR + MX3_TCN); +#endif + } + info->u.fr.mask = 0xffffffff; + info->u.fr.tsc = &tsc->full; +} +#endif /* !CONFIG_SMP */ + +static void ipipe_mach_update_tsc(void); +#endif /* CONFIG_IPIPE */ + static struct clock_event_device clockevent_mxc; static enum clock_event_mode clockevent_mode = CLOCK_EVT_MODE_UNUSED; @@ -120,6 +187,9 @@ static int __init mxc_clocksource_init(struct clk *timer_clk) if (cpu_is_mx3() || cpu_is_mx25()) clocksource_mxc.read = mx3_get_cycles; +#ifdef CONFIG_IPIPE + __ipipe_mach_ticks_per_jiffy = (c + HZ/2) / HZ; +#endif /* CONFIG_IPIPE */ clocksource_mxc.mult = clocksource_hz2mult(c, clocksource_mxc.shift); clocksource_register(&clocksource_mxc); @@ -238,7 +308,11 @@ static irqreturn_t mxc_timer_interrupt(int irq, void *dev_id) else tstat = __raw_readl(timer_base + MX1_2_TSTAT); +#ifndef CONFIG_IPIPE gpt_irq_acknowledge(); +#else /* !CONFIG_IPIPE */ + ipipe_mach_update_tsc(); +#endif /* !CONFIG_IPIPE */ evt->event_handler(evt); @@ -309,4 +383,116 @@ void __init mxc_timer_init(struct clk *timer_clk, void __iomem *base, int irq) /* Make irqs happen */ setup_irq(irq, &mxc_timer_irq); + +#ifdef CONFIG_IPIPE + __ipipe_mach_timerint = irq; +#ifndef CONFIG_SMP + tsc = (union tsc_reg *) __ipipe_tsc_area; + barrier(); +#endif /* CONFIG_SMP */ + + mxc_min_delay = ((ipipe_cpu_freq() + 500000) / 1000000) ?: 1; + mxc_timer_initialized = 1; +#endif /* CONFIG_IPIPE */ +} + +#ifdef CONFIG_IPIPE +int __ipipe_check_tickdev(const char *devname) +{ + return !strcmp(devname, clockevent_mxc.name); +} + +void __ipipe_mach_acktimer(void) +{ + gpt_irq_acknowledge(); +} + +static void ipipe_mach_update_tsc(void) +{ + union tsc_reg *local_tsc; + unsigned long stamp, flags; + + local_irq_save_hw(flags); + local_tsc = &tsc[ipipe_processor_id()]; + if (!cpu_is_mx3()) + stamp = __raw_readl(timer_base + MX1_2_TCN); + else + stamp = __raw_readl(timer_base + MX3_TCN); + if (unlikely(stamp < local_tsc->low)) + /* 32 bit counter wrapped, increment high word. */ + local_tsc->high++; + local_tsc->low = stamp; + local_irq_restore_hw(flags); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + if (likely(mxc_timer_initialized)) { + union tsc_reg *local_tsc, result; + unsigned long stamp; + + local_tsc = &tsc[ipipe_processor_id()]; + + if (!cpu_is_mx3()) { + __asm__ ("ldmia %1, %M0\n" + : "=r"(result.full), "+&r"(local_tsc) + : "m"(*local_tsc)); + barrier(); + stamp = __raw_readl(timer_base + MX1_2_TCN); + } else { + __asm__ ("ldmia %1, %M0\n" + : "=r"(result.full), "+&r"(local_tsc) + : "m"(*local_tsc)); + barrier(); + stamp = __raw_readl(timer_base + MX3_TCN); + } + if (unlikely(stamp < result.low)) + /* 32 bit counter wrapped, increment high word. */ + result.high++; + result.low = stamp; + + return result.full; + } + + return 0; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +/* + * Reprogram the timer + */ + +void __ipipe_mach_set_dec(unsigned long delay) +{ + if (delay > mxc_min_delay) { + unsigned long flags; + + local_irq_save_hw(flags); + if (!cpu_is_mx3()) + mx1_2_set_next_event(delay, NULL); + else + mx3_set_next_event(delay, NULL); + local_irq_restore_hw(flags); + } else + ipipe_trigger_irq(__ipipe_mach_timerint); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + mxc_set_mode(clockevent_mxc.mode, &clockevent_mxc); + if (clockevent_mxc.mode == CLOCK_EVT_MODE_ONESHOT) + clockevent_mxc.set_next_event(LATCH, &clockevent_mxc); +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + if (!cpu_is_mx3()) + return __raw_readl(timer_base + MX1_2_TCMP) + - __raw_readl(timer_base + MX1_2_TCN); + else + return __raw_readl(timer_base + MX3_TCMP) + - __raw_readl(timer_base + MX3_TCN); } +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-omap/Kconfig b/arch/arm/plat-omap/Kconfig index e2ea04a..7756c92 100644 --- a/arch/arm/plat-omap/Kconfig +++ b/arch/arm/plat-omap/Kconfig @@ -114,6 +114,7 @@ config OMAP_MPU_TIMER timer provides more intra-tick resolution than the 32KHz timer, but consumes more power. +if !IPIPE config OMAP_32K_TIMER bool "Use 32KHz timer" depends on ARCH_OMAP16XX || ARCH_OMAP24XX || ARCH_OMAP34XX || ARCH_OMAP4 @@ -123,6 +124,7 @@ config OMAP_32K_TIMER support for no tick during idle. The 32KHz timer provides less intra-tick resolution than OMAP_MPU_TIMER. The 32KHz timer is currently only available for OMAP16XX, 24XX, 34XX and OMAP4. +endif endchoice diff --git a/arch/arm/plat-omap/dmtimer.c b/arch/arm/plat-omap/dmtimer.c index 08ccf89..4d1abd7 100644 --- a/arch/arm/plat-omap/dmtimer.c +++ b/arch/arm/plat-omap/dmtimer.c @@ -475,6 +475,13 @@ int omap_dm_timer_get_irq(struct omap_dm_timer *timer) } EXPORT_SYMBOL_GPL(omap_dm_timer_get_irq); +#ifdef CONFIG_IPIPE +unsigned long omap_dm_timer_get_phys_counter_addr(struct omap_dm_timer *timer) +{ + return timer->phys_base + (OMAP_TIMER_COUNTER_REG & 0xff); +} +#endif /* CONFIG_IPIPE */ + #if defined(CONFIG_ARCH_OMAP1) /** diff --git a/arch/arm/plat-omap/gpio.c b/arch/arm/plat-omap/gpio.c index d2422c7..589a4bb 100644 --- a/arch/arm/plat-omap/gpio.c +++ b/arch/arm/plat-omap/gpio.c @@ -20,7 +20,9 @@ #include <linux/sysdev.h> #include <linux/err.h> #include <linux/clk.h> +#include <linux/irq.h> /* For irq_desc */ #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -1345,8 +1347,7 @@ static void gpio_irq_handler(unsigned int irq, struct irq_desc *desc) if (bank->toggle_mask & (1 << gpio_index)) _toggle_gpio_edge_triggering(bank, gpio_index); #endif - - generic_handle_irq(gpio_irq); + ipipe_handle_irq_cond(gpio_irq); } } /* if bank has any level sensitive GPIO pin interrupt diff --git a/arch/arm/plat-omap/include/plat/dmtimer.h b/arch/arm/plat-omap/include/plat/dmtimer.h index 20f1054..42a0b8d 100644 --- a/arch/arm/plat-omap/include/plat/dmtimer.h +++ b/arch/arm/plat-omap/include/plat/dmtimer.h @@ -56,6 +56,9 @@ void omap_dm_timer_enable(struct omap_dm_timer *timer); void omap_dm_timer_disable(struct omap_dm_timer *timer); int omap_dm_timer_get_irq(struct omap_dm_timer *timer); +#ifdef CONFIG_IPIPE +unsigned long omap_dm_timer_get_phys_counter_addr(struct omap_dm_timer *timer); +#endif /* CONFIG_IPIPE */ u32 omap_dm_timer_modify_idlect_mask(u32 inputmask); struct clk *omap_dm_timer_get_fclk(struct omap_dm_timer *timer); diff --git a/arch/arm/plat-pxa/gpio.c b/arch/arm/plat-pxa/gpio.c index 98548c6..e732e45 100644 --- a/arch/arm/plat-pxa/gpio.c +++ b/arch/arm/plat-pxa/gpio.c @@ -17,6 +17,7 @@ #include <linux/io.h> #include <linux/sysdev.h> #include <linux/slab.h> +#include <linux/ipipe.h> #include <mach/gpio.h> @@ -220,7 +221,7 @@ static void pxa_gpio_demux_handler(unsigned int irq, struct irq_desc *desc) while (n < BITS_PER_LONG) { loop = 1; - generic_handle_irq(gpio_to_irq(gpio_base + n)); + ipipe_handle_irq_cond(gpio_to_irq(gpio_base + n)); n = find_next_bit(&gedr, BITS_PER_LONG, n + 1); } } @@ -285,7 +286,7 @@ void __init pxa_init_gpio(int mux_irq, int start, int end, set_wake_t fn) for (irq = gpio_to_irq(start); irq <= gpio_to_irq(end); irq++) { set_irq_chip(irq, &pxa_muxed_gpio_chip); - set_irq_handler(irq, handle_edge_irq); + set_irq_handler(irq, handle_level_irq); set_irq_flags(irq, IRQF_VALID | IRQF_PROBE); } diff --git a/arch/arm/plat-s3c/time.c b/arch/arm/plat-s3c/time.c index 3b27b29..976fe03 100644 --- a/arch/arm/plat-s3c/time.c +++ b/arch/arm/plat-s3c/time.c @@ -3,6 +3,8 @@ * Copyright (C) 2003-2005 Simtec Electronics * Ben Dooks, <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -27,6 +29,7 @@ #include <linux/clk.h> #include <linux/io.h> #include <linux/platform_device.h> +#include <linux/module.h> #include <asm/system.h> #include <asm/leds.h> @@ -42,7 +45,6 @@ #include <plat/clock.h> #include <plat/cpu.h> -static unsigned long timer_startval; static unsigned long timer_usec_ticks; #ifndef TICK_MAX @@ -61,6 +63,39 @@ static unsigned long timer_usec_ticks; * Original patch by Dimitry Andric, updated by Ben Dooks */ +static unsigned long last_free_running_tcnt = 0; +static unsigned long free_running_tcon = 0; +static unsigned long timer_lxlost = 0; + +#ifdef CONFIG_IPIPE + +#ifdef CONFIG_NO_IDLE_HZ +#error "dynamic tick timer not yet supported with IPIPE" +#endif /* CONFIG_NO_IDLE_HZ */ + +unsigned int __ipipe_mach_ticks_per_jiffy; +EXPORT_SYMBOL(__ipipe_mach_ticks_per_jiffy); + +int __ipipe_mach_timerint = IRQ_TIMER4; +EXPORT_SYMBOL(__ipipe_mach_timerint); + +static unsigned long long *tsc; +static unsigned *last_cnt; +static unsigned long timer_ackval = 1UL << (IRQ_TIMER4 - IRQ_EINT0); +static IPIPE_DEFINE_SPINLOCK(timer_lock); + +int __ipipe_mach_timerstolen = 0; +EXPORT_SYMBOL(__ipipe_mach_timerstolen); + +void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info) +{ + info->type = IPIPE_TSC_TYPE_DECREMENTER; + info->u.dec.counter = (unsigned *)0x51000038; + info->u.dec.mask = 0xffff; + info->u.dec.last_cnt = last_cnt; + info->u.dec.tsc = tsc; +} +#endif /* CONFIG_IPIPE */ /* timer_mask_usec_ticks * @@ -91,38 +126,45 @@ static inline unsigned long timer_ticks_to_usec(unsigned long ticks) return res >> TIMER_USEC_SHIFT; } -/*** - * Returns microsecond since last clock interrupt. Note that interrupts - * will have been disabled by do_gettimeoffset() - * IRQs are disabled before entering here from do_gettimeofday() - */ - -static unsigned long s3c2410_gettimeoffset (void) +static inline unsigned long timer_freerunning_getvalue(void) { - unsigned long tdone; - unsigned long tval; - - /* work out how many ticks have gone since last timer interrupt */ + return __raw_readl(S3C2410_TCNTO(3)); +} - tval = __raw_readl(S3C2410_TCNTO(4)); - tdone = timer_startval - tval; +static inline unsigned long timer_freerunning_getticksoffset(unsigned long tval) +{ + long tdone; - /* check to see if there is an interrupt pending */ + tdone = last_free_running_tcnt - tval; + if (tdone < 0) + tdone += 0x10000; - if (s3c24xx_ostimer_pending()) { - /* re-read the timer, and try and fix up for the missed - * interrupt. Note, the interrupt may go off before the - * timer has re-loaded from wrapping. - */ + return tdone; +} - tval = __raw_readl(S3C2410_TCNTO(4)); - tdone = timer_startval - tval; +static inline unsigned long getticksoffset(void) +{ + return timer_freerunning_getticksoffset(timer_freerunning_getvalue()); +} - if (tval != 0) - tdone += timer_startval; - } +#ifdef CONFIG_IPIPE +static inline unsigned long getticksoffset_tscupdate(void) +{ + unsigned long tval; + unsigned long ticks; + + tval = timer_freerunning_getvalue(); + ticks = timer_freerunning_getticksoffset(tval); + last_free_running_tcnt = tval; + *tsc += ticks; + *last_cnt = last_free_running_tcnt; + return ticks; +} +#endif /* CONFIG_IPIPE */ - return timer_ticks_to_usec(tdone); +static unsigned long s3c2410_gettimeoffset (void) +{ + return timer_ticks_to_usec(timer_lxlost + getticksoffset()); } @@ -132,6 +174,13 @@ static unsigned long s3c2410_gettimeoffset (void) static irqreturn_t s3c2410_timer_interrupt(int irq, void *dev_id) { +#ifdef CONFIG_IPIPE + timer_lxlost = 0; + + if (!__ipipe_mach_timerstolen) + getticksoffset_tscupdate(); +#endif /* CONFIG_IPIPE */ + timer_tick(); return IRQ_HANDLED; } @@ -153,10 +202,10 @@ static struct clk *tdiv; static struct clk *timerclk; /* - * Set up timer interrupt, and return the current time in seconds. + * Set up timer interrupt. * - * Currently we only use timer4, as it is the only timer which has no - * other function that can be exploited externally + * Currently we use timer4 as event timer and timer3 as tick counter which + * permanently counts ticks without interrupt generation. */ static void s3c2410_timer_setup (void) { @@ -164,6 +213,7 @@ static void s3c2410_timer_setup (void) unsigned long tcnt; unsigned long tcfg1; unsigned long tcfg0; + unsigned long intmask; tcnt = TICK_MAX; /* default value for tcnt */ @@ -175,8 +225,8 @@ static void s3c2410_timer_setup (void) tcnt = 12000000 / HZ; tcfg1 = __raw_readl(S3C2410_TCFG1); - tcfg1 &= ~S3C2410_TCFG1_MUX4_MASK; - tcfg1 |= S3C2410_TCFG1_MUX4_TCLK1; + tcfg1 &= ~(S3C2410_TCFG1_MUX4_MASK | S3C2410_TCFG1_MUX3_MASK); + tcfg1 |= (S3C2410_TCFG1_MUX4_TCLK1 | S3C2410_TCFG1_MUX3_TCLK1); __raw_writel(tcfg1, S3C2410_TCFG1); } else { unsigned long pclk; @@ -210,6 +260,14 @@ static void s3c2410_timer_setup (void) tcfg0 = __raw_readl(S3C2410_TCFG0); tcfg1 = __raw_readl(S3C2410_TCFG1); +#ifdef CONFIG_IPIPE + tsc = (unsigned long long *)__ipipe_tsc_area; + last_cnt = (unsigned *)(tsc + 1); /* means: +8 bytes */ + barrier(); + + __ipipe_mach_ticks_per_jiffy = tcnt; +#endif /* CONFIG_IPIPE */ + /* timers reload after counting zero, so reduce the count by 1 */ tcnt--; @@ -226,23 +284,37 @@ static void s3c2410_timer_setup (void) __raw_writel(tcfg1, S3C2410_TCFG1); __raw_writel(tcfg0, S3C2410_TCFG0); - timer_startval = tcnt; - __raw_writel(tcnt, S3C2410_TCNTB(4)); - - /* ensure timer is stopped... */ + /* ensure timers are stopped... */ + tcon &= ~(0x3f<<17); + __raw_writel(tcon, S3C2410_TCON); - tcon &= ~(7<<20); - tcon |= S3C2410_TCON_T4RELOAD; - tcon |= S3C2410_TCON_T4MANUALUPD; + /* Mask timer3 interrupt. */ + intmask = __raw_readl(S3C2410_INTMSK); + intmask |= 1UL << (IRQ_TIMER3 - IRQ_EINT0); + __raw_writel(intmask, S3C2410_INTMSK); - __raw_writel(tcon, S3C2410_TCON); + /* Set timer values */ __raw_writel(tcnt, S3C2410_TCNTB(4)); __raw_writel(tcnt, S3C2410_TCMPB(4)); + __raw_writel(0xffff, S3C2410_TCNTB(3)); + __raw_writel(0xffff, S3C2410_TCMPB(3)); + + /* Set base tcon value for later programming of timer 4 by Xenomai. */ + free_running_tcon = tcon | S3C2410_TCON_T3RELOAD | S3C2410_TCON_T3START; + + /* Set auto reloads for both timers. */ + tcon |= S3C2410_TCON_T3RELOAD | S3C2410_TCON_T4RELOAD; - /* start the timer running */ - tcon |= S3C2410_TCON_T4START; - tcon &= ~S3C2410_TCON_T4MANUALUPD; + /* Manual update */ + __raw_writel(tcon | S3C2410_TCON_T3MANUALUPD + | S3C2410_TCON_T4MANUALUPD, S3C2410_TCON); + + tcon |= S3C2410_TCON_T3START | S3C2410_TCON_T4START; + /* Start timers.*/ __raw_writel(tcon, S3C2410_TCON); + + /* Save start value of timer 3 as begining of first period. */ + last_free_running_tcnt = 0xffff; } static void __init s3c2410_timer_resources(void) @@ -283,3 +355,58 @@ struct sys_timer s3c24xx_timer = { .offset = s3c2410_gettimeoffset, .resume = s3c2410_timer_setup }; + +#ifdef CONFIG_IPIPE +void __ipipe_mach_acktimer(void) +{ + __raw_writel(timer_ackval, S3C2410_SRCPND); + __raw_writel(timer_ackval, S3C2410_INTPND); +} + +notrace unsigned long long __ipipe_mach_get_tsc(void) +{ + unsigned long long result; + unsigned long flags; + + local_irq_save_hw_notrace(flags); + spin_lock(&timer_lock); + result = *tsc + getticksoffset(); + spin_unlock(&timer_lock); + local_irq_restore_hw_notrace(flags); + return result; +} +EXPORT_SYMBOL(__ipipe_mach_get_tsc); + +static inline void set_dec(unsigned long reload) +{ + __raw_writel(reload, S3C2410_TCNTB(4)); + /* Manual update */ + __raw_writel(free_running_tcon | S3C2410_TCON_T4MANUALUPD, S3C2410_TCON); + /* Start timer */ + __raw_writel(free_running_tcon | S3C2410_TCON_T4START, S3C2410_TCON); +} + +void __ipipe_mach_set_dec(unsigned long reload) +{ + unsigned long flags; + + spin_lock_irqsave(&timer_lock, flags); + timer_lxlost += getticksoffset_tscupdate(); + set_dec(reload); + spin_unlock_irqrestore(&timer_lock, flags); +} +EXPORT_SYMBOL(__ipipe_mach_set_dec); + +void __ipipe_mach_release_timer(void) +{ + free_running_tcon |= S3C2410_TCON_T4RELOAD; + __ipipe_mach_set_dec(__ipipe_mach_ticks_per_jiffy - 1); + free_running_tcon &= ~S3C2410_TCON_T4RELOAD; +} +EXPORT_SYMBOL(__ipipe_mach_release_timer); + +unsigned long __ipipe_mach_get_dec(void) +{ + return __raw_readl(S3C2410_TCNTO(4)); +} +#endif /* CONFIG_IPIPE */ diff --git a/arch/arm/plat-s3c24xx/irq.c b/arch/arm/plat-s3c24xx/irq.c index ad0d44e..8444c8b 100644 --- a/arch/arm/plat-s3c24xx/irq.c +++ b/arch/arm/plat-s3c24xx/irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2006, 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -23,6 +25,7 @@ #include <linux/interrupt.h> #include <linux/ioport.h> #include <linux/sysdev.h> +#include <linux/ipipe.h> #include <asm/irq.h> #include <asm/mach/irq.h> @@ -87,6 +90,9 @@ struct irq_chip s3c_irq_level_chip = { .name = "s3c-level", .ack = s3c_irq_maskack, .mask = s3c_irq_mask, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_maskack, +#endif /* CONFIG_IPIPE */ .unmask = s3c_irq_unmask, .set_wake = s3c_irq_wake }; @@ -283,6 +289,9 @@ static struct irq_chip s3c_irq_uart0 = { .mask = s3c_irq_uart0_mask, .unmask = s3c_irq_uart0_unmask, .ack = s3c_irq_uart0_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart0_ack, +#endif /* CONFIG_IPIPE */ }; /* UART1 */ @@ -310,6 +319,9 @@ static struct irq_chip s3c_irq_uart1 = { .mask = s3c_irq_uart1_mask, .unmask = s3c_irq_uart1_unmask, .ack = s3c_irq_uart1_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart1_ack, +#endif /* CONFIG_IPIPE */ }; /* UART2 */ @@ -337,6 +349,9 @@ static struct irq_chip s3c_irq_uart2 = { .mask = s3c_irq_uart2_mask, .unmask = s3c_irq_uart2_unmask, .ack = s3c_irq_uart2_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_uart2_ack, +#endif /* CONFIG_IPIPE */ }; /* ADC and Touchscreen */ @@ -385,10 +400,10 @@ static void s3c_irq_demux_adc(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_TC); + ipipe_handle_irq_cond(IRQ_TC); } if (subsrc & 2) { - generic_handle_irq(IRQ_ADC); + ipipe_handle_irq_cond(IRQ_ADC); } } } @@ -413,13 +428,13 @@ static void s3c_irq_demux_uart(unsigned int start) if (subsrc != 0) { if (subsrc & 1) - generic_handle_irq(start); + ipipe_handle_irq_cond(start); if (subsrc & 2) - generic_handle_irq(start+1); + ipipe_handle_irq_cond(start+1); if (subsrc & 4) - generic_handle_irq(start+2); + ipipe_handle_irq_cond(start+2); } } @@ -466,7 +481,7 @@ s3c_irq_demux_extint8(unsigned int irq, eintpnd &= ~(1<<irq); irq += (IRQ_EINT4 - 4); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } } @@ -489,7 +504,7 @@ s3c_irq_demux_extint4t7(unsigned int irq, irq += (IRQ_EINT4 - 4); - generic_handle_irq(irq); + ipipe_handle_irq_cond(irq); } } diff --git a/arch/arm/plat-s3c24xx/s3c244x-irq.c b/arch/arm/plat-s3c24xx/s3c244x-irq.c index a75c0c2..5dd2dd1 100644 --- a/arch/arm/plat-s3c24xx/s3c244x-irq.c +++ b/arch/arm/plat-s3c24xx/s3c244x-irq.c @@ -3,6 +3,8 @@ * Copyright (c) 2003-2004 Simtec Electronics * Ben Dooks <ben@simtec.co.uk> * + * Copyright (C) 2007 Sebastian Smolorz <ssmolorz@emlix.com>, emlix GmbH + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or @@ -25,6 +27,7 @@ #include <linux/ioport.h> #include <linux/sysdev.h> #include <linux/io.h> +#include <linux/ipipe.h> #include <mach/hardware.h> #include <asm/irq.h> @@ -57,10 +60,10 @@ static void s3c_irq_demux_cam(unsigned int irq, if (subsrc != 0) { if (subsrc & 1) { - generic_handle_irq(IRQ_S3C2440_CAM_C); + ipipe_handle_irq_cond(IRQ_S3C2440_CAM_C); } if (subsrc & 2) { - generic_handle_irq(IRQ_S3C2440_CAM_P); + ipipe_handle_irq_cond(IRQ_S3C2440_CAM_P); } } } @@ -89,6 +92,9 @@ static struct irq_chip s3c_irq_cam = { .mask = s3c_irq_cam_mask, .unmask = s3c_irq_cam_unmask, .ack = s3c_irq_cam_ack, +#ifdef CONFIG_IPIPE + .mask_ack = s3c_irq_cam_ack, +#endif /* CONFIG_IPIPE */ }; static int s3c244x_irq_add(struct sys_device *sysdev) diff --git a/arch/arm/vfp/entry.S b/arch/arm/vfp/entry.S index 4fa9903..e46bae3 100644 --- a/arch/arm/vfp/entry.S +++ b/arch/arm/vfp/entry.S @@ -25,7 +25,6 @@ ENTRY(do_vfp) add r11, r4, #1 @ increment it str r11, [r10, #TI_PREEMPT] #endif - enable_irq ldr r4, .LCvfp ldr r11, [r10, #TI_CPU] @ CPU number add r10, r10, #TI_VFPSTATE @ r10 = workspace @@ -33,6 +32,7 @@ ENTRY(do_vfp) ENDPROC(do_vfp) ENTRY(vfp_null_entry) + enable_irq #ifdef CONFIG_PREEMPT get_thread_info r10 ldr r4, [r10, #TI_PREEMPT] @ get preempt count @@ -51,6 +51,7 @@ ENDPROC(vfp_null_entry) __INIT ENTRY(vfp_testing_entry) + enable_irq #ifdef CONFIG_PREEMPT get_thread_info r10 ldr r4, [r10, #TI_PREEMPT] @ get preempt count diff --git a/arch/arm/vfp/vfphw.S b/arch/arm/vfp/vfphw.S index 66dc2d0..b1b9c31 100644 --- a/arch/arm/vfp/vfphw.S +++ b/arch/arm/vfp/vfphw.S @@ -19,7 +19,7 @@ #include "../kernel/entry-header.S" .macro DBGSTR, str -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} add r0, pc, #4 bl printk @@ -31,7 +31,7 @@ .endm .macro DBGSTR1, str, arg -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} mov r1, \arg add r0, pc, #4 @@ -44,7 +44,7 @@ .endm .macro DBGSTR3, str, arg1, arg2, arg3 -#ifdef DEBUG +#if defined(DEBUG) && !defined(CONFIG_IPIPE) stmfd sp!, {r0-r3, ip, lr} mov r3, \arg3 mov r2, \arg2 @@ -87,6 +87,11 @@ ENTRY(vfp_support_entry) @ still there. In this case, we do @ not want to drop a pending exception. + enable_irq +#ifdef CONFIG_IPIPE + disable_irq + ldr r4, [r3, r11, lsl #2] @ reload last_VFP_context pointer +#endif VFPFMXR FPEXC, r5 @ enable VFP, disable any pending @ exceptions, so we can get at the @ rest of it @@ -139,6 +144,7 @@ check_for_exception: @ out before setting an FPEXC that @ stops us reading stuff VFPFMXR FPEXC, r1 @ restore FPEXC last + enable_irq_cond sub r2, r2, #4 str r2, [sp, #S_PC] @ retry the instruction #ifdef CONFIG_PREEMPT @@ -164,6 +170,7 @@ look_for_VFP_exceptions: @ Fall into hand on to next handler - appropriate coproc instr @ not recognised by VFP + enable_irq_cond DBGSTR "not VFP" #ifdef CONFIG_PREEMPT get_thread_info r10 diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index a63c4be..1fdff52 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -44,6 +44,7 @@ unsigned int VFP_arch; static void vfp_thread_flush(struct thread_info *thread) { union vfp_state *vfp = &thread->vfpstate; + unsigned long flags; unsigned int cpu; memset(vfp, 0, sizeof(union vfp_state)); @@ -56,22 +57,23 @@ static void vfp_thread_flush(struct thread_info *thread) * that the modification of last_VFP_context[] and hardware disable * are done for the same CPU and without preemption. */ - cpu = get_cpu(); + cpu = ipipe_get_cpu(flags); if (last_VFP_context[cpu] == vfp) last_VFP_context[cpu] = NULL; fmxr(FPEXC, fmrx(FPEXC) & ~FPEXC_EN); - put_cpu(); + ipipe_put_cpu(flags); } static void vfp_thread_exit(struct thread_info *thread) { /* release case: Per-thread VFP cleanup. */ union vfp_state *vfp = &thread->vfpstate; - unsigned int cpu = get_cpu(); + unsigned long flags; + unsigned int cpu = ipipe_get_cpu(flags); if (last_VFP_context[cpu] == vfp) last_VFP_context[cpu] = NULL; - put_cpu(); + ipipe_put_cpu(flags); } /* @@ -100,10 +102,13 @@ static void vfp_thread_exit(struct thread_info *thread) static int vfp_notifier(struct notifier_block *self, unsigned long cmd, void *v) { struct thread_info *thread = v; + unsigned long flags; if (likely(cmd == THREAD_NOTIFY_SWITCH)) { - u32 fpexc = fmrx(FPEXC); + u32 fpexc; + local_irq_save_hw_cond(flags); + fpexc = fmrx(FPEXC); #ifdef CONFIG_SMP unsigned int cpu = thread->cpu; @@ -130,6 +135,7 @@ static int vfp_notifier(struct notifier_block *self, unsigned long cmd, void *v) * old state. */ fmxr(FPEXC, fpexc & ~FPEXC_EN); + local_irq_restore_hw_cond(flags); return NOTIFY_DONE; } @@ -266,7 +272,7 @@ static u32 vfp_emulate_instruction(u32 inst, u32 fpscr, struct pt_regs *regs) */ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) { - u32 fpscr, orig_fpscr, fpsid, exceptions; + u32 fpscr, orig_fpscr, fpsid, exceptions, next_trigger = 0; pr_debug("VFP: bounce: trigger %08x fpexc %08x\n", trigger, fpexc); @@ -296,6 +302,7 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) /* * Synchronous exception, emulate the trigger instruction */ + local_irq_enable_hw_cond(); goto emulate; } @@ -308,13 +315,24 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) trigger = fmrx(FPINST); regs->ARM_pc -= 4; #endif - } else if (!(fpexc & FPEXC_DEX)) { + if (fpexc & FPEXC_FP2V) { + /* + * The barrier() here prevents fpinst2 being read + * before the condition above. + */ + barrier(); + next_trigger = fmrx(FPINST2); + } + } + local_irq_enable_hw_cond(); + + if (!(fpexc & (FPEXC_EX | FPEXC_DEX))) { /* * Illegal combination of bits. It can be caused by an * unallocated VFP instruction but with FPSCR.IXE set and not * on VFP subarch 1. */ - vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs); + vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs); goto exit; } @@ -348,18 +366,14 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) if (fpexc ^ (FPEXC_EX | FPEXC_FP2V)) goto exit; - /* - * The barrier() here prevents fpinst2 being read - * before the condition above. - */ - barrier(); - trigger = fmrx(FPINST2); + trigger = next_trigger; emulate: exceptions = vfp_emulate_instruction(trigger, orig_fpscr, regs); if (exceptions) vfp_raise_exceptions(exceptions, trigger, orig_fpscr, regs); exit: + local_irq_enable_hw_cond(); preempt_enable(); } diff --git a/drivers/i2c/busses/i2c-omap.c b/drivers/i2c/busses/i2c-omap.c index 0037e31..8d608a1 100644 --- a/drivers/i2c/busses/i2c-omap.c +++ b/drivers/i2c/busses/i2c-omap.c @@ -478,6 +478,10 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, if (msg->len == 0) return -EINVAL; +#ifdef CONFIG_IPIPE + disable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ + omap_i2c_write_reg(dev, OMAP_I2C_SA_REG, msg->addr); /* REVISIT: Could the STB bit of I2C_CON be used with probing? */ @@ -521,6 +525,9 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, /* Let the user know if i2c is in a bad state */ if (time_after(jiffies, delay)) { +#ifdef CONFIG_IPIPE + enable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ dev_err(dev->dev, "controller timed out " "waiting for start condition to finish\n"); return -ETIMEDOUT; @@ -532,6 +539,9 @@ static int omap_i2c_xfer_msg(struct i2c_adapter *adap, w &= ~OMAP_I2C_CON_STT; omap_i2c_write_reg(dev, OMAP_I2C_CON_REG, w); } +#ifdef CONFIG_IPIPE + enable_irq(dev->irq); +#endif /* CONFIG_IPIPE */ /* * REVISIT: We should abort the transfer on signals, but the bus goes diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index e3551d2..d8f7317 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -46,7 +46,7 @@ config ATMEL_TCLIB config ATMEL_TCB_CLKSRC bool "TC Block Clocksource" - depends on ATMEL_TCLIB && GENERIC_TIME + depends on ATMEL_TCLIB && GENERIC_TIME && !IPIPE default y help Select this to get a high precision clocksource based on a diff --git a/drivers/pci/htirq.c b/drivers/pci/htirq.c index 737a1c4..15e81de 100644 --- a/drivers/pci/htirq.c +++ b/drivers/pci/htirq.c @@ -21,7 +21,7 @@ * With multiple simultaneous hypertransport irq devices it might pay * to make this more fine grained. But start with simple, stupid, and correct. */ -static DEFINE_SPINLOCK(ht_irq_lock); +static IPIPE_DEFINE_SPINLOCK(ht_irq_lock); struct ht_irq_cfg { struct pci_dev *dev; diff --git a/drivers/serial/8250.c b/drivers/serial/8250.c index e9b15c3..f93771a 100644 --- a/drivers/serial/8250.c +++ b/drivers/serial/8250.c @@ -3020,6 +3020,51 @@ static int serial8250_resume(struct platform_device *dev) return 0; } +#if defined(CONFIG_IPIPE_DEBUG) && defined(CONFIG_SERIAL_8250_CONSOLE) + +#include <stdarg.h> + +void __weak __ipipe_serial_debug(const char *fmt, ...) +{ + struct uart_8250_port *up = &serial8250_ports[0]; + unsigned int ier, count; + unsigned long flags; + char buf[128]; + va_list ap; + + va_start(ap, fmt); + vsprintf(buf, fmt, ap); + va_end(ap); + count = strlen(buf); + + touch_nmi_watchdog(); + + local_irq_save_hw(flags); + + /* + * First save the IER then disable the interrupts + */ + ier = serial_in(up, UART_IER); + + if (up->capabilities & UART_CAP_UUE) + serial_out(up, UART_IER, UART_IER_UUE); + else + serial_out(up, UART_IER, 0); + + uart_console_write(&up->port, buf, count, serial8250_console_putchar); + + /* + * Finally, wait for transmitter to become empty + * and restore the IER + */ + wait_for_xmitr(up, BOTH_EMPTY); + serial_out(up, UART_IER, ier); + + local_irq_restore_hw(flags); +} + +#endif + static struct platform_driver serial8250_isa_driver = { .probe = serial8250_probe, .remove = __devexit_p(serial8250_remove), diff --git a/fs/exec.c b/fs/exec.c index cce6bbd..1f2021d 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -714,6 +714,7 @@ static int exec_mmap(struct mm_struct *mm) { struct task_struct *tsk; struct mm_struct * old_mm, *active_mm; + unsigned long flags; /* Notify parent that we're no longer interested in the old VM */ tsk = current; @@ -736,8 +737,10 @@ static int exec_mmap(struct mm_struct *mm) task_lock(tsk); active_mm = tsk->active_mm; tsk->mm = mm; + ipipe_mm_switch_protect(flags); tsk->active_mm = mm; activate_mm(active_mm, mm); + ipipe_mm_switch_unprotect(flags); task_unlock(tsk); arch_pick_mmap_layout(mm); if (old_mm) { diff --git a/fs/proc/array.c b/fs/proc/array.c index 13b5d07..e2c2b55 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -144,6 +144,10 @@ static const char *task_state_array[] = { "x (dead)", /* 64 */ "K (wakekill)", /* 128 */ "W (waking)", /* 256 */ +#ifdef CONFIG_IPIPE + "A (atomic switch)", /* 512 */ + "N (wakeup disabled)", /* 1024 */ +#endif }; static inline const char *get_task_state(struct task_struct *tsk) diff --git a/include/asm-generic/atomic.h b/include/asm-generic/atomic.h index c99c64d..5d01b93 100644 --- a/include/asm-generic/atomic.h +++ b/include/asm-generic/atomic.h @@ -60,11 +60,11 @@ static inline int atomic_add_return(int i, atomic_t *v) unsigned long flags; int temp; - local_irq_save(flags); + local_irq_save_hw(flags); temp = v->counter; temp += i; v->counter = temp; - local_irq_restore(flags); + local_irq_restore_hw(flags); return temp; } @@ -82,11 +82,11 @@ static inline int atomic_sub_return(int i, atomic_t *v) unsigned long flags; int temp; - local_irq_save(flags); + local_irq_save_hw(flags); temp = v->counter; temp -= i; v->counter = temp; - local_irq_restore(flags); + local_irq_restore_hw(flags); return temp; } @@ -139,9 +139,9 @@ static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) unsigned long flags; mask = ~mask; - local_irq_save(flags); + local_irq_save_hw(flags); *addr &= mask; - local_irq_restore(flags); + local_irq_restore_hw(flags); } #define atomic_xchg(ptr, v) (xchg(&(ptr)->counter, (v))) diff --git a/include/asm-generic/bitops/atomic.h b/include/asm-generic/bitops/atomic.h index ecc44a8..5caf6e9 100644 --- a/include/asm-generic/bitops/atomic.h +++ b/include/asm-generic/bitops/atomic.h @@ -21,20 +21,20 @@ extern arch_spinlock_t __atomic_hash[ATOMIC_HASH_SIZE] __lock_aligned; * this is the substitute */ #define _atomic_spin_lock_irqsave(l,f) do { \ arch_spinlock_t *s = ATOMIC_HASH(l); \ - local_irq_save(f); \ + local_irq_save_hw(f); \ arch_spin_lock(s); \ } while(0) #define _atomic_spin_unlock_irqrestore(l,f) do { \ arch_spinlock_t *s = ATOMIC_HASH(l); \ arch_spin_unlock(s); \ - local_irq_restore(f); \ + local_irq_restore_hw(f); \ } while(0) #else -# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save(f); } while (0) -# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore(f); } while (0) +# define _atomic_spin_lock_irqsave(l,f) do { local_irq_save_hw(f); } while (0) +# define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore_hw(f); } while (0) #endif /* diff --git a/include/asm-generic/cmpxchg-local.h b/include/asm-generic/cmpxchg-local.h index b2ba2fc..ed01ab9 100644 --- a/include/asm-generic/cmpxchg-local.h +++ b/include/asm-generic/cmpxchg-local.h @@ -20,7 +20,7 @@ static inline unsigned long __cmpxchg_local_generic(volatile void *ptr, if (size == 8 && sizeof(unsigned long) != 8) wrong_size_cmpxchg(ptr); - local_irq_save(flags); + local_irq_save_hw(flags); switch (size) { case 1: prev = *(u8 *)ptr; if (prev == old) @@ -41,7 +41,7 @@ static inline unsigned long __cmpxchg_local_generic(volatile void *ptr, default: wrong_size_cmpxchg(ptr); } - local_irq_restore(flags); + local_irq_restore_hw(flags); return prev; } @@ -54,11 +54,11 @@ static inline u64 __cmpxchg64_local_generic(volatile void *ptr, u64 prev; unsigned long flags; - local_irq_save(flags); + local_irq_save_hw(flags); prev = *(u64 *)ptr; if (prev == old) *(u64 *)ptr = new; - local_irq_restore(flags); + local_irq_restore_hw(flags); return prev; } diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h index 8087b90..d5c9897 100644 --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -59,6 +59,20 @@ extern unsigned long __per_cpu_offset[NR_CPUS]; #define this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) #define __this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset) +#ifdef CONFIG_IPIPE +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) +extern int __ipipe_check_percpu_access(void); +#define __ipipe_local_cpu_offset \ + ({ \ + WARN_ON_ONCE(__ipipe_check_percpu_access()); \ + __my_cpu_offset; \ + }) +#else +#define __ipipe_local_cpu_offset __my_cpu_offset +#endif +#define __ipipe_get_cpu_var(var) \ + (*SHIFT_PERCPU_PTR(&per_cpu_var(var), __ipipe_local_cpu_offset)) +#endif /* CONFIG_IPIPE */ #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA extern void setup_per_cpu_areas(void); @@ -69,6 +83,7 @@ extern void setup_per_cpu_areas(void); #define per_cpu(var, cpu) (*((void)(cpu), &per_cpu_var(var))) #define __get_cpu_var(var) per_cpu_var(var) #define __raw_get_cpu_var(var) per_cpu_var(var) +#define __ipipe_get_cpu_var(var) __raw_get_cpu_var(var) #define this_cpu_ptr(ptr) per_cpu_ptr(ptr, 0) #define __this_cpu_ptr(ptr) this_cpu_ptr(ptr) diff --git a/include/asm-generic/resource.h b/include/asm-generic/resource.h index 587566f..66d60cd 100644 --- a/include/asm-generic/resource.h +++ b/include/asm-generic/resource.h @@ -58,6 +58,14 @@ #endif /* + * Limit the stack by to some sane default: root can always + * increase this limit if needed.. 8MB seems reasonable. + */ +#ifndef _STK_LIM +# define _STK_LIM (8*1024*1024) +#endif + +/* * RLIMIT_STACK default maximum - some architectures override it: */ #ifndef _STK_LIM_MAX diff --git a/include/linux/autoconf.h b/include/linux/autoconf.h new file mode 100644 index 0000000..9c633d6 --- /dev/null +++ b/include/linux/autoconf.h @@ -0,0 +1,552 @@ +/* + * Automatically generated C config: don't edit + * Linux kernel version: 2.6.32.7 + * Thu Feb 4 10:20:41 2010 + */ +#define AUTOCONF_INCLUDED +#define CONFIG_ACPI_AC 1 +#define CONFIG_ACPI_SYSFS_POWER 1 +#define CONFIG_DEBUG_SPINLOCK_SLEEP 1 +#define CONFIG_XENO_OPT_NATIVE_MPS 1 +#define CONFIG_FRAME_WARN 2048 +#define CONFIG_ARCH_DEFCONFIG "arch/x86/configs/x86_64_defconfig" +#define CONFIG_XENO_OPT_PRIOCPL 1 +#define CONFIG_SND_HDA_CODEC_CIRRUS 1 +#define CONFIG_TASK_XACCT 1 +#define CONFIG_POSIX_MQUEUE_SYSCTL 1 +#define CONFIG_ARCH_MAY_HAVE_PC_FDC 1 +#define CONFIG_CRYPTO_BLKCIPHER2 1 +#define CONFIG_CRYPTO_MD5 1 +#define CONFIG_X86_MCE_THRESHOLD 1 +#define CONFIG_PROC_KCORE 1 +#define CONFIG_XENO_OPT_TIMER_LIST 1 +#define CONFIG_CRYPTO_WORKQUEUE 1 +#define CONFIG_FTRACE_MCOUNT_RECORD 1 +#define CONFIG_CRYPTO_CBC 1 +#define CONFIG_TRACING 1 +#define CONFIG_DEVKMEM 1 +#define CONFIG_HAVE_FUNCTION_TRACER 1 +#define CONFIG_X86_PAT 1 +#define CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ 12 +#define CONFIG_XENO_OPT_NUCLEUS 1 +#define CONFIG_USE_GENERIC_SMP_HELPERS 1 +#define CONFIG_HAVE_KERNEL_BZIP2 1 +#define CONFIG_SND_SEQ_HRTIMER_DEFAULT 1 +#define CONFIG_SERIAL_8250_SHARE_IRQ 1 +#define CONFIG_PNPACPI 1 +#define CONFIG_TIMERFD 1 +#define CONFIG_ENABLE_MUST_CHECK 1 +#define CONFIG_EVENTFD 1 +#define CONFIG_X86_LOCAL_APIC 1 +#define CONFIG_NOP_TRACER 1 +#define CONFIG_BLK_DEV_SD 1 +#define CONFIG_HAS_DMA 1 +#define CONFIG_THERMAL 1 +#define CONFIG_X86_CPU 1 +#define CONFIG_EXT3_FS_XATTR 1 +#define CONFIG_NFS_COMMON 1 +#define CONFIG_X86_TSC 1 +#define CONFIG_HAVE_CPUMASK_OF_CPU_MAP 1 +#define CONFIG_SERIO_SERPORT 1 +#define CONFIG_SOUND_OSS_CORE_PRECLAIM 1 +#define CONFIG_HW_CONSOLE 1 +#define CONFIG_XENOMAI 1 +#define CONFIG_HPET_TIMER 1 +#define CONFIG_VGA_ARB 1 +#define CONFIG_CONSOLE_TRANSLATIONS 1 +#define CONFIG_ATA_SFF 1 +#define CONFIG_CHR_DEV_SG 1 +#define CONFIG_XENO_OPT_SYS_HEAPSZ 256 +#define CONFIG_KEYBOARD_ATKBD 1 +#define CONFIG_RCU_FANOUT 64 +#define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS 1 +#define CONFIG_HAS_IOPORT 1 +#define CONFIG_HAVE_ARCH_KGDB 1 +#define CONFIG_USB_EHCI_TT_NEWSCHED 1 +#define CONFIG_COMPAT_BINFMT_ELF 1 +#define CONFIG_PHYS_ADDR_T_64BIT 1 +#define CONFIG_XENO_OPT_PIPE 1 +#define CONFIG_MODULES 1 +#define CONFIG_HAVE_SETUP_PER_CPU_AREA 1 +#define CONFIG_OUTPUT_FORMAT "elf64-x86-64" +#define CONFIG_DYNAMIC_FTRACE 1 +#define CONFIG_USB_ARCH_HAS_EHCI 1 +#define CONFIG_CPU_SUP_INTEL 1 +#define CONFIG_PCI_QUIRKS 1 +#define CONFIG_INPUT_MISC 1 +#define CONFIG_MOUSE_PS2_TRACKPOINT 1 +#define CONFIG_BLK_DEV_SR 1 +#define CONFIG_EXT3_FS 1 +#define CONFIG_VT_CONSOLE 1 +#define CONFIG_CRYPTO_DES 1 +#define CONFIG_GENERIC_TRACER 1 +#define CONFIG_SND_MIXER_OSS 1 +#define CONFIG_PREEMPT 1 +#define CONFIG_ACPI 1 +#define CONFIG_USB_HID 1 +#define CONFIG_RTC_LIB 1 +#define CONFIG_SPARSEMEM_VMEMMAP 1 +#define CONFIG_GENERIC_IRQ_PROBE 1 +#define CONFIG_USER_STACKTRACE_SUPPORT 1 +#define CONFIG_CRYPTO_ALGAPI2 1 +#define CONFIG_SND_PCI 1 +#define CONFIG_DEFCONFIG_LIST "/lib/modules/$UNAME_RELEASE/.config" +#define CONFIG_X86_TRAMPOLINE 1 +#define CONFIG_DETECT_SOFTLOCKUP 1 +#define CONFIG_SLAB 1 +#define CONFIG_HAVE_LATENCYTOP_SUPPORT 1 +#define CONFIG_HOTPLUG 1 +#define CONFIG_USB_ARCH_HAS_OHCI 1 +#define CONFIG_SND_PCM 1 +#define CONFIG_DEVPORT 1 +#define CONFIG_SWIOTLB 1 +#define CONFIG_PCI_MSI 1 +#define CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK 1 +#define CONFIG_SPARSEMEM_VMEMMAP_ENABLE 1 +#define CONFIG_COMPAT_FOR_U64_ALIGNMENT 1 +#define CONFIG_SPARSEMEM_EXTREME 1 +#define CONFIG_SUNRPC_GSS 1 +#define CONFIG_PHYSICAL_ALIGN 0x1000000 +#define CONFIG_ARCH_PROC_KCORE_TEXT 1 +#define CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ 1024 +#define CONFIG_DEBUG_SPINLOCK 1 +#define CONFIG_VFAT_FS 1 +#define CONFIG_SERIAL_8250_PCI 1 +#define CONFIG_XENO_DRIVERS_SWITCHTEST 1 +#define CONFIG_IO_DELAY_TYPE_UDELAY 2 +#define CONFIG_R8169 1 +#define CONFIG_EMBEDDED 1 +#define CONFIG_GENERIC_ACL 1 +#define CONFIG_PROC_FS 1 +#define CONFIG_ARCH_HAS_CACHE_LINE_SIZE 1 +#define CONFIG_XENO_OPT_TIMING_VIRTICK 1000 +#define CONFIG_X86_CMPXCHG 1 +#define CONFIG_XENO_OPT_NATIVE_HEAP 1 +#define CONFIG_INET 1 +#define CONFIG_ARCH_SPARSEMEM_ENABLE 1 +#define CONFIG_RT_MUTEXES 1 +#define CONFIG_HPET 1 +#define CONFIG_SYSVIPC 1 +#define CONFIG_WLAN 1 +#define CONFIG_SCSI 1 +#define CONFIG_TCP_CONG_CUBIC 1 +#define CONFIG_X86_L1_CACHE_SHIFT 6 +#define CONFIG_SERIAL_8250_RSA 1 +#define CONFIG_ATA_VERBOSE_ERROR 1 +#define CONFIG_HAVE_IOREMAP_PROT 1 +#define CONFIG_SND_HDA_CODEC_REALTEK 1 +#define CONFIG_ACPI_PROCFS_POWER 1 +#define CONFIG_XENO_OPT_NATIVE_COND 1 +#define CONFIG_PROC_PAGE_MONITOR 1 +#define CONFIG_XENO_OPT_RTDM_FILDES 128 +#define CONFIG_IOSCHED_CFQ 1 +#define CONFIG_X86_IO_APIC 1 +#define CONFIG_PAGEFLAGS_EXTENDED 1 +#define CONFIG_NR_CPUS 8 +#define CONFIG_X86_EXTENDED_PLATFORM 1 +#define CONFIG_ARCH_HAS_DEFAULT_IDLE 1 +#define CONFIG_GENERIC_BUG 1 +#define CONFIG_SERIAL_8250_PNP 1 +#define CONFIG_CRYPTO_MANAGER 1 +#define CONFIG_X86_L1_CACHE_BYTES 64 +#define CONFIG_SWAP 1 +#define CONFIG_HAVE_MLOCKED_PAGE_BIT 1 +#define CONFIG_IPIPE_DEBUG 1 +#define CONFIG_EXT3_FS_POSIX_ACL 1 +#define CONFIG_AC97_BUS 1 +#define CONFIG_CRC32 1 +#define CONFIG_X86_MPPARSE 1 +#define CONFIG_DEFAULT_CFQ 1 +#define CONFIG_XENO_OPT_NATIVE_PIPE 1 +#define CONFIG_FAT_DEFAULT_IOCHARSET "iso8859-15" +#define CONFIG_INPUT_KEYBOARD 1 +#define CONFIG_EXTRA_FIRMWARE "" +#define CONFIG_PACKET_MMAP 1 +#define CONFIG_HAVE_DMA_ATTRS 1 +#define CONFIG_SND_SEQUENCER_OSS 1 +#define CONFIG_UNIX 1 +#define CONFIG_CPU_IDLE_GOV_MENU 1 +#define CONFIG_GENERIC_CPU 1 +#define CONFIG_XENO_OPT_RTDM_PERIOD 0 +#define CONFIG_ARCH_SELECT_MEMORY_MODEL 1 +#define CONFIG_MTRR 1 +#define CONFIG_PCI_DOMAINS 1 +#define CONFIG_ISA_DMA_API 1 +#define CONFIG_SERIAL_CORE 1 +#define CONFIG_SIGNALFD 1 +#define CONFIG_ACPI_PROCFS 1 +#define CONFIG_SCHED_OMIT_FRAME_POINTER 1 +#define CONFIG_RING_BUFFER 1 +#define CONFIG_UID16 1 +#define CONFIG_LOCK_KERNEL 1 +#define CONFIG_HAVE_SYSCALL_TRACEPOINTS 1 +#define CONFIG_64BIT 1 +#define CONFIG_USB_EHCI_ROOT_HUB_TT 1 +#define CONFIG_SCSI_WAIT_SCAN_MODULE 1 +#define CONFIG_PHYSICAL_START 0x200000 +#define CONFIG_IKCONFIG 1 +#define CONFIG_MOUSE_PS2_ALPS 1 +#define CONFIG_IP_FIB_HASH 1 +#define CONFIG_DEFAULT_MMAP_MIN_ADDR 4096 +#define CONFIG_XENO_FASTSYNCH 1 +#define CONFIG_TRACEPOINTS 1 +#define CONFIG_ARCH_USES_PG_UNCACHED 1 +#define CONFIG_ANON_INODES 1 +#define CONFIG_GENERIC_FIND_LAST_BIT 1 +#define CONFIG_SLABINFO 1 +#define CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST 1 +#define CONFIG_VGA_CONSOLE 1 +#define CONFIG_PCIEPORTBUS 1 +#define CONFIG_NLS_DEFAULT "iso8859-1" +#define CONFIG_SND_RAWMIDI 1 +#define CONFIG_ACPI_FAN 1 +#define CONFIG_ATA_ACPI 1 +#define CONFIG_SPLIT_PTLOCK_CPUS 4 +#define CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ 1 +#define CONFIG_SYSVIPC_COMPAT 1 +#define CONFIG_X86_SUPPORTS_MEMORY_FAILURE 1 +#define CONFIG_HAVE_DMA_API_DEBUG 1 +#define CONFIG_ARCH_WANT_FRAME_POINTERS 1 +#define CONFIG_ARCH_SUPPORTS_MSI 1 +#define CONFIG_HAVE_IDE 1 +#define CONFIG_IO_DELAY_TYPE_0X80 0 +#define CONFIG_LOCKD_V4 1 +#define CONFIG_SCHED_MC 1 +#define CONFIG_VM_EVENT_COUNTERS 1 +#define CONFIG_ACPI_BUTTON 1 +#define CONFIG_HAVE_FTRACE_NMI_ENTER 1 +#define CONFIG_CRYPTO_RNG2 1 +#define CONFIG_GENERIC_FIND_NEXT_BIT 1 +#define CONFIG_X86_CMOV 1 +#define CONFIG_XENO_OPT_TIMING_SCHEDLAT 0 +#define CONFIG_KERNEL_GZIP 1 +#define CONFIG_GENERIC_TIME_VSYSCALL 1 +#define CONFIG_UNUSED_SYMBOLS 1 +#define CONFIG_SYSCTL_SYSCALL 1 +#define CONFIG_MTRR_SANITIZER 1 +#define CONFIG_USB_STORAGE 1 +#define CONFIG_SND_HRTIMER 1 +#define CONFIG_SERIAL_8250_RUNTIME_UARTS 4 +#define CONFIG_COMPAT_VDSO 1 +#define CONFIG_ELF_CORE 1 +#define CONFIG_DEBUG_FS 1 +#define CONFIG_ARCH_SUSPEND_POSSIBLE 1 +#define CONFIG_SERIAL_8250_CONSOLE 1 +#define CONFIG_X86_DEBUGCTLMSR 1 +#define CONFIG_MAGIC_SYSRQ 1 +#define CONFIG_SND_DRIVERS 1 +#define CONFIG_SCSI_DMA 1 +#define CONFIG_DEFAULT_IOSCHED "cfq" +#define CONFIG_CRYPTO_HASH 1 +#define CONFIG_UNIX98_PTYS 1 +#define CONFIG_IP_PNP_DHCP 1 +#define CONFIG_ACPI_SBS 1 +#define CONFIG_X86_THERMAL_VECTOR 1 +#define CONFIG_XENO_SKIN_RTDM 1 +#define CONFIG_SOUND_OSS_CORE 1 +#define CONFIG_CPU_IDLE 1 +#define CONFIG_SND_MPU401_UART 1 +#define CONFIG_SND_VMASTER 1 +#define CONFIG_SYN_COOKIES 1 +#define CONFIG_X86_INTERNODE_CACHE_BYTES 64 +#define CONFIG_DMIID 1 +#define CONFIG_MOUSE_PS2_LIFEBOOK 1 +#define CONFIG_SCSI_MULTI_LUN 1 +#define CONFIG_GENERIC_ISA_DMA 1 +#define CONFIG_DEFAULT_IO_DELAY_TYPE 0 +#define CONFIG_PNP_DEBUG_MESSAGES 1 +#define CONFIG_BLOCK 1 +#define CONFIG_SND_OSSEMUL 1 +#define CONFIG_GENERIC_CLOCKEVENTS_BUILD 1 +#define CONFIG_GENERIC_HWEIGHT 1 +#define CONFIG_CRYPTO_MANAGER2 1 +#define CONFIG_LOCKD 1 +#define CONFIG_ZONE_DMA 1 +#define CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE 0 +#define CONFIG_ENABLE_WARN_DEPRECATED 1 +#define CONFIG_TIMER_STATS 1 +#define CONFIG_ATA_PIIX 1 +#define CONFIG_KALLSYMS 1 +#define CONFIG_ARCH_HIBERNATION_POSSIBLE 1 +#define CONFIG_HID_SUPPORT 1 +#define CONFIG_HAVE_KVM 1 +#define CONFIG_FSNOTIFY 1 +#define CONFIG_SYSVIPC_SYSCTL 1 +#define CONFIG_HAVE_UNSTABLE_SCHED_CLOCK 1 +#define CONFIG_X86_VERBOSE_BOOTUP 1 +#define CONFIG_NLS_UTF8 1 +#define CONFIG_SND_HDA_CODEC_CA0110 1 +#define CONFIG_FIRMWARE_IN_KERNEL 1 +#define CONFIG_TASK_IO_ACCOUNTING 1 +#define CONFIG_PROC_SYSCTL 1 +#define CONFIG_CONTEXT_SWITCH_TRACER 1 +#define CONFIG_IO_DELAY_TYPE_0XED 1 +#define CONFIG_IPIPE 1 +#define CONFIG_SND_PCM_OSS 1 +#define CONFIG_INPUT_MOUSE 1 +#define CONFIG_FIRMWARE_MEMMAP 1 +#define CONFIG_SERIAL_8250_MANY_PORTS 1 +#define CONFIG_TREE_RCU 1 +#define CONFIG_XENO_OPT_STATS 1 +#define CONFIG_SHMEM 1 +#define CONFIG_TASK_DELAY_ACCT 1 +#define CONFIG_ARCH_HAS_CPU_RELAX 1 +#define CONFIG_INET_LRO 1 +#define CONFIG_EPOLL 1 +#define CONFIG_CRYPTO_AEAD2 1 +#define CONFIG_RELAY 1 +#define CONFIG_XENO_OPT_NATIVE_ALARM 1 +#define CONFIG_RPCSEC_GSS_KRB5 1 +#define CONFIG_NLS_CODEPAGE_850 1 +#define CONFIG_NLS_CODEPAGE_437 1 +#define CONFIG_STACKTRACE_SUPPORT 1 +#define CONFIG_SERIO 1 +#define CONFIG_UEVENT_HELPER_PATH "/sbin/hotplug" +#define CONFIG_XENO_HW_FPU 1 +#define CONFIG_HAVE_KRETPROBES 1 +#define CONFIG_FUNCTION_TRACER 1 +#define CONFIG_FILE_LOCKING 1 +#define CONFIG_USB_SUPPORT 1 +#define CONFIG_SND_VERBOSE_PROCFS 1 +#define CONFIG_HUGETLB_PAGE 1 +#define CONFIG_IA32_AOUT 1 +#define CONFIG_DEBUG_KERNEL 1 +#define CONFIG_HAVE_ARCH_KMEMCHECK 1 +#define CONFIG_SOUND 1 +#define CONFIG_ROOT_NFS 1 +#define CONFIG_TMPFS 1 +#define CONFIG_RTC_INTF_DEV_UIE_EMUL 1 +#define CONFIG_SCHED_HRTICK 1 +#define CONFIG_GENERIC_TIME 1 +#define CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST 1 +#define CONFIG_HAVE_FTRACE_MCOUNT_RECORD 1 +#define CONFIG_FUTEX 1 +#define CONFIG_SERIO_LIBPS2 1 +#define CONFIG_ARCH_SPARSEMEM_DEFAULT 1 +#define CONFIG_BLOCK_COMPAT 1 +#define CONFIG_EXPERIMENTAL 1 +#define CONFIG_GENERIC_CLOCKEVENTS_BROADCAST 1 +#define CONFIG_X86_64 1 +#define CONFIG_USB_EHCI_HCD 1 +#define CONFIG_INPUT 1 +#define CONFIG_VIRT_TO_BUS 1 +#define CONFIG_ACPI_PROC_EVENT 1 +#define CONFIG_LOCALVERSION "" +#define CONFIG_USB_DEVICEFS 1 +#define CONFIG_SERIO_PCIPS2 1 +#define CONFIG_ZLIB_INFLATE 1 +#define CONFIG_XENO_OPT_NATIVE_EVENT 1 +#define CONFIG_INIT_ENV_ARG_LIMIT 32 +#define CONFIG_IPIPE_DELAYED_ATOMICSW 1 +#define CONFIG_CRYPTO_HASH2 1 +#define CONFIG_IO_DELAY_TYPE_NONE 3 +#define CONFIG_ARCH_PHYS_ADDR_T_64BIT 1 +#define CONFIG_XENO_OPT_NATIVE_QUEUE 1 +#define CONFIG_ACPI_BATTERY 1 +#define CONFIG_PCSPKR_PLATFORM 1 +#define CONFIG_HZ_250 1 +#define CONFIG_XENO_HW_SMI_DETECT 1 +#define CONFIG_SYSFS 1 +#define CONFIG_SND_RAWMIDI_SEQ 1 +#define CONFIG_X86_CMPXCHG64 1 +#define CONFIG_HAVE_KPROBES 1 +#define CONFIG_IPIPE_COMPAT 1 +#define CONFIG_IOSCHED_NOOP 1 +#define CONFIG_STOP_MACHINE 1 +#define CONFIG_HIGH_RES_TIMERS 1 +#define CONFIG_HAVE_DYNAMIC_FTRACE 1 +#define CONFIG_ACPI_DOCK 1 +#define CONFIG_PM 1 +#define CONFIG_GENERIC_CLOCKEVENTS 1 +#define CONFIG_SERIAL_8250_EXTENDED 1 +#define CONFIG_NO_HZ 1 +#define CONFIG_LOCKDEP_SUPPORT 1 +#define CONFIG_MSDOS_PARTITION 1 +#define CONFIG_USB_UHCI_HCD 1 +#define CONFIG_XENO_OPT_POSIX_PERIOD 0 +#define CONFIG_XENO_OPT_PERVASIVE 1 +#define CONFIG_HAVE_ARCH_TRACEHOOK 1 +#define CONFIG_X86_PM_TIMER 1 +#define CONFIG_HZ 250 +#define CONFIG_GENERIC_FIND_FIRST_BIT 1 +#define CONFIG_HUGETLBFS 1 +#define CONFIG_XENO_OPT_NATIVE_MUTEX 1 +#define CONFIG_TRACING_SUPPORT 1 +#define CONFIG_ACPI_BLACKLIST_YEAR 0 +#define CONFIG_XENO_GENERIC_STACKPOOL 1 +#define CONFIG_IOMMU_HELPER 1 +#define CONFIG_XENO_OPT_SYS_STACKPOOLSZ 128 +#define CONFIG_SSB_POSSIBLE 1 +#define CONFIG_X86_MINIMUM_CPU_FAMILY 64 +#define CONFIG_USB_LIBUSUAL 1 +#define CONFIG_VT 1 +#define CONFIG_TICK_ONESHOT 1 +#define CONFIG_TMPFS_POSIX_ACL 1 +#define CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE 1 +#define CONFIG_GENERIC_PENDING_IRQ 1 +#define CONFIG_RWSEM_GENERIC_SPINLOCK 1 +#define CONFIG_BRANCH_PROFILE_NONE 1 +#define CONFIG_SPARSEMEM 1 +#define CONFIG_BASE_FULL 1 +#define CONFIG_PREVENT_FIRMWARE_BUILD 1 +#define CONFIG_SYSFS_DEPRECATED_V2 1 +#define CONFIG_CPU_SUP_CENTAUR 1 +#define CONFIG_GENERIC_CMOS_UPDATE 1 +#define CONFIG_GENERIC_CALIBRATE_DELAY 1 +#define CONFIG_ARCH_HAS_CPU_IDLE_WAIT 1 +#define CONFIG_NFS_ACL_SUPPORT 1 +#define CONFIG_HAS_IOMEM 1 +#define CONFIG_XENO_OPT_PIPELINE_HEAD 1 +#define CONFIG_FW_LOADER 1 +#define CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE 0 +#define CONFIG_IO_DELAY_0X80 1 +#define CONFIG_FTRACE 1 +#define CONFIG_MODULE_FORCE_UNLOAD 1 +#define CONFIG_RTC_INTF_DEV 1 +#define CONFIG_PACKET 1 +#define CONFIG_BSD_PROCESS_ACCT_V3 1 +#define CONFIG_CONSTRUCTORS 1 +#define CONFIG_DEBUG_BUGVERBOSE 1 +#define CONFIG_FS_POSIX_ACL 1 +#define CONFIG_SERIAL_CORE_CONSOLE 1 +#define CONFIG_X86_WP_WORKS_OK 1 +#define CONFIG_GENERIC_HARDIRQS 1 +#define CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC 1 +#define CONFIG_HPET_MMAP 1 +#define CONFIG_CPU_IDLE_GOV_LADDER 1 +#define CONFIG_CRAMFS 1 +#define CONFIG_MOUSE_PS2_LOGIPS2PP 1 +#define CONFIG_BOUNCE 1 +#define CONFIG_SERIO_CT82C710 1 +#define CONFIG_MOUSE_PS2 1 +#define CONFIG_SCSI_PROC_FS 1 +#define CONFIG_RTC_CLASS 1 +#define CONFIG_CRYPTO_PCOMP 1 +#define CONFIG_HAVE_MMIOTRACE_SUPPORT 1 +#define CONFIG_INOTIFY_USER 1 +#define CONFIG_XENO_SKIN_POSIX 1 +#define CONFIG_IPIPE_DEBUG_CONTEXT 1 +#define CONFIG_X86_HT 1 +#define CONFIG_INPUT_PCSPKR 1 +#define CONFIG_TASKSTATS 1 +#define CONFIG_TRACE_IRQFLAGS_SUPPORT 1 +#define CONFIG_COMPAT 1 +#define CONFIG_POSIX_MQUEUE 1 +#define CONFIG_SND_TIMER 1 +#define CONFIG_RTC_DRV_CMOS 1 +#define CONFIG_X86 1 +#define CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT 0 +#define CONFIG_SELECT_MEMORY_MODEL 1 +#define CONFIG_SYSFS_DEPRECATED 1 +#define CONFIG_GENERIC_BUG_RELATIVE_POINTERS 1 +#define CONFIG_SPARSEMEM_MANUAL 1 +#define CONFIG_SND_MPU401 1 +#define CONFIG_SERIO_RAW 1 +#define CONFIG_XENO_OPT_NATIVE_BUFFER 1 +#define CONFIG_EDD 1 +#define CONFIG_NFS_V3 1 +#define CONFIG_NFS_V4 1 +#define CONFIG_JBD 1 +#define CONFIG_USB_ARCH_HAS_HCD 1 +#define CONFIG_DEFAULT_TCP_CONG "cubic" +#define CONFIG_BSD_PROCESS_ACCT 1 +#define CONFIG_ZONE_DMA32 1 +#define CONFIG_GENERIC_IOMAP 1 +#define CONFIG_FAT_FS 1 +#define CONFIG_CRYPTO_BLKCIPHER 1 +#define CONFIG_XENO_OPT_NATIVE_PERIOD 0 +#define CONFIG_FTRACE_NMI_ENTER 1 +#define CONFIG_HID 1 +#define CONFIG_NLATTR 1 +#define CONFIG_X86_MCE_INTEL 1 +#define CONFIG_FAT_DEFAULT_CODEPAGE 437 +#define CONFIG_ATA 1 +#define CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT 1 +#define CONFIG_HAVE_KERNEL_LZMA 1 +#define CONFIG_PRINTK 1 +#define CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING 1 +#define CONFIG_POWER_SUPPLY 1 +#define CONFIG_AIO 1 +#define CONFIG_NLS_ISO8859_15 1 +#define CONFIG_MOUSE_PS2_SYNAPTICS 1 +#define CONFIG_DMI 1 +#define CONFIG_HAVE_FUNCTION_GRAPH_TRACER 1 +#define CONFIG_SUNRPC 1 +#define CONFIG_FS_MBCACHE 1 +#define CONFIG_SERIAL_8250_NR_UARTS 32 +#define CONFIG_DETECT_HUNG_TASK 1 +#define CONFIG_HAVE_MEMORY_PRESENT 1 +#define CONFIG_XENO_TESTING_MODULE_MODULE 1 +#define CONFIG_PCI 1 +#define CONFIG_IKCONFIG_PROC 1 +#define CONFIG_HAVE_KERNEL_GZIP 1 +#define CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG 1 +#define CONFIG_X86_64_SMP 1 +#define CONFIG_BUG 1 +#define CONFIG_NFS_FS 1 +#define CONFIG_MII 1 +#define CONFIG_PROCESSOR_SELECT 1 +#define CONFIG_NETWORK_FILESYSTEMS 1 +#define CONFIG_FIX_EARLYCON_MEM 1 +#define CONFIG_CRYPTO 1 +#define CONFIG_IPIPE_DOMAINS 4 +#define CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB 1 +#define CONFIG_SYSCTL 1 +#define CONFIG_DEBUG_PREEMPT 1 +#define CONFIG_MISC_FILESYSTEMS 1 +#define CONFIG_HAVE_OPROFILE 1 +#define CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK 1 +#define CONFIG_HAVE_PERF_EVENTS 1 +#define CONFIG_STACKTRACE 1 +#define CONFIG_SND 1 +#define CONFIG_HPET_EMULATE_RTC 1 +#define CONFIG_NETDEV_1000 1 +#define CONFIG_IP_PNP 1 +#define CONFIG_ZONE_DMA_FLAG 1 +#define CONFIG_NET 1 +#define CONFIG_MODULE_UNLOAD 1 +#define CONFIG_STANDALONE 1 +#define CONFIG_EVENT_TRACING 1 +#define CONFIG_NETDEVICES 1 +#define CONFIG_USB 1 +#define CONFIG_FRAME_POINTER 1 +#define CONFIG_SND_HDA_INTEL 1 +#define CONFIG_SND_DMA_SGBUF 1 +#define CONFIG_NLS 1 +#define CONFIG_CLOCKSOURCE_WATCHDOG 1 +#define CONFIG_CRYPTO_ALGAPI 1 +#define CONFIG_SERIAL_8250 1 +#define CONFIG_PNP 1 +#define CONFIG_SND_SEQUENCER 1 +#define CONFIG_MMU 1 +#define CONFIG_PCI_LEGACY 1 +#define CONFIG_SMP 1 +#define CONFIG_DEBUG_MUTEXES 1 +#define CONFIG_DEBUG_INFO 1 +#define CONFIG_EXT3_FS_SECURITY 1 +#define CONFIG_BASE_SMALL 0 +#define CONFIG_XENO_SKIN_NATIVE 1 +#define CONFIG_AUDIT_ARCH 1 +#define CONFIG_SND_AC97_CODEC 1 +#define CONFIG_SND_ENS1371 1 +#define CONFIG_BINFMT_ELF 1 +#define CONFIG_PCI_DIRECT 1 +#define CONFIG_XENO_OPT_NATIVE_SEM 1 +#define CONFIG_BINARY_PRINTF 1 +#define CONFIG_XENO_OPT_PIPE_NRDEV 32 +#define CONFIG_SND_SUPPORT_OLD_API 1 +#define CONFIG_SERIO_I8042 1 +#define CONFIG_XENO_DRIVERS_TIMERBENCH 1 +#define CONFIG_XENO_OPT_REGISTRY_NRSLOTS 512 +#define CONFIG_IPIPE_DEBUG_INTERNAL 1 +#define CONFIG_SND_PCM_OSS_PLUGINS 1 +#define CONFIG_PCIEAER 1 +#define CONFIG_HAVE_MLOCK 1 +#define CONFIG_X86_MCE 1 +#define CONFIG_XENO_OPT_SEM_HEAPSZ 12 +#define CONFIG_BITREVERSE 1 +#define CONFIG_LOG_BUF_SHIFT 16 +#define CONFIG_IA32_EMULATION 1 +#define CONFIG_ARCH_POPULATES_NODE_MAP 1 +#define CONFIG_NFS_V3_ACL 1 +#define CONFIG_DUMMY_CONSOLE 1 diff --git a/include/linux/bounds.h b/include/linux/bounds.h new file mode 100644 index 0000000..2ca0fee --- /dev/null +++ b/include/linux/bounds.h @@ -0,0 +1,13 @@ +#ifndef __LINUX_BOUNDS_H__ +#define __LINUX_BOUNDS_H__ +/* + * DO NOT MODIFY. + * + * This file was generated by Kbuild + * + */ + +#define NR_PAGEFLAGS 24 /* __NR_PAGEFLAGS # */ +#define MAX_NR_ZONES 4 /* __MAX_NR_ZONES # */ + +#endif diff --git a/include/linux/compile.h b/include/linux/compile.h new file mode 100644 index 0000000..70f55aa --- /dev/null +++ b/include/linux/compile.h @@ -0,0 +1,9 @@ +/* This file is auto generated, version 92 */ +/* SMP PREEMPT */ +#define UTS_MACHINE "x86_64" +#define UTS_VERSION "#92 SMP PREEMPT Thu Feb 4 10:29:12 CET 2010" +#define LINUX_COMPILE_TIME "10:29:12" +#define LINUX_COMPILE_BY "rpm" +#define LINUX_COMPILE_HOST "redshift" +#define LINUX_COMPILE_DOMAIN "xenomai.org" +#define LINUX_COMPILER "gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9) " diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index d5b3876..010aa8b 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -207,24 +207,28 @@ extern void irq_enter(void); */ extern void irq_exit(void); -#define nmi_enter() \ - do { \ - ftrace_nmi_enter(); \ - BUG_ON(in_nmi()); \ - add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ - lockdep_off(); \ - rcu_nmi_enter(); \ - trace_hardirq_enter(); \ +#define nmi_enter() \ + do { \ + if (likely(!ipipe_test_foreign_stack())) { \ + ftrace_nmi_enter(); \ + BUG_ON(in_nmi()); \ + add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ + lockdep_off(); \ + rcu_nmi_enter(); \ + trace_hardirq_enter(); \ + } \ } while (0) -#define nmi_exit() \ - do { \ - trace_hardirq_exit(); \ - rcu_nmi_exit(); \ - lockdep_on(); \ - BUG_ON(!in_nmi()); \ - sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ - ftrace_nmi_exit(); \ +#define nmi_exit() \ + do { \ + if (likely(!ipipe_test_foreign_stack())) { \ + trace_hardirq_exit(); \ + rcu_nmi_exit(); \ + lockdep_on(); \ + BUG_ON(!in_nmi()); \ + sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET); \ + ftrace_nmi_exit(); \ + } \ } while (0) #endif /* LINUX_HARDIRQ_H */ diff --git a/include/linux/ipipe.h b/include/linux/ipipe.h new file mode 100644 index 0000000..59da296 --- /dev/null +++ b/include/linux/ipipe.h @@ -0,0 +1,684 @@ +/* -*- linux-c -*- + * include/linux/ipipe.h + * + * Copyright (C) 2002-2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_H +#define __LINUX_IPIPE_H + +#include <linux/spinlock.h> +#include <linux/cache.h> +#include <linux/percpu.h> +#include <linux/mutex.h> +#include <linux/linkage.h> +#include <linux/ipipe_base.h> +#include <asm/ipipe.h> + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + +#include <linux/cpumask.h> +#include <asm/system.h> + +static inline int ipipe_disable_context_check(int cpu) +{ + return xchg(&per_cpu(ipipe_percpu_context_check, cpu), 0); +} + +static inline void ipipe_restore_context_check(int cpu, int old_state) +{ + per_cpu(ipipe_percpu_context_check, cpu) = old_state; +} + +static inline void ipipe_context_check_off(void) +{ + int cpu; + for_each_online_cpu(cpu) + per_cpu(ipipe_percpu_context_check, cpu) = 0; +} + +#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +static inline int ipipe_disable_context_check(int cpu) +{ + return 0; +} + +static inline void ipipe_restore_context_check(int cpu, int old_state) { } + +static inline void ipipe_context_check_off(void) { } + +#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +#ifdef CONFIG_IPIPE + +#define IPIPE_VERSION_STRING IPIPE_ARCH_STRING +#define IPIPE_RELEASE_NUMBER ((IPIPE_MAJOR_NUMBER << 16) | \ + (IPIPE_MINOR_NUMBER << 8) | \ + (IPIPE_PATCH_NUMBER)) + +#ifndef BROKEN_BUILTIN_RETURN_ADDRESS +#define __BUILTIN_RETURN_ADDRESS0 ((unsigned long)__builtin_return_address(0)) +#define __BUILTIN_RETURN_ADDRESS1 ((unsigned long)__builtin_return_address(1)) +#endif /* !BUILTIN_RETURN_ADDRESS */ + +#define IPIPE_ROOT_PRIO 100 +#define IPIPE_ROOT_ID 0 +#define IPIPE_ROOT_NPTDKEYS 4 /* Must be <= BITS_PER_LONG */ + +#define IPIPE_RESET_TIMER 0x1 +#define IPIPE_GRAB_TIMER 0x2 + +/* Global domain flags */ +#define IPIPE_SPRINTK_FLAG 0 /* Synchronous printk() allowed */ +#define IPIPE_AHEAD_FLAG 1 /* Domain always heads the pipeline */ + +/* Interrupt control bits */ +#define IPIPE_HANDLE_FLAG 0 +#define IPIPE_PASS_FLAG 1 +#define IPIPE_ENABLE_FLAG 2 +#define IPIPE_DYNAMIC_FLAG IPIPE_HANDLE_FLAG +#define IPIPE_STICKY_FLAG 3 +#define IPIPE_SYSTEM_FLAG 4 +#define IPIPE_LOCK_FLAG 5 +#define IPIPE_WIRED_FLAG 6 +#define IPIPE_EXCLUSIVE_FLAG 7 + +#define IPIPE_HANDLE_MASK (1 << IPIPE_HANDLE_FLAG) +#define IPIPE_PASS_MASK (1 << IPIPE_PASS_FLAG) +#define IPIPE_ENABLE_MASK (1 << IPIPE_ENABLE_FLAG) +#define IPIPE_DYNAMIC_MASK IPIPE_HANDLE_MASK +#define IPIPE_STICKY_MASK (1 << IPIPE_STICKY_FLAG) +#define IPIPE_SYSTEM_MASK (1 << IPIPE_SYSTEM_FLAG) +#define IPIPE_LOCK_MASK (1 << IPIPE_LOCK_FLAG) +#define IPIPE_WIRED_MASK (1 << IPIPE_WIRED_FLAG) +#define IPIPE_EXCLUSIVE_MASK (1 << IPIPE_EXCLUSIVE_FLAG) + +#define IPIPE_DEFAULT_MASK (IPIPE_HANDLE_MASK|IPIPE_PASS_MASK) +#define IPIPE_STDROOT_MASK (IPIPE_HANDLE_MASK|IPIPE_PASS_MASK|IPIPE_SYSTEM_MASK) + +#define IPIPE_EVENT_SELF 0x80000000 + +#define IPIPE_NR_CPUS NR_CPUS + +/* This accessor assumes hw IRQs are off on SMP; allows assignment. */ +#define __ipipe_current_domain __ipipe_get_cpu_var(ipipe_percpu_domain) +/* This read-only accessor makes sure that hw IRQs are off on SMP. */ +#define ipipe_current_domain \ + ({ \ + struct ipipe_domain *__ipd__; \ + unsigned long __flags__; \ + local_irq_save_hw_smp(__flags__); \ + __ipd__ = __ipipe_current_domain; \ + local_irq_restore_hw_smp(__flags__); \ + __ipd__; \ + }) + +#define ipipe_virtual_irq_p(irq) ((irq) >= IPIPE_VIRQ_BASE && \ + (irq) < IPIPE_NR_IRQS) + +#define IPIPE_SAME_HANDLER ((ipipe_irq_handler_t)(-1)) + +struct irq_desc; + +typedef void (*ipipe_irq_ackfn_t)(unsigned irq, struct irq_desc *desc); + +typedef int (*ipipe_event_handler_t)(unsigned event, + struct ipipe_domain *from, + void *data); +struct ipipe_domain { + + int slot; /* Slot number in percpu domain data array. */ + struct list_head p_link; /* Link in pipeline */ + ipipe_event_handler_t evhand[IPIPE_NR_EVENTS]; /* Event handlers. */ + unsigned long long evself; /* Self-monitored event bits. */ + + struct ipipe_irqdesc { + unsigned long control; + ipipe_irq_ackfn_t acknowledge; + ipipe_irq_handler_t handler; + void *cookie; + } ____cacheline_aligned irqs[IPIPE_NR_IRQS]; + + int priority; + void *pdd; + unsigned long flags; + unsigned domid; + const char *name; + struct mutex mutex; +}; + +#define IPIPE_HEAD_PRIORITY (-1) /* For domains always heading the pipeline */ + +struct ipipe_domain_attr { + + unsigned domid; /* Domain identifier -- Magic value set by caller */ + const char *name; /* Domain name -- Warning: won't be dup'ed! */ + int priority; /* Priority in interrupt pipeline */ + void (*entry) (void); /* Domain entry point */ + void *pdd; /* Per-domain (opaque) data pointer */ +}; + +#define __ipipe_irq_cookie(ipd, irq) (ipd)->irqs[irq].cookie +#define __ipipe_irq_handler(ipd, irq) (ipd)->irqs[irq].handler +#define __ipipe_cpudata_irq_hits(ipd, cpu, irq) ipipe_percpudom(ipd, irqall, cpu)[irq] + +extern unsigned __ipipe_printk_virq; + +extern unsigned long __ipipe_virtual_irq_map; + +extern struct list_head __ipipe_pipeline; + +extern int __ipipe_event_monitors[]; + +/* Private interface */ + +void ipipe_init(void); + +#ifdef CONFIG_PROC_FS +void ipipe_init_proc(void); + +#ifdef CONFIG_IPIPE_TRACE +void __ipipe_init_tracer(void); +#else /* !CONFIG_IPIPE_TRACE */ +#define __ipipe_init_tracer() do { } while(0) +#endif /* CONFIG_IPIPE_TRACE */ + +#else /* !CONFIG_PROC_FS */ +#define ipipe_init_proc() do { } while(0) +#endif /* CONFIG_PROC_FS */ + +void __ipipe_init_stage(struct ipipe_domain *ipd); + +void __ipipe_cleanup_domain(struct ipipe_domain *ipd); + +void __ipipe_add_domain_proc(struct ipipe_domain *ipd); + +void __ipipe_remove_domain_proc(struct ipipe_domain *ipd); + +void __ipipe_flush_printk(unsigned irq, void *cookie); + +void __ipipe_walk_pipeline(struct list_head *pos); + +void __ipipe_pend_irq(unsigned irq, struct list_head *head); + +int __ipipe_dispatch_event(unsigned event, void *data); + +void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq); + +void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq); + +void __ipipe_sync_stage(int dovirt); + +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq); + +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq); + +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq); + +void __ipipe_pin_range_globally(unsigned long start, unsigned long end); + +/* Must be called hw IRQs off. */ +static inline void ipipe_irq_lock(unsigned irq) +{ + __ipipe_lock_irq(__ipipe_current_domain, ipipe_processor_id(), irq); +} + +/* Must be called hw IRQs off. */ +static inline void ipipe_irq_unlock(unsigned irq) +{ + __ipipe_unlock_irq(__ipipe_current_domain, irq); +} + +#ifndef __ipipe_sync_pipeline +#define __ipipe_sync_pipeline(dovirt) __ipipe_sync_stage(dovirt) +#endif + +#ifndef __ipipe_run_irqtail +#define __ipipe_run_irqtail() do { } while(0) +#endif + +#define __ipipe_pipeline_head_p(ipd) (&(ipd)->p_link == __ipipe_pipeline.next) + +#define __ipipe_ipending_p(p) ((p)->irqpend_himap != 0) + +/* + * Keep the following as a macro, so that client code could check for + * the support of the invariant pipeline head optimization. + */ +#define __ipipe_pipeline_head() \ + list_entry(__ipipe_pipeline.next, struct ipipe_domain, p_link) + +#define local_irq_enable_hw_cond() local_irq_enable_hw() +#define local_irq_disable_hw_cond() local_irq_disable_hw() +#define local_irq_save_hw_cond(flags) local_irq_save_hw(flags) +#define local_irq_restore_hw_cond(flags) local_irq_restore_hw(flags) + +#ifdef CONFIG_SMP +cpumask_t __ipipe_set_irq_affinity(unsigned irq, cpumask_t cpumask); +int __ipipe_send_ipi(unsigned ipi, cpumask_t cpumask); +#define local_irq_save_hw_smp(flags) local_irq_save_hw(flags) +#define local_irq_restore_hw_smp(flags) local_irq_restore_hw(flags) +#else /* !CONFIG_SMP */ +#define local_irq_save_hw_smp(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_smp(flags) do { } while(0) +#endif /* CONFIG_SMP */ + +#define local_irq_save_full(vflags, rflags) \ + do { \ + local_irq_save(vflags); \ + local_irq_save_hw(rflags); \ + } while(0) + +#define local_irq_restore_full(vflags, rflags) \ + do { \ + local_irq_restore_hw(rflags); \ + local_irq_restore(vflags); \ + } while(0) + +static inline void __local_irq_restore_nosync(unsigned long x) +{ + struct ipipe_percpu_domain_data *p = ipipe_root_cpudom_ptr(); + + if (raw_irqs_disabled_flags(x)) { + set_bit(IPIPE_STALL_FLAG, &p->status); + trace_hardirqs_off(); + } else { + trace_hardirqs_on(); + clear_bit(IPIPE_STALL_FLAG, &p->status); + } +} + +static inline void local_irq_restore_nosync(unsigned long x) +{ + unsigned long flags; + local_irq_save_hw_smp(flags); + __local_irq_restore_nosync(x); + local_irq_restore_hw_smp(flags); +} + +#define __ipipe_root_domain_p (__ipipe_current_domain == ipipe_root_domain) +#define ipipe_root_domain_p (ipipe_current_domain == ipipe_root_domain) + +static inline int __ipipe_event_monitored_p(int ev) +{ + if (__ipipe_event_monitors[ev] > 0) + return 1; + + return (ipipe_current_domain->evself & (1LL << ev)) != 0; +} + +#define ipipe_sigwake_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SIGWAKE)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SIGWAKE, p); \ +} while(0) + +#define ipipe_exit_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_EXIT)) \ + __ipipe_dispatch_event(IPIPE_EVENT_EXIT, p); \ +} while(0) + +#define ipipe_setsched_notify(p) \ +do { \ + if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SETSCHED)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SETSCHED, p); \ +} while(0) + +#define ipipe_schedule_notify(prev, next) \ +do { \ + if ((((prev)->flags|(next)->flags) & PF_EVNOTIFY) && \ + __ipipe_event_monitored_p(IPIPE_EVENT_SCHEDULE)) \ + __ipipe_dispatch_event(IPIPE_EVENT_SCHEDULE,next); \ +} while(0) + +#define ipipe_trap_notify(ex, regs) \ +({ \ + unsigned long __flags__; \ + int __ret__ = 0; \ + local_irq_save_hw_smp(__flags__); \ + if ((test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)) || \ + ((current)->flags & PF_EVNOTIFY)) && \ + __ipipe_event_monitored_p(ex)) { \ + local_irq_restore_hw_smp(__flags__); \ + __ret__ = __ipipe_dispatch_event(ex, regs); \ + } else \ + local_irq_restore_hw_smp(__flags__); \ + __ret__; \ +}) + +static inline void ipipe_init_notify(struct task_struct *p) +{ + if (__ipipe_event_monitored_p(IPIPE_EVENT_INIT)) + __ipipe_dispatch_event(IPIPE_EVENT_INIT, p); +} + +struct mm_struct; + +static inline void ipipe_cleanup_notify(struct mm_struct *mm) +{ + if (__ipipe_event_monitored_p(IPIPE_EVENT_CLEANUP)) + __ipipe_dispatch_event(IPIPE_EVENT_CLEANUP, mm); +} + +/* Public interface */ + +int ipipe_register_domain(struct ipipe_domain *ipd, + struct ipipe_domain_attr *attr); + +int ipipe_unregister_domain(struct ipipe_domain *ipd); + +void ipipe_suspend_domain(void); + +int ipipe_virtualize_irq(struct ipipe_domain *ipd, + unsigned irq, + ipipe_irq_handler_t handler, + void *cookie, + ipipe_irq_ackfn_t acknowledge, + unsigned modemask); + +int ipipe_control_irq(unsigned irq, + unsigned clrmask, + unsigned setmask); + +unsigned ipipe_alloc_virq(void); + +int ipipe_free_virq(unsigned virq); + +int ipipe_trigger_irq(unsigned irq); + +static inline void __ipipe_propagate_irq(unsigned irq) +{ + struct list_head *next = __ipipe_current_domain->p_link.next; + if (next == &ipipe_root.p_link) { + /* Fast path: root must handle all interrupts. */ + __ipipe_set_irq_pending(&ipipe_root, irq); + return; + } + __ipipe_pend_irq(irq, next); +} + +static inline void __ipipe_schedule_irq(unsigned irq) +{ + __ipipe_pend_irq(irq, &__ipipe_current_domain->p_link); +} + +static inline void __ipipe_schedule_irq_head(unsigned irq) +{ + __ipipe_set_irq_pending(__ipipe_pipeline_head(), irq); +} + +static inline void __ipipe_schedule_irq_root(unsigned irq) +{ + __ipipe_set_irq_pending(&ipipe_root, irq); +} + +static inline void ipipe_propagate_irq(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_propagate_irq(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq_head(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq_head(irq); + local_irq_restore_hw(flags); +} + +static inline void ipipe_schedule_irq_root(unsigned irq) +{ + unsigned long flags; + + local_irq_save_hw(flags); + __ipipe_schedule_irq_root(irq); + local_irq_restore_hw(flags); +} + +void ipipe_stall_pipeline_from(struct ipipe_domain *ipd); + +unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd); + +unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd); + +static inline void ipipe_unstall_pipeline_from(struct ipipe_domain *ipd) +{ + ipipe_test_and_unstall_pipeline_from(ipd); +} + +void ipipe_restore_pipeline_from(struct ipipe_domain *ipd, + unsigned long x); + +static inline unsigned long ipipe_test_pipeline_from(struct ipipe_domain *ipd) +{ + return test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline void ipipe_stall_pipeline_head(void) +{ + local_irq_disable_hw(); + __set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status)); +} + +static inline unsigned long ipipe_test_and_stall_pipeline_head(void) +{ + local_irq_disable_hw(); + return __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status)); +} + +void ipipe_unstall_pipeline_head(void); + +void __ipipe_restore_pipeline_head(unsigned long x); + +static inline void ipipe_restore_pipeline_head(unsigned long x) +{ + /* On some archs, __test_and_set_bit() might return different + * truth value than test_bit(), so we test the exclusive OR of + * both statuses, assuming that the lowest bit is always set in + * the truth value (if this is wrong, the failed optimization will + * be caught in __ipipe_restore_pipeline_head() if + * CONFIG_DEBUG_KERNEL is set). */ + if ((x ^ test_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status))) & 1) + __ipipe_restore_pipeline_head(x); +} + +#define ipipe_unstall_pipeline() \ + ipipe_unstall_pipeline_from(ipipe_current_domain) + +#define ipipe_test_and_unstall_pipeline() \ + ipipe_test_and_unstall_pipeline_from(ipipe_current_domain) + +#define ipipe_test_pipeline() \ + ipipe_test_pipeline_from(ipipe_current_domain) + +#define ipipe_test_and_stall_pipeline() \ + ipipe_test_and_stall_pipeline_from(ipipe_current_domain) + +#define ipipe_stall_pipeline() \ + ipipe_stall_pipeline_from(ipipe_current_domain) + +#define ipipe_restore_pipeline(x) \ + ipipe_restore_pipeline_from(ipipe_current_domain, (x)) + +void ipipe_init_attr(struct ipipe_domain_attr *attr); + +int ipipe_get_sysinfo(struct ipipe_sysinfo *sysinfo); + +unsigned long ipipe_critical_enter(void (*syncfn) (void)); + +void ipipe_critical_exit(unsigned long flags); + +static inline void ipipe_set_printk_sync(struct ipipe_domain *ipd) +{ + set_bit(IPIPE_SPRINTK_FLAG, &ipd->flags); +} + +static inline void ipipe_set_printk_async(struct ipipe_domain *ipd) +{ + clear_bit(IPIPE_SPRINTK_FLAG, &ipd->flags); +} + +static inline void ipipe_set_foreign_stack(struct ipipe_domain *ipd) +{ + /* Must be called hw interrupts off. */ + __set_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline void ipipe_clear_foreign_stack(struct ipipe_domain *ipd) +{ + /* Must be called hw interrupts off. */ + __clear_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status)); +} + +static inline int ipipe_test_foreign_stack(void) +{ + /* Must be called hw interrupts off. */ + return test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)); +} + +#ifndef ipipe_safe_current +#define ipipe_safe_current() \ +({ \ + struct task_struct *p; \ + unsigned long flags; \ + local_irq_save_hw_smp(flags); \ + p = ipipe_test_foreign_stack() ? &init_task : current; \ + local_irq_restore_hw_smp(flags); \ + p; \ +}) +#endif + +ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd, + unsigned event, + ipipe_event_handler_t handler); + +cpumask_t ipipe_set_irq_affinity(unsigned irq, + cpumask_t cpumask); + +int ipipe_send_ipi(unsigned ipi, + cpumask_t cpumask); + +int ipipe_setscheduler_root(struct task_struct *p, + int policy, + int prio); + +int ipipe_reenter_root(struct task_struct *prev, + int policy, + int prio); + +int ipipe_alloc_ptdkey(void); + +int ipipe_free_ptdkey(int key); + +int ipipe_set_ptd(int key, + void *value); + +void *ipipe_get_ptd(int key); + +int ipipe_disable_ondemand_mappings(struct task_struct *tsk); + +static inline void ipipe_nmi_enter(void) +{ + int cpu = ipipe_processor_id(); + + per_cpu(ipipe_nmi_saved_root, cpu) = ipipe_root_cpudom_var(status); + __set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + per_cpu(ipipe_saved_context_check_state, cpu) = + ipipe_disable_context_check(cpu); +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ +} + +static inline void ipipe_nmi_exit(void) +{ + int cpu = ipipe_processor_id(); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + ipipe_restore_context_check + (cpu, per_cpu(ipipe_saved_context_check_state, cpu)); +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ + + if (!test_bit(IPIPE_STALL_FLAG, &per_cpu(ipipe_nmi_saved_root, cpu))) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)); +} + +#else /* !CONFIG_IPIPE */ + +#define ipipe_init() do { } while(0) +#define ipipe_suspend_domain() do { } while(0) +#define ipipe_sigwake_notify(p) do { } while(0) +#define ipipe_setsched_notify(p) do { } while(0) +#define ipipe_init_notify(p) do { } while(0) +#define ipipe_exit_notify(p) do { } while(0) +#define ipipe_cleanup_notify(mm) do { } while(0) +#define ipipe_trap_notify(t,r) 0 +#define ipipe_init_proc() do { } while(0) + +static inline void __ipipe_pin_range_globally(unsigned long start, + unsigned long end) +{ +} + +static inline int ipipe_test_foreign_stack(void) +{ + return 0; +} + +#define local_irq_enable_hw_cond() do { } while(0) +#define local_irq_disable_hw_cond() do { } while(0) +#define local_irq_save_hw_cond(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_cond(flags) do { } while(0) +#define local_irq_save_hw_smp(flags) do { (void)(flags); } while(0) +#define local_irq_restore_hw_smp(flags) do { } while(0) + +#define ipipe_irq_lock(irq) do { } while(0) +#define ipipe_irq_unlock(irq) do { } while(0) + +#define __ipipe_root_domain_p 1 +#define ipipe_root_domain_p 1 +#define ipipe_safe_current current +#define ipipe_processor_id() smp_processor_id() + +#define ipipe_nmi_enter() do { } while (0) +#define ipipe_nmi_exit() do { } while (0) + +#define local_irq_disable_head() local_irq_disable() + +#define local_irq_save_full(vflags, rflags) do { (void)(vflags); local_irq_save(rflags); } while(0) +#define local_irq_restore_full(vflags, rflags) do { (void)(vflags); local_irq_restore(rflags); } while(0) +#define local_irq_restore_nosync(vflags) local_irq_restore(vflags) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_H */ diff --git a/include/linux/ipipe_base.h b/include/linux/ipipe_base.h new file mode 100644 index 0000000..2b0705d --- /dev/null +++ b/include/linux/ipipe_base.h @@ -0,0 +1,133 @@ +/* -*- linux-c -*- + * include/linux/ipipe_base.h + * + * Copyright (C) 2002-2007 Philippe Gerum. + * 2007 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_BASE_H +#define __LINUX_IPIPE_BASE_H + +#ifdef CONFIG_IPIPE + +#include <asm/ipipe_base.h> +#include <asm/bitsperlong.h> + +#define __bpl_up(x) (((x)+(BITS_PER_LONG-1)) & ~(BITS_PER_LONG-1)) +/* Number of virtual IRQs (must be a multiple of BITS_PER_LONG) */ +#define IPIPE_NR_VIRQS BITS_PER_LONG +/* First virtual IRQ # (must be aligned on BITS_PER_LONG) */ +#define IPIPE_VIRQ_BASE __bpl_up(IPIPE_NR_XIRQS) +/* Total number of IRQ slots */ +#define IPIPE_NR_IRQS (IPIPE_VIRQ_BASE+IPIPE_NR_VIRQS) + +#define IPIPE_IRQ_LOMAPSZ (IPIPE_NR_IRQS / BITS_PER_LONG) +#if IPIPE_IRQ_LOMAPSZ > BITS_PER_LONG +/* + * We need a 3-level mapping. This allows us to handle up to 32k IRQ + * vectors on 32bit machines, 256k on 64bit ones. + */ +#define __IPIPE_3LEVEL_IRQMAP 1 +#define IPIPE_IRQ_MDMAPSZ (__bpl_up(IPIPE_IRQ_LOMAPSZ) / BITS_PER_LONG) +#else +/* + * 2-level mapping is enough. This allows us to handle up to 1024 IRQ + * vectors on 32bit machines, 4096 on 64bit ones. + */ +#define __IPIPE_2LEVEL_IRQMAP 1 +#endif + +#define IPIPE_IRQ_DOALL 0 +#define IPIPE_IRQ_DOVIRT 1 + +/* Per-cpu pipeline status */ +#define IPIPE_STALL_FLAG 0 /* Stalls a pipeline stage -- guaranteed at bit #0 */ +#define IPIPE_SYNC_FLAG 1 /* The interrupt syncer is running for the domain */ +#define IPIPE_NOSTACK_FLAG 2 /* Domain currently runs on a foreign stack */ + +#define IPIPE_STALL_MASK (1L << IPIPE_STALL_FLAG) +#define IPIPE_SYNC_MASK (1L << IPIPE_SYNC_FLAG) +#define IPIPE_NOSTACK_MASK (1L << IPIPE_NOSTACK_FLAG) + +typedef void (*ipipe_irq_handler_t)(unsigned int irq, + void *cookie); + +extern struct ipipe_domain ipipe_root; + +#define ipipe_root_domain (&ipipe_root) + +void __ipipe_unstall_root(void); + +void __ipipe_restore_root(unsigned long x); + +#define ipipe_preempt_disable(flags) \ + do { \ + local_irq_save_hw(flags); \ + if (__ipipe_root_domain_p) \ + preempt_disable(); \ + } while (0) + +#define ipipe_preempt_enable(flags) \ + do { \ + if (__ipipe_root_domain_p) { \ + preempt_enable_no_resched(); \ + local_irq_restore_hw(flags); \ + preempt_check_resched(); \ + } else \ + local_irq_restore_hw(flags); \ + } while (0) + +#define ipipe_get_cpu(flags) ({ ipipe_preempt_disable(flags); ipipe_processor_id(); }) +#define ipipe_put_cpu(flags) ipipe_preempt_enable(flags) + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT +void ipipe_check_context(struct ipipe_domain *border_ipd); +#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */ +static inline void ipipe_check_context(struct ipipe_domain *border_ipd) { } +#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */ + +/* Generic features */ + +#ifdef CONFIG_GENERIC_CLOCKEVENTS +#define __IPIPE_FEATURE_REQUEST_TICKDEV 1 +#endif +#define __IPIPE_FEATURE_DELAYED_ATOMICSW 1 +#define __IPIPE_FEATURE_FASTPEND_IRQ 1 +#define __IPIPE_FEATURE_TRACE_EVENT 1 + +#else /* !CONFIG_IPIPE */ + +#define ipipe_preempt_disable(flags) \ + do { \ + preempt_disable(); \ + (void)(flags); \ + } while (0) +#define ipipe_preempt_enable(flags) preempt_enable() + +#define ipipe_get_cpu(flags) ({ (void)(flags); get_cpu(); }) +#define ipipe_put_cpu(flags) \ + do { \ + (void)(flags); \ + put_cpu(); \ + } while (0) + +#define ipipe_check_context(ipd) do { } while(0) + +#endif /* CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_BASE_H */ diff --git a/include/linux/ipipe_lock.h b/include/linux/ipipe_lock.h new file mode 100644 index 0000000..b4d0e4b --- /dev/null +++ b/include/linux/ipipe_lock.h @@ -0,0 +1,197 @@ +/* -*- linux-c -*- + * include/linux/ipipe_lock.h + * + * Copyright (C) 2009 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_LOCK_H +#define __LINUX_IPIPE_LOCK_H + +typedef struct { + arch_spinlock_t arch_lock; +} __ipipe_spinlock_t; + +#define ipipe_spinlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), __ipipe_spinlock_t *) + +#define std_spinlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), spinlock_t *) + +#define ipipe_spinlock(lock) ((__ipipe_spinlock_t *)(lock)) +#define std_spinlock(lock) ((spinlock_t *)(lock)) + +#define PICK_SPINLOCK_IRQSAVE(lock, flags) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + (flags) = __ipipe_spin_lock_irqsave(ipipe_spinlock(lock)); \ + else if (std_spinlock_p(lock)) \ + raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \ + else __bad_lock_type(); \ + } while (0) + +#define PICK_SPINTRYLOCK_IRQSAVE(lock, flags) \ + ({ \ + int __ret__; \ + if (ipipe_spinlock_p(lock)) \ + __ret__ = __ipipe_spin_trylock_irqsave(ipipe_spinlock(lock), &(flags)); \ + else if (std_spinlock_p(lock)) \ + __ret__ = raw_spin_trylock_irqsave(&std_spinlock(lock)->rlock, flags); \ + else __bad_lock_type(); \ + __ret__; \ + }) + +#define PICK_SPINUNLOCK_IRQRESTORE(lock, flags) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + __ipipe_spin_unlock_irqrestore(ipipe_spinlock(lock), flags); \ + else if (std_spinlock_p(lock)) \ + raw_spin_unlock_irqrestore(&std_spinlock(lock)->rlock, flags); \ + } while (0) + +#define PICK_SPINOP(op, lock) \ + do { \ + if (ipipe_spinlock_p(lock)) \ + arch_spin##op(&ipipe_spinlock(lock)->arch_lock); \ + else if (std_spinlock_p(lock)) \ + raw_spin##op(&std_spinlock(lock)->rlock); \ + else __bad_lock_type(); \ + } while (0) + +#define PICK_SPINOP_RET(op, lock, type) \ + ({ \ + type __ret__; \ + if (ipipe_spinlock_p(lock)) \ + __ret__ = arch_spin##op(&ipipe_spinlock(lock)->arch_lock); \ + else if (std_spinlock_p(lock)) \ + __ret__ = raw_spin##op(&std_spinlock(lock)->rlock); \ + else { __ret__ = -1; __bad_lock_type(); } \ + __ret__; \ + }) + +#define arch_spin_lock_init(lock) \ + do { \ + IPIPE_DEFINE_SPINLOCK(__lock__); \ + *((ipipe_spinlock_t *)lock) = __lock__; \ + } while (0) + +#define arch_spin_lock_irq(lock) \ + do { \ + local_irq_disable_hw(); \ + arch_spin_lock(lock); \ + } while (0) + +#define arch_spin_unlock_irq(lock) \ + do { \ + arch_spin_unlock(lock); \ + local_irq_enable_hw(); \ + } while (0) + +typedef struct { + arch_rwlock_t arch_lock; +} __ipipe_rwlock_t; + +#define ipipe_rwlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), __ipipe_rwlock_t *) + +#define std_rwlock_p(lock) \ + __builtin_types_compatible_p(typeof(lock), rwlock_t *) + +#define ipipe_rwlock(lock) ((__ipipe_rwlock_t *)(lock)) +#define std_rwlock(lock) ((rwlock_t *)(lock)) + +#define PICK_RWOP(op, lock) \ + do { \ + if (ipipe_rwlock_p(lock)) \ + arch##op(&ipipe_rwlock(lock)->arch_lock); \ + else if (std_rwlock_p(lock)) \ + _raw##op(std_rwlock(lock)); \ + else __bad_lock_type(); \ + } while (0) + +extern int __bad_lock_type(void); + +#ifdef CONFIG_IPIPE + +#define ipipe_spinlock_t __ipipe_spinlock_t +#define IPIPE_DEFINE_SPINLOCK(x) ipipe_spinlock_t x = IPIPE_SPIN_LOCK_UNLOCKED +#define IPIPE_DECLARE_SPINLOCK(x) extern ipipe_spinlock_t x + +#define IPIPE_SPIN_LOCK_UNLOCKED \ + (__ipipe_spinlock_t) { .arch_lock = __ARCH_SPIN_LOCK_UNLOCKED } + +#define spin_lock_irqsave_cond(lock, flags) \ + spin_lock_irqsave(lock, flags) + +#define spin_unlock_irqrestore_cond(lock, flags) \ + spin_unlock_irqrestore(lock, flags) + +void __ipipe_spin_lock_irq(ipipe_spinlock_t *lock); + +void __ipipe_spin_unlock_irq(ipipe_spinlock_t *lock); + +unsigned long __ipipe_spin_lock_irqsave(ipipe_spinlock_t *lock); + +int __ipipe_spin_trylock_irqsave(ipipe_spinlock_t *lock, + unsigned long *x); + +void __ipipe_spin_unlock_irqrestore(ipipe_spinlock_t *lock, + unsigned long x); + +void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock); + +void __ipipe_spin_unlock_irqcomplete(unsigned long x); + +#define ipipe_rwlock_t __ipipe_rwlock_t +#define IPIPE_DEFINE_RWLOCK(x) ipipe_rwlock_t x = IPIPE_RW_LOCK_UNLOCKED +#define IPIPE_DECLARE_RWLOCK(x) extern ipipe_rwlock_t x + +#define IPIPE_RW_LOCK_UNLOCKED \ + (__ipipe_rwlock_t) { .arch_lock = __ARCH_RW_LOCK_UNLOCKED } + +#else /* !CONFIG_IPIPE */ + +#define ipipe_spinlock_t spinlock_t +#define IPIPE_DEFINE_SPINLOCK(x) DEFINE_SPINLOCK(x) +#define IPIPE_DECLARE_SPINLOCK(x) extern spinlock_t x +#define IPIPE_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED + +#define spin_lock_irqsave_cond(lock, flags) \ + do { \ + (void)(flags); \ + spin_lock(lock); \ + } while(0) + +#define spin_unlock_irqrestore_cond(lock, flags) \ + spin_unlock(lock) + +#define __ipipe_spin_lock_irq(lock) do { } while (0) +#define __ipipe_spin_unlock_irq(lock) do { } while (0) +#define __ipipe_spin_lock_irqsave(lock) 0 +#define __ipipe_spin_trylock_irqsave(lock, x) ({ (void)(x); 1; }) +#define __ipipe_spin_unlock_irqrestore(lock, x) do { (void)(x); } while (0) +#define __ipipe_spin_unlock_irqbegin(lock) do { } while (0) +#define __ipipe_spin_unlock_irqcomplete(x) do { (void)(x); } while (0) + +#define ipipe_rwlock_t rwlock_t +#define IPIPE_DEFINE_RWLOCK(x) DEFINE_RWLOCK(x) +#define IPIPE_DECLARE_RWLOCK(x) extern rwlock_t x +#define IPIPE_RW_LOCK_UNLOCKED RW_LOCK_UNLOCKED + +#endif /* !CONFIG_IPIPE */ + +#endif /* !__LINUX_IPIPE_LOCK_H */ diff --git a/include/linux/ipipe_percpu.h b/include/linux/ipipe_percpu.h new file mode 100644 index 0000000..f6727e3 --- /dev/null +++ b/include/linux/ipipe_percpu.h @@ -0,0 +1,89 @@ +/* -*- linux-c -*- + * include/linux/ipipe_percpu.h + * + * Copyright (C) 2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_PERCPU_H +#define __LINUX_IPIPE_PERCPU_H + +#include <asm/percpu.h> +#include <asm/ptrace.h> + +struct ipipe_domain; + +struct ipipe_percpu_domain_data { + unsigned long status; /* <= Must be first in struct. */ + unsigned long irqpend_himap; +#ifdef __IPIPE_3LEVEL_IRQMAP + unsigned long irqpend_mdmap[IPIPE_IRQ_MDMAPSZ]; +#endif + unsigned long irqpend_lomap[IPIPE_IRQ_LOMAPSZ]; + unsigned long irqheld_map[IPIPE_IRQ_LOMAPSZ]; + unsigned long irqall[IPIPE_NR_IRQS]; + u64 evsync; +}; + +/* + * CAREFUL: all accessors based on __raw_get_cpu_var() you may find in + * this file should be used only while hw interrupts are off, to + * prevent from CPU migration regardless of the running domain. + */ +#ifdef CONFIG_SMP +#define ipipe_percpudom_ptr(ipd, cpu) \ + (&per_cpu(ipipe_percpu_darray, cpu)[(ipd)->slot]) +#define ipipe_cpudom_ptr(ipd) \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[(ipd)->slot]) +#else +DECLARE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]); +#define ipipe_percpudom_ptr(ipd, cpu) \ + (per_cpu(ipipe_percpu_daddr, cpu)[(ipd)->slot]) +#define ipipe_cpudom_ptr(ipd) \ + (__ipipe_get_cpu_var(ipipe_percpu_daddr)[(ipd)->slot]) +#endif +#define ipipe_percpudom(ipd, var, cpu) (ipipe_percpudom_ptr(ipd, cpu)->var) +#define ipipe_cpudom_var(ipd, var) (ipipe_cpudom_ptr(ipd)->var) + +#define IPIPE_ROOT_SLOT 0 +#define IPIPE_HEAD_SLOT (CONFIG_IPIPE_DOMAINS - 1) + +DECLARE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]); + +DECLARE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain); + +DECLARE_PER_CPU(unsigned long, ipipe_nmi_saved_root); + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT +DECLARE_PER_CPU(int, ipipe_percpu_context_check); +DECLARE_PER_CPU(int, ipipe_saved_context_check_state); +#endif + +#define ipipe_root_cpudom_ptr(var) \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT]) + +#define ipipe_root_cpudom_var(var) ipipe_root_cpudom_ptr()->var + +#define ipipe_this_cpudom_var(var) \ + ipipe_cpudom_var(__ipipe_current_domain, var) + +#define ipipe_head_cpudom_ptr() \ + (&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_HEAD_SLOT]) + +#define ipipe_head_cpudom_var(var) ipipe_head_cpudom_ptr()->var + +#endif /* !__LINUX_IPIPE_PERCPU_H */ diff --git a/include/linux/ipipe_tickdev.h b/include/linux/ipipe_tickdev.h new file mode 100644 index 0000000..4a1cb1b --- /dev/null +++ b/include/linux/ipipe_tickdev.h @@ -0,0 +1,58 @@ +/* -*- linux-c -*- + * include/linux/ipipe_tickdev.h + * + * Copyright (C) 2007 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef __LINUX_IPIPE_TICKDEV_H +#define __LINUX_IPIPE_TICKDEV_H + +#if defined(CONFIG_IPIPE) && defined(CONFIG_GENERIC_CLOCKEVENTS) + +#include <linux/clockchips.h> + +struct tick_device; + +struct ipipe_tick_device { + + void (*emul_set_mode)(enum clock_event_mode, + struct clock_event_device *cdev); + int (*emul_set_tick)(unsigned long delta, + struct clock_event_device *cdev); + void (*real_set_mode)(enum clock_event_mode mode, + struct clock_event_device *cdev); + int (*real_set_tick)(unsigned long delta, + struct clock_event_device *cdev); + struct tick_device *slave; + unsigned long real_max_delta_ns; + unsigned long real_mult; + int real_shift; +}; + +int ipipe_request_tickdev(const char *devname, + void (*emumode)(enum clock_event_mode mode, + struct clock_event_device *cdev), + int (*emutick)(unsigned long evt, + struct clock_event_device *cdev), + int cpu, unsigned long *tmfreq); + +void ipipe_release_tickdev(int cpu); + +#endif /* CONFIG_IPIPE && CONFIG_GENERIC_CLOCKEVENTS */ + +#endif /* !__LINUX_IPIPE_TICKDEV_H */ diff --git a/include/linux/ipipe_trace.h b/include/linux/ipipe_trace.h new file mode 100644 index 0000000..627b354 --- /dev/null +++ b/include/linux/ipipe_trace.h @@ -0,0 +1,72 @@ +/* -*- linux-c -*- + * include/linux/ipipe_trace.h + * + * Copyright (C) 2005 Luotao Fu. + * 2005-2007 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#ifndef _LINUX_IPIPE_TRACE_H +#define _LINUX_IPIPE_TRACE_H + +#ifdef CONFIG_IPIPE_TRACE + +#include <linux/types.h> + +void ipipe_trace_begin(unsigned long v); +void ipipe_trace_end(unsigned long v); +void ipipe_trace_freeze(unsigned long v); +void ipipe_trace_special(unsigned char special_id, unsigned long v); +void ipipe_trace_pid(pid_t pid, short prio); +void ipipe_trace_event(unsigned char id, unsigned long delay_tsc); +int ipipe_trace_max_reset(void); +int ipipe_trace_frozen_reset(void); + +#else /* !CONFIG_IPIPE_TRACE */ + +#define ipipe_trace_begin(v) do { (void)(v); } while(0) +#define ipipe_trace_end(v) do { (void)(v); } while(0) +#define ipipe_trace_freeze(v) do { (void)(v); } while(0) +#define ipipe_trace_special(id, v) do { (void)(id); (void)(v); } while(0) +#define ipipe_trace_pid(pid, prio) do { (void)(pid); (void)(prio); } while(0) +#define ipipe_trace_event(id, delay_tsc) do { (void)(id); (void)(delay_tsc); } while(0) +#define ipipe_trace_max_reset() do { } while(0) +#define ipipe_trace_froze_reset() do { } while(0) + +#endif /* !CONFIG_IPIPE_TRACE */ + +#ifdef CONFIG_IPIPE_TRACE_PANIC +void ipipe_trace_panic_freeze(void); +void ipipe_trace_panic_dump(void); +#else +static inline void ipipe_trace_panic_freeze(void) { } +static inline void ipipe_trace_panic_dump(void) { } +#endif + +#ifdef CONFIG_IPIPE_TRACE_IRQSOFF +#define ipipe_trace_irq_entry(irq) ipipe_trace_begin(irq) +#define ipipe_trace_irq_exit(irq) ipipe_trace_end(irq) +#define ipipe_trace_irqsoff() ipipe_trace_begin(0x80000000UL) +#define ipipe_trace_irqson() ipipe_trace_end(0x80000000UL) +#else +#define ipipe_trace_irq_entry(irq) do { (void)(irq);} while(0) +#define ipipe_trace_irq_exit(irq) do { (void)(irq);} while(0) +#define ipipe_trace_irqsoff() do { } while(0) +#define ipipe_trace_irqson() do { } while(0) +#endif + +#endif /* !__LINUX_IPIPE_TRACE_H */ diff --git a/include/linux/irq.h b/include/linux/irq.h index 451481c..3dbfa17 100644 --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -124,6 +124,9 @@ struct irq_chip { void (*end)(unsigned int irq); int (*set_affinity)(unsigned int irq, const struct cpumask *dest); +#ifdef CONFIG_IPIPE + void (*move)(unsigned int irq); +#endif /* CONFIG_IPIPE */ int (*retrigger)(unsigned int irq); int (*set_type)(unsigned int irq, unsigned int flow_type); int (*set_wake)(unsigned int irq, unsigned int on); @@ -173,6 +176,12 @@ struct irq_2_iommu; * @name: flow handler name for /proc/interrupts output */ struct irq_desc { +#ifdef CONFIG_IPIPE + void (*ipipe_ack)(unsigned int irq, + struct irq_desc *desc); + void (*ipipe_end)(unsigned int irq, + struct irq_desc *desc); +#endif /* CONFIG_IPIPE */ unsigned int irq; struct timer_rand_state *timer_rand_state; unsigned int *kstat_irqs; @@ -346,6 +355,10 @@ extern void set_irq_chip_and_handler_name(unsigned int irq, struct irq_chip *chip, irq_flow_handler_t handle, const char *name); +extern irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, + int is_chained); + extern void __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, const char *name); @@ -357,6 +370,7 @@ static inline void __set_irq_handler_unlocked(int irq, struct irq_desc *desc; desc = irq_to_desc(irq); + handler = __fixup_irq_handler(desc, handler, 0); desc->handle_irq = handler; } diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 328bca6..4ed869d 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -14,6 +14,7 @@ #include <linux/compiler.h> #include <linux/bitops.h> #include <linux/log2.h> +#include <linux/ipipe_base.h> #include <linux/typecheck.h> #include <linux/dynamic_debug.h> #include <asm/byteorder.h> @@ -118,9 +119,12 @@ struct user; #ifdef CONFIG_PREEMPT_VOLUNTARY extern int _cond_resched(void); -# define might_resched() _cond_resched() +# define might_resched() do { \ + ipipe_check_context(ipipe_root_domain); \ + _cond_resched(); \ + } while (0) #else -# define might_resched() do { } while (0) +# define might_resched() ipipe_check_context(ipipe_root_domain) #endif #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 9ccf0e2..f500f1e 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -395,7 +395,7 @@ do { \ #endif /* CONFIG_LOCK_STAT */ -#ifdef CONFIG_LOCKDEP +#if defined (CONFIG_LOCKDEP) || defined(CONFIG_IPIPE) /* * On lockdep we dont want the hand-coded irq-enable of diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 2e681d9..130b7d5 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -9,13 +9,20 @@ #include <linux/thread_info.h> #include <linux/linkage.h> #include <linux/list.h> +#include <linux/ipipe_base.h> #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER) extern void add_preempt_count(int val); extern void sub_preempt_count(int val); #else -# define add_preempt_count(val) do { preempt_count() += (val); } while (0) -# define sub_preempt_count(val) do { preempt_count() -= (val); } while (0) +# define add_preempt_count(val) do { \ + ipipe_check_context(ipipe_root_domain); \ + preempt_count() += (val); \ + } while (0) +# define sub_preempt_count(val) do { \ + ipipe_check_context(ipipe_root_domain); \ + preempt_count() -= (val); \ + } while (0) #endif #define inc_preempt_count() add_preempt_count(1) diff --git a/include/linux/resource.h b/include/linux/resource.h index f1e914e..bc3dfa0 100644 --- a/include/linux/resource.h +++ b/include/linux/resource.h @@ -51,12 +51,6 @@ struct rlimit { #define PRIO_USER 2 /* - * Limit the stack by to some sane default: root can always - * increase this limit if needed.. 8MB seems reasonable. - */ -#define _STK_LIM (8*1024*1024) - -/* * GPG2 wants 64kB of mlocked memory, to make sure pass phrases * and other sensitive information are never written to disk. */ diff --git a/include/linux/rwlock.h b/include/linux/rwlock.h index 71e0b00..8119f0c 100644 --- a/include/linux/rwlock.h +++ b/include/linux/rwlock.h @@ -61,8 +61,8 @@ do { \ #define read_trylock(lock) __cond_lock(lock, _raw_read_trylock(lock)) #define write_trylock(lock) __cond_lock(lock, _raw_write_trylock(lock)) -#define write_lock(lock) _raw_write_lock(lock) -#define read_lock(lock) _raw_read_lock(lock) +#define write_lock(lock) PICK_RWOP(_write_lock, lock) +#define read_lock(lock) PICK_RWOP(_read_lock, lock) #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) @@ -96,8 +96,8 @@ do { \ #define read_lock_bh(lock) _raw_read_lock_bh(lock) #define write_lock_irq(lock) _raw_write_lock_irq(lock) #define write_lock_bh(lock) _raw_write_lock_bh(lock) -#define read_unlock(lock) _raw_read_unlock(lock) -#define write_unlock(lock) _raw_write_unlock(lock) +#define read_unlock(lock) PICK_RWOP(_read_unlock, lock) +#define write_unlock(lock) PICK_RWOP(_write_unlock, lock) #define read_unlock_irq(lock) _raw_read_unlock_irq(lock) #define write_unlock_irq(lock) _raw_write_unlock_irq(lock) diff --git a/include/linux/rwlock_api_smp.h b/include/linux/rwlock_api_smp.h index 9c9f049..62c8941 100644 --- a/include/linux/rwlock_api_smp.h +++ b/include/linux/rwlock_api_smp.h @@ -141,7 +141,9 @@ static inline int __raw_write_trylock(rwlock_t *lock) * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) static inline void __raw_read_lock(rwlock_t *lock) { diff --git a/include/linux/sched.h b/include/linux/sched.h index 78efe7c..c2eb9a4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -61,6 +61,7 @@ struct sched_param { #include <linux/errno.h> #include <linux/nodemask.h> #include <linux/mm_types.h> +#include <linux/ipipe.h> #include <asm/system.h> #include <asm/page.h> @@ -192,9 +193,17 @@ print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) #define TASK_DEAD 64 #define TASK_WAKEKILL 128 #define TASK_WAKING 256 +#ifdef CONFIG_IPIPE +#define TASK_ATOMICSWITCH 512 +#define TASK_NOWAKEUP 1024 +#define TASK_STATE_MAX 2048 +#define TASK_STATE_TO_CHAR_STR "RSDTtZXxKWAN" +#else /* !CONFIG_IPIPE */ +#define TASK_ATOMICSWITCH 0 +#define TASK_NOWAKEUP 0 #define TASK_STATE_MAX 512 - #define TASK_STATE_TO_CHAR_STR "RSDTtZXxKW" +#endif /* CONFIG_IPIPE */ extern char ___assert_task_state[1 - 2*!!( sizeof(TASK_STATE_TO_CHAR_STR)-1 != ilog2(TASK_STATE_MAX)+1)]; @@ -305,6 +314,15 @@ extern void trap_init(void); extern void update_process_times(int user); extern void scheduler_tick(void); +#ifdef CONFIG_IPIPE +void update_root_process_times(struct pt_regs *regs); +#else /* !CONFIG_IPIPE */ +static inline void update_root_process_times(struct pt_regs *regs) +{ + update_process_times(user_mode(regs)); +} +#endif /* CONFIG_IPIPE */ + extern void sched_show_task(struct task_struct *p); #ifdef CONFIG_DETECT_SOFTLOCKUP @@ -356,7 +374,7 @@ extern signed long schedule_timeout(signed long timeout); extern signed long schedule_timeout_interruptible(signed long timeout); extern signed long schedule_timeout_killable(signed long timeout); extern signed long schedule_timeout_uninterruptible(signed long timeout); -asmlinkage void schedule(void); +asmlinkage int schedule(void); extern int mutex_spin_on_owner(struct mutex *lock, struct thread_info *owner); struct nsproxy; @@ -486,6 +504,9 @@ extern int get_dumpable(struct mm_struct *mm); #endif /* leave room for more dump flags */ #define MMF_VM_MERGEABLE 16 /* KSM may merge identical pages */ +#ifdef CONFIG_IPIPE +#define MMF_VM_PINNED 31 /* ondemand load up and COW disabled */ +#endif #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK) @@ -1512,6 +1533,9 @@ struct task_struct { #endif atomic_t fs_excl; /* holding fs exclusive resources */ struct rcu_head rcu; +#ifdef CONFIG_IPIPE + void *ptd[IPIPE_ROOT_NPTDKEYS]; +#endif /* * cache last used pipe for splice @@ -1759,6 +1783,11 @@ extern void thread_group_times(struct task_struct *p, cputime_t *ut, cputime_t * #define PF_EXITING 0x00000004 /* getting shut down */ #define PF_EXITPIDONE 0x00000008 /* pi exit done on shut down */ #define PF_VCPU 0x00000010 /* I'm a virtual CPU */ +#ifdef CONFIG_IPIPE +#define PF_EVNOTIFY 0x00000020 /* Notify other domains about internal events */ +#else +#define PF_EVNOTIFY 0 +#endif /* CONFIG_IPIPE */ #define PF_FORKNOEXEC 0x00000040 /* forked but didn't exec */ #define PF_MCE_PROCESS 0x00000080 /* process policy on mce errors */ #define PF_SUPERPRIV 0x00000100 /* used super-user privileges */ diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 8608821..c93ec16 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -88,6 +88,8 @@ # include <linux/spinlock_up.h> #endif +#include <linux/ipipe_lock.h> + #ifdef CONFIG_DEBUG_SPINLOCK extern void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name, struct lock_class_key *key); @@ -270,16 +272,9 @@ static inline raw_spinlock_t *spinlock_check(spinlock_t *lock) return &lock->rlock; } -#define spin_lock_init(_lock) \ -do { \ - spinlock_check(_lock); \ - raw_spin_lock_init(&(_lock)->rlock); \ -} while (0) +#define spin_lock_init(_lock) PICK_SPINOP(_lock_init, _lock) -static inline void spin_lock(spinlock_t *lock) -{ - raw_spin_lock(&lock->rlock); -} +#define spin_lock(lock) PICK_SPINOP(_lock, lock) static inline void spin_lock_bh(spinlock_t *lock) { @@ -288,7 +283,7 @@ static inline void spin_lock_bh(spinlock_t *lock) static inline int spin_trylock(spinlock_t *lock) { - return raw_spin_trylock(&lock->rlock); + return PICK_SPINOP_RET(_trylock, lock, int); } #define spin_lock_nested(lock, subclass) \ @@ -301,40 +296,27 @@ do { \ raw_spin_lock_nest_lock(spinlock_check(lock), nest_lock); \ } while (0) -static inline void spin_lock_irq(spinlock_t *lock) -{ - raw_spin_lock_irq(&lock->rlock); -} +#define spin_lock_irq(lock) PICK_SPINOP(_lock_irq, lock) -#define spin_lock_irqsave(lock, flags) \ -do { \ - raw_spin_lock_irqsave(spinlock_check(lock), flags); \ -} while (0) +#define spin_lock_irqsave(lock, flags) \ + PICK_SPINLOCK_IRQSAVE(lock, flags) #define spin_lock_irqsave_nested(lock, flags, subclass) \ do { \ raw_spin_lock_irqsave_nested(spinlock_check(lock), flags, subclass); \ } while (0) -static inline void spin_unlock(spinlock_t *lock) -{ - raw_spin_unlock(&lock->rlock); -} +#define spin_unlock(lock) PICK_SPINOP(_unlock, lock) static inline void spin_unlock_bh(spinlock_t *lock) { raw_spin_unlock_bh(&lock->rlock); } -static inline void spin_unlock_irq(spinlock_t *lock) -{ - raw_spin_unlock_irq(&lock->rlock); -} +#define spin_unlock_irq(lock) PICK_SPINOP(_unlock_irq, lock) -static inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) -{ - raw_spin_unlock_irqrestore(&lock->rlock, flags); -} +#define spin_unlock_irqrestore(lock, flags) \ + PICK_SPINUNLOCK_IRQRESTORE(lock, flags) static inline int spin_trylock_bh(spinlock_t *lock) { @@ -347,9 +329,7 @@ static inline int spin_trylock_irq(spinlock_t *lock) } #define spin_trylock_irqsave(lock, flags) \ -({ \ - raw_spin_trylock_irqsave(spinlock_check(lock), flags); \ -}) + PICK_SPINTRYLOCK_IRQSAVE(lock, flags) static inline void spin_unlock_wait(spinlock_t *lock) { diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_smp.h index e253ccd..378e01e 100644 --- a/include/linux/spinlock_api_smp.h +++ b/include/linux/spinlock_api_smp.h @@ -99,7 +99,9 @@ static inline int __raw_spin_trylock(raw_spinlock_t *lock) * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock) { @@ -113,7 +115,7 @@ static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock) * do_raw_spin_lock_flags() code, because lockdep assumes * that interrupts are not re-enabled during lock-acquire: */ -#ifdef CONFIG_LOCKDEP +#if defined(CONFIG_LOCKDEP) || defined(CONFIG_IPIPE) LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); #else do_raw_spin_lock_flags(lock, &flags); diff --git a/include/linux/spinlock_up.h b/include/linux/spinlock_up.h index b14f6a9..e400972 100644 --- a/include/linux/spinlock_up.h +++ b/include/linux/spinlock_up.h @@ -49,13 +49,6 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock) /* * Read-write spinlocks. No debug version. */ -#define arch_read_lock(lock) do { (void)(lock); } while (0) -#define arch_write_lock(lock) do { (void)(lock); } while (0) -#define arch_read_trylock(lock) ({ (void)(lock); 1; }) -#define arch_write_trylock(lock) ({ (void)(lock); 1; }) -#define arch_read_unlock(lock) do { (void)(lock); } while (0) -#define arch_write_unlock(lock) do { (void)(lock); } while (0) - #else /* DEBUG_SPINLOCK */ #define arch_spin_is_locked(lock) ((void)(lock), 0) /* for sched.c and kernel_lock.c: */ @@ -65,6 +58,13 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock) # define arch_spin_trylock(lock) ({ (void)(lock); 1; }) #endif /* DEBUG_SPINLOCK */ +#define arch_read_lock(lock) do { (void)(lock); } while (0) +#define arch_write_lock(lock) do { (void)(lock); } while (0) +#define arch_read_trylock(lock) ({ (void)(lock); 1; }) +#define arch_write_trylock(lock) ({ (void)(lock); 1; }) +#define arch_read_unlock(lock) do { (void)(lock); } while (0) +#define arch_write_unlock(lock) do { (void)(lock); } while (0) + #define arch_spin_is_contended(lock) (((void)(lock), 0)) #define arch_read_can_lock(lock) (((void)(lock), 1)) diff --git a/include/linux/utsrelease.h b/include/linux/utsrelease.h new file mode 100644 index 0000000..a7e4640 --- /dev/null +++ b/include/linux/utsrelease.h @@ -0,0 +1 @@ +#define UTS_RELEASE "2.6.32.7" diff --git a/init/Kconfig b/init/Kconfig index d95ca7c..1c6c3bf 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -78,6 +78,7 @@ config INIT_ENV_ARG_LIMIT config LOCALVERSION string "Local version - append to kernel release" + default "-ipipe" help Append an extra string to the end of your kernel version. This will show up when you type uname, for example. diff --git a/init/main.c b/init/main.c index 4cb47a1..e714997 100644 --- a/init/main.c +++ b/init/main.c @@ -530,7 +530,7 @@ asmlinkage void __init start_kernel(void) cgroup_init_early(); - local_irq_disable(); + local_irq_disable_hw(); early_boot_irqs_off(); early_init_irq_lock_class(); @@ -593,6 +593,11 @@ asmlinkage void __init start_kernel(void) softirq_init(); timekeeping_init(); time_init(); + /* + * We need to wait for the interrupt and time subsystems to be + * initialized before enabling the pipeline. + */ + ipipe_init(); profile_init(); if (!irqs_disabled()) printk(KERN_CRIT "start_kernel(): bug: interrupts were " @@ -774,6 +779,7 @@ static void __init do_basic_setup(void) init_tmpfs(); driver_init(); init_irq_proc(); + ipipe_init_proc(); do_ctors(); do_initcalls(); } diff --git a/kernel/Makefile b/kernel/Makefile index 864ff75..864aa99 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -85,6 +85,7 @@ obj-$(CONFIG_TREE_PREEMPT_RCU) += rcutree.o obj-$(CONFIG_TREE_RCU_TRACE) += rcutree_trace.o obj-$(CONFIG_TINY_RCU) += rcutiny.o obj-$(CONFIG_RELAY) += relay.o +obj-$(CONFIG_IPIPE) += ipipe/ obj-$(CONFIG_SYSCTL) += utsname_sysctl.o obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o diff --git a/kernel/exit.c b/kernel/exit.c index 546774a..1c3531a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -967,6 +967,7 @@ NORET_TYPE void do_exit(long code) acct_process(); trace_sched_process_exit(tsk); + ipipe_exit_notify(tsk); exit_sem(tsk); exit_files(tsk); exit_fs(tsk); diff --git a/kernel/fork.c b/kernel/fork.c index f88bd98..545fd6e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -513,6 +513,7 @@ void mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); exit_mmap(mm); + ipipe_cleanup_notify(mm); set_mm_exe_file(mm, NULL); if (!list_empty(&mm->mmlist)) { spin_lock(&mmlist_lock); @@ -923,7 +924,7 @@ static void copy_flags(unsigned long clone_flags, struct task_struct *p) { unsigned long new_flags = p->flags; - new_flags &= ~PF_SUPERPRIV; + new_flags &= ~(PF_SUPERPRIV | PF_EVNOTIFY); new_flags |= PF_FORKNOEXEC; new_flags |= PF_STARTING; p->flags = new_flags; @@ -1300,6 +1301,9 @@ static struct task_struct *copy_process(unsigned long clone_flags, write_unlock_irq(&tasklist_lock); proc_fork_connector(p); cgroup_post_fork(p); +#ifdef CONFIG_IPIPE + memset(p->ptd, 0, sizeof(p->ptd)); +#endif /* CONFIG_IPIPE */ perf_event_fork(p); return p; @@ -1698,11 +1702,14 @@ SYSCALL_DEFINE1(unshare, unsigned long, unshare_flags) } if (new_mm) { + unsigned long flags; mm = current->mm; active_mm = current->active_mm; current->mm = new_mm; + ipipe_mm_switch_protect(flags); current->active_mm = new_mm; activate_mm(active_mm, new_mm); + ipipe_mm_switch_unprotect(flags); new_mm = mm; } diff --git a/kernel/ipipe/Kconfig b/kernel/ipipe/Kconfig new file mode 100644 index 0000000..8ff9d37 --- /dev/null +++ b/kernel/ipipe/Kconfig @@ -0,0 +1,23 @@ +config IPIPE + bool "Interrupt pipeline" + default y + ---help--- + Activate this option if you want the interrupt pipeline to be + compiled in. + +config IPIPE_DOMAINS + int "Max domains" + depends on IPIPE + default 4 + ---help--- + The maximum number of I-pipe domains to run concurrently. + +config IPIPE_DELAYED_ATOMICSW + bool + depends on IPIPE + default n + +config IPIPE_UNMASKED_CONTEXT_SWITCH + bool + depends on IPIPE + default n diff --git a/kernel/ipipe/Kconfig.debug b/kernel/ipipe/Kconfig.debug new file mode 100644 index 0000000..629c894 --- /dev/null +++ b/kernel/ipipe/Kconfig.debug @@ -0,0 +1,97 @@ +config IPIPE_DEBUG + bool "I-pipe debugging" + depends on IPIPE + +config IPIPE_DEBUG_CONTEXT + bool "Check for illicit cross-domain calls" + depends on IPIPE_DEBUG + default y + ---help--- + Enable this feature to arm checkpoints in the kernel that + verify the correct invocation context. On entry of critical + Linux services a warning is issued if the caller is not + running over the root domain. + +config IPIPE_DEBUG_INTERNAL + bool "Enable internal debug checks" + depends on IPIPE_DEBUG + default y + ---help--- + When this feature is enabled, I-pipe will perform internal + consistency checks of its subsystems, e.g. on per-cpu variable + access. + +config IPIPE_TRACE + bool "Latency tracing" + depends on IPIPE_DEBUG + select FRAME_POINTER + select KALLSYMS + select PROC_FS + ---help--- + Activate this option if you want to use per-function tracing of + the kernel. The tracer will collect data via instrumentation + features like the one below or with the help of explicite calls + of ipipe_trace_xxx(). See include/linux/ipipe_trace.h for the + in-kernel tracing API. The collected data and runtime control + is available via /proc/ipipe/trace/*. + +if IPIPE_TRACE + +config IPIPE_TRACE_ENABLE + bool "Enable tracing on boot" + default y + ---help--- + Disable this option if you want to arm the tracer after booting + manually ("echo 1 > /proc/ipipe/tracer/enable"). This can reduce + boot time on slow embedded devices due to the tracer overhead. + +config IPIPE_TRACE_MCOUNT + bool "Instrument function entries" + default y + select FUNCTION_TRACER + select TRACING + select CONTEXT_SWITCH_TRACER + select FTRACE_MCOUNT_RECORD + select DYNAMIC_FTRACE + ---help--- + When enabled, records every kernel function entry in the tracer + log. While this slows down the system noticeably, it provides + the highest level of information about the flow of events. + However, it can be switch off in order to record only explicit + I-pipe trace points. + +config IPIPE_TRACE_IRQSOFF + bool "Trace IRQs-off times" + default y + ---help--- + Activate this option if I-pipe shall trace the longest path + with hard-IRQs switched off. + +config IPIPE_TRACE_SHIFT + int "Depth of trace log (14 => 16Kpoints, 15 => 32Kpoints)" + range 10 18 + default 14 + ---help--- + The number of trace points to hold tracing data for each + trace path, as a power of 2. + +config IPIPE_TRACE_VMALLOC + bool "Use vmalloc'ed trace buffer" + default y if EMBEDDED + ---help--- + Instead of reserving static kernel data, the required buffer + is allocated via vmalloc during boot-up when this option is + enabled. This can help to start systems that are low on memory, + but it slightly degrades overall performance. Try this option + when a traced kernel hangs unexpectedly at boot time. + +config IPIPE_TRACE_PANIC + bool "Enable panic back traces" + default y + ---help--- + Provides services to freeze and dump a back trace on panic + situations. This is used on IPIPE_DEBUG_CONTEXT exceptions + as well as ordinary kernel oopses. You can control the number + of printed back trace points via /proc/ipipe/trace. + +endif diff --git a/kernel/ipipe/Makefile b/kernel/ipipe/Makefile new file mode 100644 index 0000000..6257dfa --- /dev/null +++ b/kernel/ipipe/Makefile @@ -0,0 +1,3 @@ + +obj-$(CONFIG_IPIPE) += core.o +obj-$(CONFIG_IPIPE_TRACE) += tracer.o diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c new file mode 100644 index 0000000..af1f7e2 --- /dev/null +++ b/kernel/ipipe/core.c @@ -0,0 +1,1970 @@ +/* -*- linux-c -*- + * linux/kernel/ipipe/core.c + * + * Copyright (C) 2002-2005 Philippe Gerum. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Architecture-independent I-PIPE core support. + */ + +#include <linux/version.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/kernel.h> +#include <linux/sched.h> +#include <linux/sched.h> +#include <linux/kallsyms.h> +#include <linux/interrupt.h> +#include <linux/bitops.h> +#include <linux/tick.h> +#include <linux/prefetch.h> +#ifdef CONFIG_PROC_FS +#include <linux/proc_fs.h> +#include <linux/seq_file.h> +#endif /* CONFIG_PROC_FS */ +#include <linux/ipipe_trace.h> +#include <linux/ipipe_tickdev.h> +#include <linux/irq.h> + +static int __ipipe_ptd_key_count; + +static unsigned long __ipipe_ptd_key_map; + +static unsigned long __ipipe_domain_slot_map; + +struct ipipe_domain ipipe_root; + +#ifndef CONFIG_SMP +/* + * Create an alias to the unique root status, so that arch-dep code + * may get simple and easy access to this percpu variable. We also + * create an array of pointers to the percpu domain data; this tends + * to produce a better code when reaching non-root domains. We make + * sure that the early boot code would be able to dereference the + * pointer to the root domain data safely by statically initializing + * its value (local_irq*() routines depend on this). + */ +#if __GNUC__ >= 4 +extern unsigned long __ipipe_root_status +__attribute__((alias(__stringify(__raw_get_cpu_var(ipipe_percpu_darray))))); +EXPORT_SYMBOL(__ipipe_root_status); +#else /* __GNUC__ < 4 */ +/* + * Work around a GCC 3.x issue making alias symbols unusable as + * constant initializers. + */ +unsigned long *const __ipipe_root_status_addr = + &__raw_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT].status; +EXPORT_SYMBOL(__ipipe_root_status_addr); +#endif /* __GNUC__ < 4 */ + +DEFINE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]) = +{ [IPIPE_ROOT_SLOT] = (struct ipipe_percpu_domain_data *)&__raw_get_cpu_var(ipipe_percpu_darray) }; +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_daddr); +#endif /* !CONFIG_SMP */ + +DEFINE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]) = +{ [IPIPE_ROOT_SLOT] = { .status = IPIPE_STALL_MASK } }; /* Root domain stalled on each CPU at startup. */ + +DEFINE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain) = { &ipipe_root }; + +DEFINE_PER_CPU(unsigned long, ipipe_nmi_saved_root); /* Copy of root status during NMI */ + +static IPIPE_DEFINE_SPINLOCK(__ipipe_pipelock); + +LIST_HEAD(__ipipe_pipeline); + +unsigned long __ipipe_virtual_irq_map; + +#ifdef CONFIG_PRINTK +unsigned __ipipe_printk_virq; +#endif /* CONFIG_PRINTK */ + +int __ipipe_event_monitors[IPIPE_NR_EVENTS]; + +#ifdef CONFIG_GENERIC_CLOCKEVENTS + +DECLARE_PER_CPU(struct tick_device, tick_cpu_device); + +static DEFINE_PER_CPU(struct ipipe_tick_device, ipipe_tick_cpu_device); + +int ipipe_request_tickdev(const char *devname, + void (*emumode)(enum clock_event_mode mode, + struct clock_event_device *cdev), + int (*emutick)(unsigned long delta, + struct clock_event_device *cdev), + int cpu, unsigned long *tmfreq) +{ + struct ipipe_tick_device *itd; + struct tick_device *slave; + struct clock_event_device *evtdev; + unsigned long long freq; + unsigned long flags; + int status; + + flags = ipipe_critical_enter(NULL); + + itd = &per_cpu(ipipe_tick_cpu_device, cpu); + + if (itd->slave != NULL) { + status = -EBUSY; + goto out; + } + + slave = &per_cpu(tick_cpu_device, cpu); + + if (strcmp(slave->evtdev->name, devname)) { + /* + * No conflict so far with the current tick device, + * check whether the requested device is sane and has + * been blessed by the kernel. + */ + status = __ipipe_check_tickdev(devname) ? + CLOCK_EVT_MODE_UNUSED : CLOCK_EVT_MODE_SHUTDOWN; + goto out; + } + + /* + * Our caller asks for using the same clock event device for + * ticking than we do, let's create a tick emulation device to + * interpose on the set_next_event() method, so that we may + * both manage the device in oneshot mode. Only the tick + * emulation code will actually program the clockchip hardware + * for the next shot, though. + * + * CAUTION: we still have to grab the tick device even when it + * current runs in periodic mode, since the kernel may switch + * to oneshot dynamically (highres/no_hz tick mode). + */ + + evtdev = slave->evtdev; + status = evtdev->mode; + + if (status == CLOCK_EVT_MODE_SHUTDOWN) + goto out; + + itd->slave = slave; + itd->emul_set_mode = emumode; + itd->emul_set_tick = emutick; + itd->real_set_mode = evtdev->set_mode; + itd->real_set_tick = evtdev->set_next_event; + itd->real_max_delta_ns = evtdev->max_delta_ns; + itd->real_mult = evtdev->mult; + itd->real_shift = evtdev->shift; + freq = (1000000000ULL * evtdev->mult) >> evtdev->shift; + *tmfreq = (unsigned long)freq; + evtdev->set_mode = emumode; + evtdev->set_next_event = emutick; + evtdev->max_delta_ns = ULONG_MAX; + evtdev->mult = 1; + evtdev->shift = 0; +out: + ipipe_critical_exit(flags); + + return status; +} + +void ipipe_release_tickdev(int cpu) +{ + struct ipipe_tick_device *itd; + struct tick_device *slave; + struct clock_event_device *evtdev; + unsigned long flags; + + flags = ipipe_critical_enter(NULL); + + itd = &per_cpu(ipipe_tick_cpu_device, cpu); + + if (itd->slave != NULL) { + slave = &per_cpu(tick_cpu_device, cpu); + evtdev = slave->evtdev; + evtdev->set_mode = itd->real_set_mode; + evtdev->set_next_event = itd->real_set_tick; + evtdev->max_delta_ns = itd->real_max_delta_ns; + evtdev->mult = itd->real_mult; + evtdev->shift = itd->real_shift; + itd->slave = NULL; + } + + ipipe_critical_exit(flags); +} + +#endif /* CONFIG_GENERIC_CLOCKEVENTS */ + +/* + * ipipe_init() -- Initialization routine of the IPIPE layer. Called + * by the host kernel early during the boot procedure. + */ +void __init ipipe_init(void) +{ + struct ipipe_domain *ipd = &ipipe_root; + + __ipipe_check_platform(); /* Do platform dependent checks first. */ + + /* + * A lightweight registration code for the root domain. We are + * running on the boot CPU, hw interrupts are off, and + * secondary CPUs are still lost in space. + */ + + /* Reserve percpu data slot #0 for the root domain. */ + ipd->slot = 0; + set_bit(0, &__ipipe_domain_slot_map); + + ipd->name = "Linux"; + ipd->domid = IPIPE_ROOT_ID; + ipd->priority = IPIPE_ROOT_PRIO; + + __ipipe_init_stage(ipd); + + INIT_LIST_HEAD(&ipd->p_link); + list_add_tail(&ipd->p_link, &__ipipe_pipeline); + + __ipipe_init_platform(); + +#ifdef CONFIG_PRINTK + __ipipe_printk_virq = ipipe_alloc_virq(); /* Cannot fail here. */ + ipd->irqs[__ipipe_printk_virq].handler = &__ipipe_flush_printk; + ipd->irqs[__ipipe_printk_virq].cookie = NULL; + ipd->irqs[__ipipe_printk_virq].acknowledge = NULL; + ipd->irqs[__ipipe_printk_virq].control = IPIPE_HANDLE_MASK; +#endif /* CONFIG_PRINTK */ + + __ipipe_enable_pipeline(); + + printk(KERN_INFO "I-pipe %s: pipeline enabled.\n", + IPIPE_VERSION_STRING); +} + +void __ipipe_init_stage(struct ipipe_domain *ipd) +{ + struct ipipe_percpu_domain_data *p; + unsigned long status; + int cpu, n; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + status = p->status; + memset(p, 0, sizeof(*p)); + p->status = status; + } + + for (n = 0; n < IPIPE_NR_IRQS; n++) { + ipd->irqs[n].acknowledge = NULL; + ipd->irqs[n].handler = NULL; + ipd->irqs[n].control = IPIPE_PASS_MASK; /* Pass but don't handle */ + } + + for (n = 0; n < IPIPE_NR_EVENTS; n++) + ipd->evhand[n] = NULL; + + ipd->evself = 0LL; + mutex_init(&ipd->mutex); + + __ipipe_hook_critical_ipi(ipd); +} + +void __ipipe_cleanup_domain(struct ipipe_domain *ipd) +{ + ipipe_unstall_pipeline_from(ipd); + +#ifdef CONFIG_SMP + { + struct ipipe_percpu_domain_data *p; + int cpu; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + while (__ipipe_ipending_p(p)) + cpu_relax(); + } + } +#else + __raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = NULL; +#endif + + clear_bit(ipd->slot, &__ipipe_domain_slot_map); +} + +void __ipipe_unstall_root(void) +{ + struct ipipe_percpu_domain_data *p; + + local_irq_disable_hw(); + +#ifdef CONFIG_IPIPE_DEBUG_INTERNAL + /* This helps catching bad usage from assembly call sites. */ + BUG_ON(!__ipipe_root_domain_p); +#endif + + p = ipipe_root_cpudom_ptr(); + + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (unlikely(__ipipe_ipending_p(p))) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + local_irq_enable_hw(); +} + +void __ipipe_restore_root(unsigned long x) +{ +#ifdef CONFIG_IPIPE_DEBUG_INTERNAL + BUG_ON(!ipipe_root_domain_p); +#endif + + if (x) + __ipipe_stall_root(); + else + __ipipe_unstall_root(); +} + +void ipipe_stall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags; + /* + * We have to prevent against race on updating the status + * variable _and_ CPU migration at the same time, so disable + * hw IRQs here. + */ + local_irq_save_hw(flags); + + __set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (!__ipipe_pipeline_head_p(ipd)) + local_irq_restore_hw(flags); +} + +unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags, x; + + /* See ipipe_stall_pipeline_from() */ + local_irq_save_hw(flags); + + x = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (!__ipipe_pipeline_head_p(ipd)) + local_irq_restore_hw(flags); + + return x; +} + +unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd) +{ + unsigned long flags, x; + struct list_head *pos; + + local_irq_save_hw(flags); + + x = __test_and_clear_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)); + + if (ipd == __ipipe_current_domain) + pos = &ipd->p_link; + else + pos = __ipipe_pipeline.next; + + __ipipe_walk_pipeline(pos); + + if (likely(__ipipe_pipeline_head_p(ipd))) + local_irq_enable_hw(); + else + local_irq_restore_hw(flags); + + return x; +} + +void ipipe_restore_pipeline_from(struct ipipe_domain *ipd, + unsigned long x) +{ + if (x) + ipipe_stall_pipeline_from(ipd); + else + ipipe_unstall_pipeline_from(ipd); +} + +void ipipe_unstall_pipeline_head(void) +{ + struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr(); + struct ipipe_domain *head_domain; + + local_irq_disable_hw(); + + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (unlikely(__ipipe_ipending_p(p))) { + head_domain = __ipipe_pipeline_head(); + if (likely(head_domain == __ipipe_current_domain)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else + __ipipe_walk_pipeline(&head_domain->p_link); + } + + local_irq_enable_hw(); +} + +void __ipipe_restore_pipeline_head(unsigned long x) +{ + struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr(); + struct ipipe_domain *head_domain; + + local_irq_disable_hw(); + + if (x) { +#ifdef CONFIG_DEBUG_KERNEL + static int warned; + if (!warned && test_and_set_bit(IPIPE_STALL_FLAG, &p->status)) { + /* + * Already stalled albeit ipipe_restore_pipeline_head() + * should have detected it? Send a warning once. + */ + warned = 1; + printk(KERN_WARNING + "I-pipe: ipipe_restore_pipeline_head() optimization failed.\n"); + dump_stack(); + } +#else /* !CONFIG_DEBUG_KERNEL */ + set_bit(IPIPE_STALL_FLAG, &p->status); +#endif /* CONFIG_DEBUG_KERNEL */ + } + else { + __clear_bit(IPIPE_STALL_FLAG, &p->status); + if (unlikely(__ipipe_ipending_p(p))) { + head_domain = __ipipe_pipeline_head(); + if (likely(head_domain == __ipipe_current_domain)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else + __ipipe_walk_pipeline(&head_domain->p_link); + } + local_irq_enable_hw(); + } +} + +void __ipipe_spin_lock_irq(ipipe_spinlock_t *lock) +{ + local_irq_disable_hw(); + arch_spin_lock(&lock->arch_lock); + __set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); +} + +void __ipipe_spin_unlock_irq(ipipe_spinlock_t *lock) +{ + arch_spin_unlock(&lock->arch_lock); + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_enable_hw(); +} + +unsigned long __ipipe_spin_lock_irqsave(ipipe_spinlock_t *lock) +{ + unsigned long flags; + int s; + + local_irq_save_hw(flags); + arch_spin_lock(&lock->arch_lock); + s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + + return raw_mangle_irq_bits(s, flags); +} + +int __ipipe_spin_trylock_irqsave(ipipe_spinlock_t *lock, + unsigned long *x) +{ + unsigned long flags; + int s; + + local_irq_save_hw(flags); + if (!arch_spin_trylock(&lock->arch_lock)) { + local_irq_restore_hw(flags); + return 0; + } + s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + *x = raw_mangle_irq_bits(s, flags); + + return 1; +} + +void __ipipe_spin_unlock_irqrestore(ipipe_spinlock_t *lock, + unsigned long x) +{ + arch_spin_unlock(&lock->arch_lock); + if (!raw_demangle_irq_bits(&x)) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_restore_hw(x); +} + +void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock) +{ + arch_spin_unlock(&lock->arch_lock); +} + +void __ipipe_spin_unlock_irqcomplete(unsigned long x) +{ + if (!raw_demangle_irq_bits(&x)) + __clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status)); + local_irq_restore_hw(x); +} + +#ifdef __IPIPE_3LEVEL_IRQMAP + +/* Must be called hw IRQs off. */ +static inline void __ipipe_set_irq_held(struct ipipe_percpu_domain_data *p, + unsigned int irq) +{ + __set_bit(irq, p->irqheld_map); + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd); + int l0b, l1b; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + prefetchw(p); + + if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) { + __set_bit(irq, p->irqpend_lomap); + __set_bit(l1b, p->irqpend_mdmap); + __set_bit(l0b, &p->irqpend_himap); + } else + __set_bit(irq, p->irqheld_map); + + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b, l1b; + + if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + + p = ipipe_percpudom_ptr(ipd, cpu); + if (__test_and_clear_bit(irq, p->irqpend_lomap)) { + __set_bit(irq, p->irqheld_map); + if (p->irqpend_lomap[l1b] == 0) { + __clear_bit(l1b, p->irqpend_mdmap); + if (p->irqpend_mdmap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } + } +} + +/* Must be called hw IRQs off. */ +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned int irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b, l1b, cpu; + + if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + l0b = irq / (BITS_PER_LONG * BITS_PER_LONG); + l1b = irq / BITS_PER_LONG; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + if (test_and_clear_bit(irq, p->irqheld_map)) { + /* We need atomic ops here: */ + set_bit(irq, p->irqpend_lomap); + set_bit(l1b, p->irqpend_mdmap); + set_bit(l0b, &p->irqpend_himap); + } + } +} + +static inline int __ipipe_next_irq(struct ipipe_percpu_domain_data *p, + int dovirt) +{ + unsigned long l0m, l1m, l2m, himask, mdmask; + int l0b, l1b, l2b, vl0b, vl1b; + unsigned int irq; + + if (dovirt) { + /* + * All virtual IRQs are mapped by a single long word. + * There is exactly BITS_PER_LONG virqs, and they are + * always last in the interrupt map, starting at + * IPIPE_VIRQ_BASE. Therefore, we only need to test a + * single bit within the high and middle maps to check + * whether a virtual IRQ is pending (the computations + * below are constant). + */ + vl0b = IPIPE_VIRQ_BASE / (BITS_PER_LONG * BITS_PER_LONG); + himask = (1L << vl0b); + vl1b = IPIPE_VIRQ_BASE / BITS_PER_LONG; + mdmask = (1L << (vl1b & (BITS_PER_LONG-1))); + } else + himask = mdmask = ~0L; + + l0m = p->irqpend_himap & himask; + if (unlikely(l0m == 0)) + return -1; + + l0b = __ipipe_ffnz(l0m); + l1m = p->irqpend_mdmap[l0b] & mdmask; + if (unlikely(l1m == 0)) + return -1; + + l1b = __ipipe_ffnz(l1m) + l0b * BITS_PER_LONG; + l2m = p->irqpend_lomap[l1b]; + if (unlikely(l2m == 0)) + return -1; + + l2b = __ipipe_ffnz(l2m); + irq = l1b * BITS_PER_LONG + l2b; + + __clear_bit(irq, p->irqpend_lomap); + if (p->irqpend_lomap[l1b] == 0) { + __clear_bit(l1b, p->irqpend_mdmap); + if (p->irqpend_mdmap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } + + return irq; +} + +#else /* __IPIPE_2LEVEL_IRQMAP */ + +/* Must be called hw IRQs off. */ +static inline void __ipipe_set_irq_held(struct ipipe_percpu_domain_data *p, + unsigned int irq) +{ + __set_bit(irq, p->irqheld_map); + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd); + int l0b = irq / BITS_PER_LONG; + + prefetchw(p); + + if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) { + __set_bit(irq, p->irqpend_lomap); + __set_bit(l0b, &p->irqpend_himap); + } else + __set_bit(irq, p->irqheld_map); + + p->irqall[irq]++; +} + +/* Must be called hw IRQs off. */ +void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b = irq / BITS_PER_LONG; + + if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + p = ipipe_percpudom_ptr(ipd, cpu); + if (__test_and_clear_bit(irq, p->irqpend_lomap)) { + __set_bit(irq, p->irqheld_map); + if (p->irqpend_lomap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + } +} + +/* Must be called hw IRQs off. */ +void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq) +{ + struct ipipe_percpu_domain_data *p; + int l0b = irq / BITS_PER_LONG, cpu; + + if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, + &ipd->irqs[irq].control))) + return; + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + if (test_and_clear_bit(irq, p->irqheld_map)) { + /* We need atomic ops here: */ + set_bit(irq, p->irqpend_lomap); + set_bit(l0b, &p->irqpend_himap); + } + } +} + +static inline int __ipipe_next_irq(struct ipipe_percpu_domain_data *p, + int dovirt) +{ + unsigned long l0m, l1m, himask = ~0L; + int l0b, l1b; + + himask <<= dovirt ? IPIPE_VIRQ_BASE/BITS_PER_LONG : 0; + + l0m = p->irqpend_himap & himask; + if (unlikely(l0m == 0)) + return -1; + + l0b = __ipipe_ffnz(l0m); + l1m = p->irqpend_lomap[l0b]; + if (unlikely(l1m == 0)) + return -1; + + l1b = __ipipe_ffnz(l1m); + __clear_bit(l1b, &p->irqpend_lomap[l0b]); + if (p->irqpend_lomap[l0b] == 0) + __clear_bit(l0b, &p->irqpend_himap); + + return l0b * BITS_PER_LONG + l1b; +} + +#endif /* __IPIPE_2LEVEL_IRQMAP */ + +/* + * __ipipe_walk_pipeline(): Plays interrupts pending in the log. Must + * be called with local hw interrupts disabled. + */ +void __ipipe_walk_pipeline(struct list_head *pos) +{ + struct ipipe_domain *this_domain = __ipipe_current_domain, *next_domain; + struct ipipe_percpu_domain_data *p, *np; + + p = ipipe_cpudom_ptr(this_domain); + + while (pos != &__ipipe_pipeline) { + + next_domain = list_entry(pos, struct ipipe_domain, p_link); + np = ipipe_cpudom_ptr(next_domain); + + if (test_bit(IPIPE_STALL_FLAG, &np->status)) + break; /* Stalled stage -- do not go further. */ + + if (__ipipe_ipending_p(np)) { + if (next_domain == this_domain) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + else { + + p->evsync = 0; + __ipipe_current_domain = next_domain; + ipipe_suspend_domain(); /* Sync stage and propagate interrupts. */ + + if (__ipipe_current_domain == next_domain) + __ipipe_current_domain = this_domain; + /* + * Otherwise, something changed the current domain under our + * feet recycling the register set; do not override the new + * domain. + */ + + if (__ipipe_ipending_p(p) && + !test_bit(IPIPE_STALL_FLAG, &p->status)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + } + break; + } else if (next_domain == this_domain) + break; + + pos = next_domain->p_link.next; + } +} + +/* + * ipipe_suspend_domain() -- Suspend the current domain, switching to + * the next one which has pending work down the pipeline. + */ +void ipipe_suspend_domain(void) +{ + struct ipipe_domain *this_domain, *next_domain; + struct ipipe_percpu_domain_data *p; + struct list_head *ln; + unsigned long flags; + + local_irq_save_hw(flags); + + this_domain = next_domain = __ipipe_current_domain; + p = ipipe_cpudom_ptr(this_domain); + p->status &= ~(IPIPE_STALL_MASK|IPIPE_SYNC_MASK); + + if (__ipipe_ipending_p(p)) + goto sync_stage; + + for (;;) { + ln = next_domain->p_link.next; + + if (ln == &__ipipe_pipeline) + break; + + next_domain = list_entry(ln, struct ipipe_domain, p_link); + p = ipipe_cpudom_ptr(next_domain); + + if (p->status & IPIPE_STALL_MASK) + break; + + if (!__ipipe_ipending_p(p)) + continue; + + __ipipe_current_domain = next_domain; +sync_stage: + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + if (__ipipe_current_domain != next_domain) + /* + * Something has changed the current domain under our + * feet, recycling the register set; take note. + */ + this_domain = __ipipe_current_domain; + } + + __ipipe_current_domain = this_domain; + + local_irq_restore_hw(flags); +} + + +/* ipipe_alloc_virq() -- Allocate a pipelined virtual/soft interrupt. + * Virtual interrupts are handled in exactly the same way than their + * hw-generated counterparts wrt pipelining. + */ +unsigned ipipe_alloc_virq(void) +{ + unsigned long flags, irq = 0; + int ipos; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + if (__ipipe_virtual_irq_map != ~0) { + ipos = ffz(__ipipe_virtual_irq_map); + set_bit(ipos, &__ipipe_virtual_irq_map); + irq = ipos + IPIPE_VIRQ_BASE; + } + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return irq; +} + +/* + * ipipe_control_irq() -- Change modes of a pipelined interrupt for + * the current domain. + */ +int ipipe_virtualize_irq(struct ipipe_domain *ipd, + unsigned irq, + ipipe_irq_handler_t handler, + void *cookie, + ipipe_irq_ackfn_t acknowledge, + unsigned modemask) +{ + ipipe_irq_handler_t old_handler; + struct irq_desc *desc; + unsigned long flags; + int err; + + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + + if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) + return -EPERM; + + if (!test_bit(IPIPE_AHEAD_FLAG, &ipd->flags)) + /* Silently unwire interrupts for non-heading domains. */ + modemask &= ~IPIPE_WIRED_MASK; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + old_handler = ipd->irqs[irq].handler; + + if (handler != NULL) { + if (handler == IPIPE_SAME_HANDLER) { + handler = old_handler; + cookie = ipd->irqs[irq].cookie; + + if (handler == NULL) { + err = -EINVAL; + goto unlock_and_exit; + } + } else if ((modemask & IPIPE_EXCLUSIVE_MASK) != 0 && + old_handler != NULL) { + err = -EBUSY; + goto unlock_and_exit; + } + + /* Wired interrupts can only be delivered to domains + * always heading the pipeline, and using dynamic + * propagation. */ + + if ((modemask & IPIPE_WIRED_MASK) != 0) { + if ((modemask & (IPIPE_PASS_MASK | IPIPE_STICKY_MASK)) != 0) { + err = -EINVAL; + goto unlock_and_exit; + } + modemask |= (IPIPE_HANDLE_MASK); + } + + if ((modemask & IPIPE_STICKY_MASK) != 0) + modemask |= IPIPE_HANDLE_MASK; + } else + modemask &= + ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK | + IPIPE_EXCLUSIVE_MASK | IPIPE_WIRED_MASK); + + if (acknowledge == NULL && !ipipe_virtual_irq_p(irq)) + /* + * Acknowledge handler unspecified for a hw interrupt: + * use the Linux-defined handler instead. + */ + acknowledge = ipipe_root_domain->irqs[irq].acknowledge; + + ipd->irqs[irq].handler = handler; + ipd->irqs[irq].cookie = cookie; + ipd->irqs[irq].acknowledge = acknowledge; + ipd->irqs[irq].control = modemask; + + if (irq < NR_IRQS && !ipipe_virtual_irq_p(irq)) { + desc = irq_to_desc(irq); + if (handler != NULL) { + if (desc) + __ipipe_enable_irqdesc(ipd, irq); + + if ((modemask & IPIPE_ENABLE_MASK) != 0) { + if (ipd != __ipipe_current_domain) { + /* + * IRQ enable/disable state is domain-sensitive, so we + * may not change it for another domain. What is + * allowed however is forcing some domain to handle an + * interrupt source, by passing the proper 'ipd' + * descriptor which thus may be different from + * __ipipe_current_domain. + */ + err = -EPERM; + goto unlock_and_exit; + } + if (desc) + __ipipe_enable_irq(irq); + } + } else if (old_handler != NULL && desc) + __ipipe_disable_irqdesc(ipd, irq); + } + + err = 0; + + unlock_and_exit: + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return err; +} + +/* ipipe_control_irq() -- Change modes of a pipelined interrupt for + * the current domain. */ + +int ipipe_control_irq(unsigned irq, unsigned clrmask, unsigned setmask) +{ + struct ipipe_domain *ipd; + unsigned long flags; + + if (irq >= IPIPE_NR_IRQS) + return -EINVAL; + + spin_lock_irqsave(&__ipipe_pipelock, flags); + + ipd = __ipipe_current_domain; + + if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) { + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + return -EPERM; + } + + if (ipd->irqs[irq].handler == NULL) + setmask &= ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK); + + if ((setmask & IPIPE_STICKY_MASK) != 0) + setmask |= IPIPE_HANDLE_MASK; + + if ((clrmask & (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK)) != 0) /* If one goes, both go. */ + clrmask |= (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK); + + ipd->irqs[irq].control &= ~clrmask; + ipd->irqs[irq].control |= setmask; + + if ((setmask & IPIPE_ENABLE_MASK) != 0) + __ipipe_enable_irq(irq); + else if ((clrmask & IPIPE_ENABLE_MASK) != 0) + __ipipe_disable_irq(irq); + + spin_unlock_irqrestore(&__ipipe_pipelock, flags); + + return 0; +} + +/* __ipipe_dispatch_event() -- Low-level event dispatcher. */ + +int __ipipe_dispatch_event (unsigned event, void *data) +{ + struct ipipe_domain *start_domain, *this_domain, *next_domain; + struct ipipe_percpu_domain_data *np; + ipipe_event_handler_t evhand; + struct list_head *pos, *npos; + unsigned long flags; + int propagate = 1; + + local_irq_save_hw(flags); + + start_domain = this_domain = __ipipe_current_domain; + + list_for_each_safe(pos, npos, &__ipipe_pipeline) { + /* + * Note: Domain migration may occur while running + * event or interrupt handlers, in which case the + * current register set is going to be recycled for a + * different domain than the initiating one. We do + * care for that, always tracking the current domain + * descriptor upon return from those handlers. + */ + next_domain = list_entry(pos, struct ipipe_domain, p_link); + np = ipipe_cpudom_ptr(next_domain); + + /* + * Keep a cached copy of the handler's address since + * ipipe_catch_event() may clear it under our feet. + */ + evhand = next_domain->evhand[event]; + + if (evhand != NULL) { + __ipipe_current_domain = next_domain; + np->evsync |= (1LL << event); + local_irq_restore_hw(flags); + propagate = !evhand(event, start_domain, data); + local_irq_save_hw(flags); + /* + * We may have a migration issue here, if the + * current task is migrated to another CPU on + * behalf of the invoked handler, usually when + * a syscall event is processed. However, + * ipipe_catch_event() will make sure that a + * CPU that clears a handler for any given + * event will not attempt to wait for itself + * to clear the evsync bit for that event, + * which practically plugs the hole, without + * resorting to a much more complex strategy. + */ + np->evsync &= ~(1LL << event); + if (__ipipe_current_domain != next_domain) + this_domain = __ipipe_current_domain; + } + + /* NEVER sync the root stage here. */ + if (next_domain != ipipe_root_domain && + __ipipe_ipending_p(np) && + !test_bit(IPIPE_STALL_FLAG, &np->status)) { + __ipipe_current_domain = next_domain; + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + if (__ipipe_current_domain != next_domain) + this_domain = __ipipe_current_domain; + } + + __ipipe_current_domain = this_domain; + + if (next_domain == this_domain || !propagate) + break; + } + + local_irq_restore_hw(flags); + + return !propagate; +} + +/* + * __ipipe_dispatch_wired -- Wired interrupt dispatcher. Wired + * interrupts are immediately and unconditionally delivered to the + * domain heading the pipeline upon receipt, and such domain must have + * been registered as an invariant head for the system (priority == + * IPIPE_HEAD_PRIORITY). The motivation for using wired interrupts is + * to get an extra-fast dispatching path for those IRQs, by relying on + * a straightforward logic based on assumptions that must always be + * true for invariant head domains. The following assumptions are + * made when dealing with such interrupts: + * + * 1- Wired interrupts are purely dynamic, i.e. the decision to + * propagate them down the pipeline must be done from the head domain + * ISR. + * 2- Wired interrupts cannot be shared or sticky. + * 3- The root domain cannot be an invariant pipeline head, in + * consequence of what the root domain cannot handle wired + * interrupts. + * 4- Wired interrupts must have a valid acknowledge handler for the + * head domain (if needed, see __ipipe_handle_irq). + * + * Called with hw interrupts off. + */ + +void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq) +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head); + + prefetchw(p); + + if (unlikely(test_bit(IPIPE_LOCK_FLAG, &head->irqs[irq].control))) { + /* + * If we can't process this IRQ right now, we must + * mark it as held, so that it will get played during + * normal log sync when the corresponding interrupt + * source is eventually unlocked. + */ + __ipipe_set_irq_held(p, irq); + return; + } + + if (test_bit(IPIPE_STALL_FLAG, &p->status)) { + __ipipe_set_irq_pending(head, irq); + return; + } + + __ipipe_dispatch_wired_nocheck(head, irq); +} + +void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq) /* hw interrupts off */ +{ + struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head); + struct ipipe_domain *old; + + prefetchw(p); + + old = __ipipe_current_domain; + __ipipe_current_domain = head; /* Switch to the head domain. */ + + p->irqall[irq]++; + __set_bit(IPIPE_STALL_FLAG, &p->status); + head->irqs[irq].handler(irq, head->irqs[irq].cookie); /* Call the ISR. */ + __ipipe_run_irqtail(); + __clear_bit(IPIPE_STALL_FLAG, &p->status); + + if (__ipipe_current_domain == head) { + __ipipe_current_domain = old; + if (old == head) { + if (__ipipe_ipending_p(p)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + return; + } + } + + __ipipe_walk_pipeline(&head->p_link); +} + +/* + * __ipipe_sync_stage() -- Flush the pending IRQs for the current + * domain (and processor). This routine flushes the interrupt log + * (see "Optimistic interrupt protection" from D. Stodolsky et al. for + * more on the deferred interrupt scheme). Every interrupt that + * occurred while the pipeline was stalled gets played. WARNING: + * callers on SMP boxen should always check for CPU migration on + * return of this routine. + * + * This routine must be called with hw interrupts off. + */ +void __ipipe_sync_stage(int dovirt) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *ipd; + int cpu, irq; + + ipd = __ipipe_current_domain; + p = ipipe_cpudom_ptr(ipd); + + if (__test_and_set_bit(IPIPE_SYNC_FLAG, &p->status)) { + /* + * Some questionable code in the root domain may enter + * busy waits for IRQs over interrupt context, so we + * unfortunately have to allow piling up IRQs for + * them. Non-root domains are not allowed to do this. + */ + if (ipd != ipipe_root_domain) + return; + } + + cpu = ipipe_processor_id(); + + for (;;) { + irq = __ipipe_next_irq(p, dovirt); + if (irq < 0) + break; + /* + * Make sure the compiler does not reorder + * wrongly, so that all updates to maps are + * done before the handler gets called. + */ + barrier(); + + if (test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control)) + continue; + + __set_bit(IPIPE_STALL_FLAG, &p->status); + smp_wmb(); + + if (ipd == ipipe_root_domain) + trace_hardirqs_off(); + + __ipipe_run_isr(ipd, irq); + barrier(); + p = ipipe_cpudom_ptr(__ipipe_current_domain); +#ifdef CONFIG_SMP + { + int newcpu = ipipe_processor_id(); + + if (newcpu != cpu) { /* Handle CPU migration. */ + /* + * We expect any domain to clear the SYNC bit each + * time it switches in a new task, so that preemptions + * and/or CPU migrations (in the SMP case) over the + * ISR do not lock out the log syncer for some + * indefinite amount of time. In the Linux case, + * schedule() handles this (see kernel/sched.c). For + * this reason, we don't bother clearing it here for + * the source CPU in the migration handling case, + * since it must have scheduled another task in by + * now. + */ + __set_bit(IPIPE_SYNC_FLAG, &p->status); + cpu = newcpu; + } + } +#endif /* CONFIG_SMP */ +#ifdef CONFIG_TRACE_IRQFLAGS + if (__ipipe_root_domain_p && + test_bit(IPIPE_STALL_FLAG, &p->status)) + trace_hardirqs_on(); +#endif + __clear_bit(IPIPE_STALL_FLAG, &p->status); + } + + __clear_bit(IPIPE_SYNC_FLAG, &p->status); +} + +/* ipipe_register_domain() -- Link a new domain to the pipeline. */ + +int ipipe_register_domain(struct ipipe_domain *ipd, + struct ipipe_domain_attr *attr) +{ + struct ipipe_percpu_domain_data *p; + struct list_head *pos = NULL; + struct ipipe_domain *_ipd; + unsigned long flags; + + if (!ipipe_root_domain_p) { + printk(KERN_WARNING + "I-pipe: Only the root domain may register a new domain.\n"); + return -EPERM; + } + + flags = ipipe_critical_enter(NULL); + + if (attr->priority == IPIPE_HEAD_PRIORITY) { + if (test_bit(IPIPE_HEAD_SLOT, &__ipipe_domain_slot_map)) { + ipipe_critical_exit(flags); + return -EAGAIN; /* Cannot override current head. */ + } + ipd->slot = IPIPE_HEAD_SLOT; + } else + ipd->slot = ffz(__ipipe_domain_slot_map); + + if (ipd->slot < CONFIG_IPIPE_DOMAINS) { + set_bit(ipd->slot, &__ipipe_domain_slot_map); + list_for_each(pos, &__ipipe_pipeline) { + _ipd = list_entry(pos, struct ipipe_domain, p_link); + if (_ipd->domid == attr->domid) + break; + } + } + + ipipe_critical_exit(flags); + + if (pos != &__ipipe_pipeline) { + if (ipd->slot < CONFIG_IPIPE_DOMAINS) + clear_bit(ipd->slot, &__ipipe_domain_slot_map); + return -EBUSY; + } + +#ifndef CONFIG_SMP + /* + * Set up the perdomain pointers for direct access to the + * percpu domain data. This saves a costly multiply each time + * we need to refer to the contents of the percpu domain data + * array. + */ + __raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = &__raw_get_cpu_var(ipipe_percpu_darray)[ipd->slot]; +#endif + + ipd->name = attr->name; + ipd->domid = attr->domid; + ipd->pdd = attr->pdd; + ipd->flags = 0; + + if (attr->priority == IPIPE_HEAD_PRIORITY) { + ipd->priority = INT_MAX; + __set_bit(IPIPE_AHEAD_FLAG,&ipd->flags); + } + else + ipd->priority = attr->priority; + + __ipipe_init_stage(ipd); + + INIT_LIST_HEAD(&ipd->p_link); + +#ifdef CONFIG_PROC_FS + __ipipe_add_domain_proc(ipd); +#endif /* CONFIG_PROC_FS */ + + flags = ipipe_critical_enter(NULL); + + list_for_each(pos, &__ipipe_pipeline) { + _ipd = list_entry(pos, struct ipipe_domain, p_link); + if (ipd->priority > _ipd->priority) + break; + } + + list_add_tail(&ipd->p_link, pos); + + ipipe_critical_exit(flags); + + printk(KERN_INFO "I-pipe: Domain %s registered.\n", ipd->name); + + if (attr->entry == NULL) + return 0; + + /* + * Finally, allow the new domain to perform its initialization + * duties. + */ + local_irq_save_hw_smp(flags); + __ipipe_current_domain = ipd; + local_irq_restore_hw_smp(flags); + attr->entry(); + local_irq_save_hw(flags); + __ipipe_current_domain = ipipe_root_domain; + p = ipipe_root_cpudom_ptr(); + + if (__ipipe_ipending_p(p) && + !test_bit(IPIPE_STALL_FLAG, &p->status)) + __ipipe_sync_pipeline(IPIPE_IRQ_DOALL); + + local_irq_restore_hw(flags); + + return 0; +} + +/* ipipe_unregister_domain() -- Remove a domain from the pipeline. */ + +int ipipe_unregister_domain(struct ipipe_domain *ipd) +{ + unsigned long flags; + + if (!ipipe_root_domain_p) { + printk(KERN_WARNING + "I-pipe: Only the root domain may unregister a domain.\n"); + return -EPERM; + } + + if (ipd == ipipe_root_domain) { + printk(KERN_WARNING + "I-pipe: Cannot unregister the root domain.\n"); + return -EPERM; + } +#ifdef CONFIG_SMP + { + struct ipipe_percpu_domain_data *p; + unsigned int irq; + int cpu; + + /* + * In the SMP case, wait for the logged events to drain on + * other processors before eventually removing the domain + * from the pipeline. + */ + + ipipe_unstall_pipeline_from(ipd); + + flags = ipipe_critical_enter(NULL); + + for (irq = 0; irq < IPIPE_NR_IRQS; irq++) { + clear_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control); + clear_bit(IPIPE_STICKY_FLAG, &ipd->irqs[irq].control); + set_bit(IPIPE_PASS_FLAG, &ipd->irqs[irq].control); + } + + ipipe_critical_exit(flags); + + for_each_online_cpu(cpu) { + p = ipipe_percpudom_ptr(ipd, cpu); + while (__ipipe_ipending_p(p)) + cpu_relax(); + } + } +#endif /* CONFIG_SMP */ + + mutex_lock(&ipd->mutex); + +#ifdef CONFIG_PROC_FS + __ipipe_remove_domain_proc(ipd); +#endif /* CONFIG_PROC_FS */ + + /* + * Simply remove the domain from the pipeline and we are almost done. + */ + + flags = ipipe_critical_enter(NULL); + list_del_init(&ipd->p_link); + ipipe_critical_exit(flags); + + __ipipe_cleanup_domain(ipd); + + mutex_unlock(&ipd->mutex); + + printk(KERN_INFO "I-pipe: Domain %s unregistered.\n", ipd->name); + + return 0; +} + +/* + * ipipe_propagate_irq() -- Force a given IRQ propagation on behalf of + * a running interrupt handler to the next domain down the pipeline. + * ipipe_schedule_irq() -- Does almost the same as above, but attempts + * to pend the interrupt for the current domain first. + * Must be called hw IRQs off. + */ +void __ipipe_pend_irq(unsigned irq, struct list_head *head) +{ + struct ipipe_domain *ipd; + struct list_head *ln; + +#ifdef CONFIG_IPIPE_DEBUG + BUG_ON(irq >= IPIPE_NR_IRQS || + (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))); +#endif + for (ln = head; ln != &__ipipe_pipeline; ln = ipd->p_link.next) { + ipd = list_entry(ln, struct ipipe_domain, p_link); + if (test_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control)) { + __ipipe_set_irq_pending(ipd, irq); + return; + } + } +} + +/* ipipe_free_virq() -- Release a virtual/soft interrupt. */ + +int ipipe_free_virq(unsigned virq) +{ + if (!ipipe_virtual_irq_p(virq)) + return -EINVAL; + + clear_bit(virq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map); + + return 0; +} + +void ipipe_init_attr(struct ipipe_domain_attr *attr) +{ + attr->name = "anon"; + attr->domid = 1; + attr->entry = NULL; + attr->priority = IPIPE_ROOT_PRIO; + attr->pdd = NULL; +} + +/* + * ipipe_catch_event() -- Interpose or remove an event handler for a + * given domain. + */ +ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd, + unsigned event, + ipipe_event_handler_t handler) +{ + ipipe_event_handler_t old_handler; + unsigned long flags; + int self = 0, cpu; + + if (event & IPIPE_EVENT_SELF) { + event &= ~IPIPE_EVENT_SELF; + self = 1; + } + + if (event >= IPIPE_NR_EVENTS) + return NULL; + + flags = ipipe_critical_enter(NULL); + + if (!(old_handler = xchg(&ipd->evhand[event],handler))) { + if (handler) { + if (self) + ipd->evself |= (1LL << event); + else + __ipipe_event_monitors[event]++; + } + } + else if (!handler) { + if (ipd->evself & (1LL << event)) + ipd->evself &= ~(1LL << event); + else + __ipipe_event_monitors[event]--; + } else if ((ipd->evself & (1LL << event)) && !self) { + __ipipe_event_monitors[event]++; + ipd->evself &= ~(1LL << event); + } else if (!(ipd->evself & (1LL << event)) && self) { + __ipipe_event_monitors[event]--; + ipd->evself |= (1LL << event); + } + + ipipe_critical_exit(flags); + + if (!handler && ipipe_root_domain_p) { + /* + * If we cleared a handler on behalf of the root + * domain, we have to wait for any current invocation + * to drain, since our caller might subsequently unmap + * the target domain. To this aim, this code + * synchronizes with __ipipe_dispatch_event(), + * guaranteeing that either the dispatcher sees a null + * handler in which case it discards the invocation + * (which also prevents from entering a livelock), or + * finds a valid handler and calls it. Symmetrically, + * ipipe_catch_event() ensures that the called code + * won't be unmapped under our feet until the event + * synchronization flag is cleared for the given event + * on all CPUs. + */ + preempt_disable(); + cpu = smp_processor_id(); + /* + * Hack: this solves the potential migration issue + * raised in __ipipe_dispatch_event(). This is a + * work-around which makes the assumption that other + * CPUs will subsequently, either process at least one + * interrupt for the target domain, or call + * __ipipe_dispatch_event() without going through a + * migration while running the handler at least once; + * practically, this is safe on any normally running + * system. + */ + ipipe_percpudom(ipd, evsync, cpu) &= ~(1LL << event); + preempt_enable(); + + for_each_online_cpu(cpu) { + while (ipipe_percpudom(ipd, evsync, cpu) & (1LL << event)) + schedule_timeout_interruptible(HZ / 50); + } + } + + return old_handler; +} + +cpumask_t ipipe_set_irq_affinity (unsigned irq, cpumask_t cpumask) +{ +#ifdef CONFIG_SMP + if (irq >= IPIPE_NR_XIRQS) + /* Allow changing affinity of external IRQs only. */ + return CPU_MASK_NONE; + + if (num_online_cpus() > 1) + return __ipipe_set_irq_affinity(irq,cpumask); +#endif /* CONFIG_SMP */ + + return CPU_MASK_NONE; +} + +int ipipe_send_ipi (unsigned ipi, cpumask_t cpumask) + +{ +#ifdef CONFIG_SMP + return __ipipe_send_ipi(ipi,cpumask); +#else /* !CONFIG_SMP */ + return -EINVAL; +#endif /* CONFIG_SMP */ +} + +int ipipe_alloc_ptdkey (void) +{ + unsigned long flags; + int key = -1; + + spin_lock_irqsave(&__ipipe_pipelock,flags); + + if (__ipipe_ptd_key_count < IPIPE_ROOT_NPTDKEYS) { + key = ffz(__ipipe_ptd_key_map); + set_bit(key,&__ipipe_ptd_key_map); + __ipipe_ptd_key_count++; + } + + spin_unlock_irqrestore(&__ipipe_pipelock,flags); + + return key; +} + +int ipipe_free_ptdkey (int key) +{ + unsigned long flags; + + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return -EINVAL; + + spin_lock_irqsave(&__ipipe_pipelock,flags); + + if (test_and_clear_bit(key,&__ipipe_ptd_key_map)) + __ipipe_ptd_key_count--; + + spin_unlock_irqrestore(&__ipipe_pipelock,flags); + + return 0; +} + +int ipipe_set_ptd (int key, void *value) + +{ + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return -EINVAL; + + current->ptd[key] = value; + + return 0; +} + +void *ipipe_get_ptd (int key) + +{ + if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS) + return NULL; + + return current->ptd[key]; +} + +#ifdef CONFIG_PROC_FS + +struct proc_dir_entry *ipipe_proc_root; + +static int __ipipe_version_info_proc(char *page, + char **start, + off_t off, int count, int *eof, void *data) +{ + int len = sprintf(page, "%s\n", IPIPE_VERSION_STRING); + + len -= off; + + if (len <= off + count) + *eof = 1; + + *start = page + off; + + if(len > count) + len = count; + + if(len < 0) + len = 0; + + return len; +} + +static int __ipipe_common_info_show(struct seq_file *p, void *data) +{ + struct ipipe_domain *ipd = (struct ipipe_domain *)p->private; + char handling, stickiness, lockbit, exclusive, virtuality; + + unsigned long ctlbits; + unsigned irq; + + seq_printf(p, " +----- Handling ([A]ccepted, [G]rabbed, [W]ired, [D]iscarded)\n"); + seq_printf(p, " |+---- Sticky\n"); + seq_printf(p, " ||+--- Locked\n"); + seq_printf(p, " |||+-- Exclusive\n"); + seq_printf(p, " ||||+- Virtual\n"); + seq_printf(p, "[IRQ] |||||\n"); + + mutex_lock(&ipd->mutex); + + for (irq = 0; irq < IPIPE_NR_IRQS; irq++) { + /* Remember to protect against + * ipipe_virtual_irq/ipipe_control_irq if more fields + * get involved. */ + ctlbits = ipd->irqs[irq].control; + + if (irq >= IPIPE_NR_XIRQS && !ipipe_virtual_irq_p(irq)) + /* + * There might be a hole between the last external + * IRQ and the first virtual one; skip it. + */ + continue; + + if (ipipe_virtual_irq_p(irq) + && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map)) + /* Non-allocated virtual IRQ; skip it. */ + continue; + + /* + * Statuses are as follows: + * o "accepted" means handled _and_ passed down the pipeline. + * o "grabbed" means handled, but the interrupt might be + * terminated _or_ passed down the pipeline depending on + * what the domain handler asks for to the I-pipe. + * o "wired" is basically the same as "grabbed", except that + * the interrupt is unconditionally delivered to an invariant + * pipeline head domain. + * o "passed" means unhandled by the domain but passed + * down the pipeline. + * o "discarded" means unhandled and _not_ passed down the + * pipeline. The interrupt merely disappears from the + * current domain down to the end of the pipeline. + */ + if (ctlbits & IPIPE_HANDLE_MASK) { + if (ctlbits & IPIPE_PASS_MASK) + handling = 'A'; + else if (ctlbits & IPIPE_WIRED_MASK) + handling = 'W'; + else + handling = 'G'; + } else if (ctlbits & IPIPE_PASS_MASK) + /* Do not output if no major action is taken. */ + continue; + else + handling = 'D'; + + if (ctlbits & IPIPE_STICKY_MASK) + stickiness = 'S'; + else + stickiness = '.'; + + if (ctlbits & IPIPE_LOCK_MASK) + lockbit = 'L'; + else + lockbit = '.'; + + if (ctlbits & IPIPE_EXCLUSIVE_MASK) + exclusive = 'X'; + else + exclusive = '.'; + + if (ipipe_virtual_irq_p(irq)) + virtuality = 'V'; + else + virtuality = '.'; + + seq_printf(p, " %3u: %c%c%c%c%c\n", + irq, handling, stickiness, lockbit, exclusive, virtuality); + } + + seq_printf(p, "[Domain info]\n"); + + seq_printf(p, "id=0x%.8x\n", ipd->domid); + + if (test_bit(IPIPE_AHEAD_FLAG,&ipd->flags)) + seq_printf(p, "priority=topmost\n"); + else + seq_printf(p, "priority=%d\n", ipd->priority); + + mutex_unlock(&ipd->mutex); + + return 0; +} + +static int __ipipe_common_info_open(struct inode *inode, struct file *file) +{ + return single_open(file, __ipipe_common_info_show, PROC_I(inode)->pde->data); +} + +static struct file_operations __ipipe_info_proc_ops = { + .owner = THIS_MODULE, + .open = __ipipe_common_info_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +void __ipipe_add_domain_proc(struct ipipe_domain *ipd) +{ + struct proc_dir_entry *e = create_proc_entry(ipd->name, 0444, ipipe_proc_root); + if (e) { + e->proc_fops = &__ipipe_info_proc_ops; + e->data = (void*) ipd; + } +} + +void __ipipe_remove_domain_proc(struct ipipe_domain *ipd) +{ + remove_proc_entry(ipd->name,ipipe_proc_root); +} + +void __init ipipe_init_proc(void) +{ + ipipe_proc_root = create_proc_entry("ipipe",S_IFDIR, 0); + create_proc_read_entry("version",0444,ipipe_proc_root,&__ipipe_version_info_proc,NULL); + __ipipe_add_domain_proc(ipipe_root_domain); + + __ipipe_init_tracer(); +} + +#endif /* CONFIG_PROC_FS */ + +#ifdef CONFIG_IPIPE_DEBUG_CONTEXT + +DEFINE_PER_CPU(int, ipipe_percpu_context_check) = { 1 }; +DEFINE_PER_CPU(int, ipipe_saved_context_check_state); + +void ipipe_check_context(struct ipipe_domain *border_domain) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *this_domain; + unsigned long flags; + int cpu; + + local_irq_save_hw_smp(flags); + + this_domain = __ipipe_current_domain; + p = ipipe_head_cpudom_ptr(); + if (likely(this_domain->priority <= border_domain->priority && + !test_bit(IPIPE_STALL_FLAG, &p->status))) { + local_irq_restore_hw_smp(flags); + return; + } + + cpu = ipipe_processor_id(); + if (!per_cpu(ipipe_percpu_context_check, cpu)) { + local_irq_restore_hw_smp(flags); + return; + } + + local_irq_restore_hw_smp(flags); + + ipipe_context_check_off(); + ipipe_trace_panic_freeze(); + ipipe_set_printk_sync(__ipipe_current_domain); + + if (this_domain->priority > border_domain->priority) + printk(KERN_ERR "I-pipe: Detected illicit call from domain " + "'%s'\n" + KERN_ERR " into a service reserved for domain " + "'%s' and below.\n", + this_domain->name, border_domain->name); + else + printk(KERN_ERR "I-pipe: Detected stalled topmost domain, " + "probably caused by a bug.\n" + " A critical section may have been " + "left unterminated.\n"); + dump_stack(); + ipipe_trace_panic_dump(); +} + +EXPORT_SYMBOL(ipipe_check_context); + +#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */ + +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) + +int notrace __ipipe_check_percpu_access(void) +{ + struct ipipe_percpu_domain_data *p; + struct ipipe_domain *this_domain; + unsigned long flags; + int ret = 0; + + local_irq_save_hw_notrace(flags); + + this_domain = __raw_get_cpu_var(ipipe_percpu_domain); + + /* + * Only the root domain may implement preemptive CPU migration + * of tasks, so anything above in the pipeline should be fine. + */ + if (this_domain->priority > IPIPE_ROOT_PRIO) + goto out; + + if (raw_irqs_disabled_flags(flags)) + goto out; + + /* + * Last chance: hw interrupts were enabled on entry while + * running over the root domain, but the root stage might be + * currently stalled, in which case preemption would be + * disabled, and no migration could occur. + */ + if (this_domain == ipipe_root_domain) { + p = ipipe_root_cpudom_ptr(); + if (test_bit(IPIPE_STALL_FLAG, &p->status)) + goto out; + } + /* + * Our caller may end up accessing the wrong per-cpu variable + * instance due to CPU migration; tell it to complain about + * this. + */ + ret = 1; +out: + local_irq_restore_hw_notrace(flags); + + return ret; +} + +#endif /* CONFIG_IPIPE_DEBUG_INTERNAL && CONFIG_SMP */ + +EXPORT_SYMBOL(ipipe_virtualize_irq); +EXPORT_SYMBOL(ipipe_control_irq); +EXPORT_SYMBOL(ipipe_suspend_domain); +EXPORT_SYMBOL(ipipe_alloc_virq); +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_domain); +EXPORT_PER_CPU_SYMBOL(ipipe_percpu_darray); +EXPORT_SYMBOL(ipipe_root); +EXPORT_SYMBOL(ipipe_stall_pipeline_from); +EXPORT_SYMBOL(ipipe_test_and_stall_pipeline_from); +EXPORT_SYMBOL(ipipe_test_and_unstall_pipeline_from); +EXPORT_SYMBOL(ipipe_restore_pipeline_from); +EXPORT_SYMBOL(ipipe_unstall_pipeline_head); +EXPORT_SYMBOL(__ipipe_restore_pipeline_head); +EXPORT_SYMBOL(__ipipe_unstall_root); +EXPORT_SYMBOL(__ipipe_restore_root); +EXPORT_SYMBOL(__ipipe_spin_lock_irq); +EXPORT_SYMBOL(__ipipe_spin_unlock_irq); +EXPORT_SYMBOL(__ipipe_spin_lock_irqsave); +EXPORT_SYMBOL(__ipipe_spin_trylock_irqsave); +EXPORT_SYMBOL(__ipipe_spin_unlock_irqrestore); +EXPORT_SYMBOL(__ipipe_pipeline); +EXPORT_SYMBOL(__ipipe_lock_irq); +EXPORT_SYMBOL(__ipipe_unlock_irq); +EXPORT_SYMBOL(ipipe_register_domain); +EXPORT_SYMBOL(ipipe_unregister_domain); +EXPORT_SYMBOL(ipipe_free_virq); +EXPORT_SYMBOL(ipipe_init_attr); +EXPORT_SYMBOL(ipipe_catch_event); +EXPORT_SYMBOL(ipipe_alloc_ptdkey); +EXPORT_SYMBOL(ipipe_free_ptdkey); +EXPORT_SYMBOL(ipipe_set_ptd); +EXPORT_SYMBOL(ipipe_get_ptd); +EXPORT_SYMBOL(ipipe_set_irq_affinity); +EXPORT_SYMBOL(ipipe_send_ipi); +EXPORT_SYMBOL(__ipipe_pend_irq); +EXPORT_SYMBOL(__ipipe_set_irq_pending); +#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP) +EXPORT_SYMBOL(__ipipe_check_percpu_access); +#endif +#ifdef CONFIG_GENERIC_CLOCKEVENTS +EXPORT_SYMBOL(ipipe_request_tickdev); +EXPORT_SYMBOL(ipipe_release_tickdev); +#endif + +EXPORT_SYMBOL(ipipe_critical_enter); +EXPORT_SYMBOL(ipipe_critical_exit); +EXPORT_SYMBOL(ipipe_trigger_irq); +EXPORT_SYMBOL(ipipe_get_sysinfo); diff --git a/kernel/ipipe/tracer.c b/kernel/ipipe/tracer.c new file mode 100644 index 0000000..001a83e --- /dev/null +++ b/kernel/ipipe/tracer.c @@ -0,0 +1,1441 @@ +/* -*- linux-c -*- + * kernel/ipipe/tracer.c + * + * Copyright (C) 2005 Luotao Fu. + * 2005-2008 Jan Kiszka. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139, + * USA; either version 2 of the License, or (at your option) any later + * version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + */ + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/version.h> +#include <linux/kallsyms.h> +#include <linux/seq_file.h> +#include <linux/proc_fs.h> +#include <linux/ctype.h> +#include <linux/vmalloc.h> +#include <linux/pid.h> +#include <linux/vermagic.h> +#include <linux/sched.h> +#include <linux/ipipe.h> +#include <linux/ftrace.h> +#include <asm/uaccess.h> + +#define IPIPE_TRACE_PATHS 4 /* <!> Do not lower below 3 */ +#define IPIPE_DEFAULT_ACTIVE 0 +#define IPIPE_DEFAULT_MAX 1 +#define IPIPE_DEFAULT_FROZEN 2 + +#define IPIPE_TRACE_POINTS (1 << CONFIG_IPIPE_TRACE_SHIFT) +#define WRAP_POINT_NO(point) ((point) & (IPIPE_TRACE_POINTS-1)) + +#define IPIPE_DEFAULT_PRE_TRACE 10 +#define IPIPE_DEFAULT_POST_TRACE 10 +#define IPIPE_DEFAULT_BACK_TRACE 100 + +#define IPIPE_DELAY_NOTE 1000 /* in nanoseconds */ +#define IPIPE_DELAY_WARN 10000 /* in nanoseconds */ + +#define IPIPE_TFLG_NMI_LOCK 0x0001 +#define IPIPE_TFLG_NMI_HIT 0x0002 +#define IPIPE_TFLG_NMI_FREEZE_REQ 0x0004 + +#define IPIPE_TFLG_HWIRQ_OFF 0x0100 +#define IPIPE_TFLG_FREEZING 0x0200 +#define IPIPE_TFLG_CURRDOM_SHIFT 10 /* bits 10..11: current domain */ +#define IPIPE_TFLG_CURRDOM_MASK 0x0C00 +#define IPIPE_TFLG_DOMSTATE_SHIFT 12 /* bits 12..15: domain stalled? */ +#define IPIPE_TFLG_DOMSTATE_BITS 3 + +#define IPIPE_TFLG_DOMAIN_STALLED(point, n) \ + (point->flags & (1 << (n + IPIPE_TFLG_DOMSTATE_SHIFT))) +#define IPIPE_TFLG_CURRENT_DOMAIN(point) \ + ((point->flags & IPIPE_TFLG_CURRDOM_MASK) >> IPIPE_TFLG_CURRDOM_SHIFT) + +struct ipipe_trace_point { + short type; + short flags; + unsigned long eip; + unsigned long parent_eip; + unsigned long v; + unsigned long long timestamp; +}; + +struct ipipe_trace_path { + volatile int flags; + int dump_lock; /* separated from flags due to cross-cpu access */ + int trace_pos; /* next point to fill */ + int begin, end; /* finalised path begin and end */ + int post_trace; /* non-zero when in post-trace phase */ + unsigned long long length; /* max path length in cycles */ + unsigned long nmi_saved_eip; /* for deferred requests from NMIs */ + unsigned long nmi_saved_parent_eip; + unsigned long nmi_saved_v; + struct ipipe_trace_point point[IPIPE_TRACE_POINTS]; +} ____cacheline_aligned_in_smp; + +enum ipipe_trace_type +{ + IPIPE_TRACE_FUNC = 0, + IPIPE_TRACE_BEGIN, + IPIPE_TRACE_END, + IPIPE_TRACE_FREEZE, + IPIPE_TRACE_SPECIAL, + IPIPE_TRACE_PID, + IPIPE_TRACE_EVENT, +}; + +#define IPIPE_TYPE_MASK 0x0007 +#define IPIPE_TYPE_BITS 3 + +#ifdef CONFIG_IPIPE_TRACE_VMALLOC +static DEFINE_PER_CPU(struct ipipe_trace_path *, trace_path); +#else /* !CONFIG_IPIPE_TRACE_VMALLOC */ +static DEFINE_PER_CPU(struct ipipe_trace_path, trace_path[IPIPE_TRACE_PATHS]) = + { [0 ... IPIPE_TRACE_PATHS-1] = { .begin = -1, .end = -1 } }; +#endif /* CONFIG_IPIPE_TRACE_VMALLOC */ + +int ipipe_trace_enable = 0; + +static DEFINE_PER_CPU(int, active_path) = { IPIPE_DEFAULT_ACTIVE }; +static DEFINE_PER_CPU(int, max_path) = { IPIPE_DEFAULT_MAX }; +static DEFINE_PER_CPU(int, frozen_path) = { IPIPE_DEFAULT_FROZEN }; +static IPIPE_DEFINE_SPINLOCK(global_path_lock); +static int pre_trace = IPIPE_DEFAULT_PRE_TRACE; +static int post_trace = IPIPE_DEFAULT_POST_TRACE; +static int back_trace = IPIPE_DEFAULT_BACK_TRACE; +static int verbose_trace = 1; +static unsigned long trace_overhead; + +static unsigned long trigger_begin; +static unsigned long trigger_end; + +static DEFINE_MUTEX(out_mutex); +static struct ipipe_trace_path *print_path; +#ifdef CONFIG_IPIPE_TRACE_PANIC +static struct ipipe_trace_path *panic_path; +#endif /* CONFIG_IPIPE_TRACE_PANIC */ +static int print_pre_trace; +static int print_post_trace; + + +static long __ipipe_signed_tsc2us(long long tsc); +static void +__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point); +static void __ipipe_print_symname(struct seq_file *m, unsigned long eip); + + +static notrace void +__ipipe_store_domain_states(struct ipipe_trace_point *point) +{ + struct ipipe_domain *ipd; + struct list_head *pos; + int i = 0; + + list_for_each_prev(pos, &__ipipe_pipeline) { + ipd = list_entry(pos, struct ipipe_domain, p_link); + + if (test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status))) + point->flags |= 1 << (i + IPIPE_TFLG_DOMSTATE_SHIFT); + + if (ipd == __ipipe_current_domain) + point->flags |= i << IPIPE_TFLG_CURRDOM_SHIFT; + + if (++i > IPIPE_TFLG_DOMSTATE_BITS) + break; + } +} + +static notrace int __ipipe_get_free_trace_path(int old, int cpu) +{ + int new_active = old; + struct ipipe_trace_path *tp; + + do { + if (++new_active == IPIPE_TRACE_PATHS) + new_active = 0; + tp = &per_cpu(trace_path, cpu)[new_active]; + } while (new_active == per_cpu(max_path, cpu) || + new_active == per_cpu(frozen_path, cpu) || + tp->dump_lock); + + return new_active; +} + +static notrace void +__ipipe_migrate_pre_trace(struct ipipe_trace_path *new_tp, + struct ipipe_trace_path *old_tp, int old_pos) +{ + int i; + + new_tp->trace_pos = pre_trace+1; + + for (i = new_tp->trace_pos; i > 0; i--) + memcpy(&new_tp->point[WRAP_POINT_NO(new_tp->trace_pos-i)], + &old_tp->point[WRAP_POINT_NO(old_pos-i)], + sizeof(struct ipipe_trace_point)); + + /* mark the end (i.e. the point before point[0]) invalid */ + new_tp->point[IPIPE_TRACE_POINTS-1].eip = 0; +} + +static notrace struct ipipe_trace_path * +__ipipe_trace_end(int cpu, struct ipipe_trace_path *tp, int pos) +{ + struct ipipe_trace_path *old_tp = tp; + long active = per_cpu(active_path, cpu); + unsigned long long length; + + /* do we have a new worst case? */ + length = tp->point[tp->end].timestamp - + tp->point[tp->begin].timestamp; + if (length > per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)].length) { + /* we need protection here against other cpus trying + to start a proc dump */ + spin_lock(&global_path_lock); + + /* active path holds new worst case */ + tp->length = length; + per_cpu(max_path, cpu) = active; + + /* find next unused trace path */ + active = __ipipe_get_free_trace_path(active, cpu); + + spin_unlock(&global_path_lock); + + tp = &per_cpu(trace_path, cpu)[active]; + + /* migrate last entries for pre-tracing */ + __ipipe_migrate_pre_trace(tp, old_tp, pos); + } + + return tp; +} + +static notrace struct ipipe_trace_path * +__ipipe_trace_freeze(int cpu, struct ipipe_trace_path *tp, int pos) +{ + struct ipipe_trace_path *old_tp = tp; + long active = per_cpu(active_path, cpu); + int n; + + /* frozen paths have no core (begin=end) */ + tp->begin = tp->end; + + /* we need protection here against other cpus trying + * to set their frozen path or to start a proc dump */ + spin_lock(&global_path_lock); + + per_cpu(frozen_path, cpu) = active; + + /* find next unused trace path */ + active = __ipipe_get_free_trace_path(active, cpu); + + /* check if this is the first frozen path */ + for_each_possible_cpu(n) { + if (n != cpu && + per_cpu(trace_path, n)[per_cpu(frozen_path, n)].end >= 0) + tp->end = -1; + } + + spin_unlock(&global_path_lock); + + tp = &per_cpu(trace_path, cpu)[active]; + + /* migrate last entries for pre-tracing */ + __ipipe_migrate_pre_trace(tp, old_tp, pos); + + return tp; +} + +void notrace +__ipipe_trace(enum ipipe_trace_type type, unsigned long eip, + unsigned long parent_eip, unsigned long v) +{ + struct ipipe_trace_path *tp, *old_tp; + int pos, next_pos, begin; + struct ipipe_trace_point *point; + unsigned long flags; + int cpu; + + local_irq_save_hw_notrace(flags); + + cpu = ipipe_processor_id(); + restart: + tp = old_tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + /* here starts a race window with NMIs - catched below */ + + /* check for NMI recursion */ + if (unlikely(tp->flags & IPIPE_TFLG_NMI_LOCK)) { + tp->flags |= IPIPE_TFLG_NMI_HIT; + + /* first freeze request from NMI context? */ + if ((type == IPIPE_TRACE_FREEZE) && + !(tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ)) { + /* save arguments and mark deferred freezing */ + tp->flags |= IPIPE_TFLG_NMI_FREEZE_REQ; + tp->nmi_saved_eip = eip; + tp->nmi_saved_parent_eip = parent_eip; + tp->nmi_saved_v = v; + } + return; /* no need for restoring flags inside IRQ */ + } + + /* clear NMI events and set lock (atomically per cpu) */ + tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT | + IPIPE_TFLG_NMI_FREEZE_REQ)) + | IPIPE_TFLG_NMI_LOCK; + + /* check active_path again - some nasty NMI may have switched + * it meanwhile */ + if (unlikely(tp != + &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)])) { + /* release lock on wrong path and restart */ + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* there is no chance that the NMI got deferred + * => no need to check for pending freeze requests */ + goto restart; + } + + /* get the point buffer */ + pos = tp->trace_pos; + point = &tp->point[pos]; + + /* store all trace point data */ + point->type = type; + point->flags = raw_irqs_disabled_flags(flags) ? IPIPE_TFLG_HWIRQ_OFF : 0; + point->eip = eip; + point->parent_eip = parent_eip; + point->v = v; + ipipe_read_tsc(point->timestamp); + + __ipipe_store_domain_states(point); + + /* forward to next point buffer */ + next_pos = WRAP_POINT_NO(pos+1); + tp->trace_pos = next_pos; + + /* only mark beginning if we haven't started yet */ + begin = tp->begin; + if (unlikely(type == IPIPE_TRACE_BEGIN) && (begin < 0)) + tp->begin = pos; + + /* end of critical path, start post-trace if not already started */ + if (unlikely(type == IPIPE_TRACE_END) && + (begin >= 0) && !tp->post_trace) + tp->post_trace = post_trace + 1; + + /* freeze only if the slot is free and we are not already freezing */ + if ((unlikely(type == IPIPE_TRACE_FREEZE) || + (unlikely(eip >= trigger_begin && eip <= trigger_end) && + type == IPIPE_TRACE_FUNC)) && + per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)].begin < 0 && + !(tp->flags & IPIPE_TFLG_FREEZING)) { + tp->post_trace = post_trace + 1; + tp->flags |= IPIPE_TFLG_FREEZING; + } + + /* enforce end of trace in case of overflow */ + if (unlikely(WRAP_POINT_NO(next_pos + 1) == begin)) { + tp->end = pos; + goto enforce_end; + } + + /* stop tracing this path if we are in post-trace and + * a) that phase is over now or + * b) a new TRACE_BEGIN came in but we are not freezing this path */ + if (unlikely((tp->post_trace > 0) && ((--tp->post_trace == 0) || + ((type == IPIPE_TRACE_BEGIN) && + !(tp->flags & IPIPE_TFLG_FREEZING))))) { + /* store the path's end (i.e. excluding post-trace) */ + tp->end = WRAP_POINT_NO(pos - post_trace + tp->post_trace); + + enforce_end: + if (tp->flags & IPIPE_TFLG_FREEZING) + tp = __ipipe_trace_freeze(cpu, tp, pos); + else + tp = __ipipe_trace_end(cpu, tp, pos); + + /* reset the active path, maybe already start a new one */ + tp->begin = (type == IPIPE_TRACE_BEGIN) ? + WRAP_POINT_NO(tp->trace_pos - 1) : -1; + tp->end = -1; + tp->post_trace = 0; + tp->flags = 0; + + /* update active_path not earlier to avoid races with NMIs */ + per_cpu(active_path, cpu) = tp - per_cpu(trace_path, cpu); + } + + /* we still have old_tp and point, + * let's reset NMI lock and check for catches */ + old_tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + if (unlikely(old_tp->flags & IPIPE_TFLG_NMI_HIT)) { + /* well, this late tagging may not immediately be visible for + * other cpus already dumping this path - a minor issue */ + point->flags |= IPIPE_TFLG_NMI_HIT; + + /* handle deferred freezing from NMI context */ + if (old_tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ) + __ipipe_trace(IPIPE_TRACE_FREEZE, old_tp->nmi_saved_eip, + old_tp->nmi_saved_parent_eip, + old_tp->nmi_saved_v); + } + + local_irq_restore_hw_notrace(flags); +} + +static unsigned long __ipipe_global_path_lock(void) +{ + unsigned long flags; + int cpu; + struct ipipe_trace_path *tp; + + spin_lock_irqsave(&global_path_lock, flags); + + cpu = ipipe_processor_id(); + restart: + tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + /* here is small race window with NMIs - catched below */ + + /* clear NMI events and set lock (atomically per cpu) */ + tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT | + IPIPE_TFLG_NMI_FREEZE_REQ)) + | IPIPE_TFLG_NMI_LOCK; + + /* check active_path again - some nasty NMI may have switched + * it meanwhile */ + if (tp != &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]) { + /* release lock on wrong path and restart */ + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* there is no chance that the NMI got deferred + * => no need to check for pending freeze requests */ + goto restart; + } + + return flags; +} + +static void __ipipe_global_path_unlock(unsigned long flags) +{ + int cpu; + struct ipipe_trace_path *tp; + + /* release spinlock first - it's not involved in the NMI issue */ + __ipipe_spin_unlock_irqbegin(&global_path_lock); + + cpu = ipipe_processor_id(); + tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + tp->flags &= ~IPIPE_TFLG_NMI_LOCK; + + /* handle deferred freezing from NMI context */ + if (tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ) + __ipipe_trace(IPIPE_TRACE_FREEZE, tp->nmi_saved_eip, + tp->nmi_saved_parent_eip, tp->nmi_saved_v); + + /* See __ipipe_spin_lock_irqsave() and friends. */ + __ipipe_spin_unlock_irqcomplete(flags); +} + +void notrace ipipe_trace_begin(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_BEGIN, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_begin); + +void notrace ipipe_trace_end(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_END, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_end); + +void notrace ipipe_trace_freeze(unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_FREEZE, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_freeze); + +void notrace ipipe_trace_special(unsigned char id, unsigned long v) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_SPECIAL | (id << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, v); +} +EXPORT_SYMBOL(ipipe_trace_special); + +void notrace ipipe_trace_pid(pid_t pid, short prio) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_PID | (prio << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, pid); +} +EXPORT_SYMBOL(ipipe_trace_pid); + +void notrace ipipe_trace_event(unsigned char id, unsigned long delay_tsc) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_EVENT | (id << IPIPE_TYPE_BITS), + __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, delay_tsc); +} +EXPORT_SYMBOL(ipipe_trace_event); + +int ipipe_trace_max_reset(void) +{ + int cpu; + unsigned long flags; + struct ipipe_trace_path *path; + int ret = 0; + + flags = __ipipe_global_path_lock(); + + for_each_possible_cpu(cpu) { + path = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)]; + + if (path->dump_lock) { + ret = -EBUSY; + break; + } + + path->begin = -1; + path->end = -1; + path->trace_pos = 0; + path->length = 0; + } + + __ipipe_global_path_unlock(flags); + + return ret; +} +EXPORT_SYMBOL(ipipe_trace_max_reset); + +int ipipe_trace_frozen_reset(void) +{ + int cpu; + unsigned long flags; + struct ipipe_trace_path *path; + int ret = 0; + + flags = __ipipe_global_path_lock(); + + for_each_online_cpu(cpu) { + path = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)]; + + if (path->dump_lock) { + ret = -EBUSY; + break; + } + + path->begin = -1; + path->end = -1; + path->trace_pos = 0; + path->length = 0; + } + + __ipipe_global_path_unlock(flags); + + return ret; +} +EXPORT_SYMBOL(ipipe_trace_frozen_reset); + +static void +__ipipe_get_task_info(char *task_info, struct ipipe_trace_point *point, + int trylock) +{ + struct task_struct *task = NULL; + char buf[8]; + int i; + int locked = 1; + + if (trylock) { + if (!read_trylock(&tasklist_lock)) + locked = 0; + } else + read_lock(&tasklist_lock); + + if (locked) + task = find_task_by_pid_ns((pid_t)point->v, &init_pid_ns); + + if (task) + strncpy(task_info, task->comm, 11); + else + strcpy(task_info, "-<?>-"); + + if (locked) + read_unlock(&tasklist_lock); + + for (i = strlen(task_info); i < 11; i++) + task_info[i] = ' '; + + sprintf(buf, " %d ", point->type >> IPIPE_TYPE_BITS); + strcpy(task_info + (11 - strlen(buf)), buf); +} + +static void +__ipipe_get_event_date(char *buf,struct ipipe_trace_path *path, + struct ipipe_trace_point *point) +{ + long time; + int type; + + time = __ipipe_signed_tsc2us(point->timestamp - + path->point[path->begin].timestamp + point->v); + type = point->type >> IPIPE_TYPE_BITS; + + if (type == 0) + /* + * Event type #0 is predefined, stands for the next + * timer tick. + */ + sprintf(buf, "tick@%-6ld", time); + else + sprintf(buf, "%3d@%-7ld", type, time); +} + +#ifdef CONFIG_IPIPE_TRACE_PANIC +void ipipe_trace_panic_freeze(void) +{ + unsigned long flags; + int cpu; + + if (!ipipe_trace_enable) + return; + + ipipe_trace_enable = 0; + local_irq_save_hw_notrace(flags); + + cpu = ipipe_processor_id(); + + panic_path = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]; + + local_irq_restore_hw(flags); +} +EXPORT_SYMBOL(ipipe_trace_panic_freeze); + +void ipipe_trace_panic_dump(void) +{ + int cnt = back_trace; + int start, pos; + char buf[16]; + + if (!panic_path) + return; + + ipipe_context_check_off(); + + printk("I-pipe tracer log (%d points):\n", cnt); + + start = pos = WRAP_POINT_NO(panic_path->trace_pos-1); + + while (cnt-- > 0) { + struct ipipe_trace_point *point = &panic_path->point[pos]; + long time; + char info[16]; + int i; + + printk(" %c", + (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' '); + + for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--) + printk("%c", + (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ? + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '#' : '+') : + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '*' : ' ')); + + if (!point->eip) + printk("-<invalid>-\n"); + else { + __ipipe_trace_point_type(buf, point); + printk("%s", buf); + + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + printk(" "); + break; + + case IPIPE_TRACE_PID: + __ipipe_get_task_info(info, + point, 1); + printk("%s", info); + break; + + case IPIPE_TRACE_EVENT: + __ipipe_get_event_date(info, + panic_path, point); + printk("%s", info); + break; + + default: + printk("0x%08lx ", point->v); + } + + time = __ipipe_signed_tsc2us(point->timestamp - + panic_path->point[start].timestamp); + printk(" %5ld ", time); + + __ipipe_print_symname(NULL, point->eip); + printk(" ("); + __ipipe_print_symname(NULL, point->parent_eip); + printk(")\n"); + } + pos = WRAP_POINT_NO(pos - 1); + } + + panic_path = NULL; +} +EXPORT_SYMBOL(ipipe_trace_panic_dump); +#endif /* CONFIG_IPIPE_TRACE_PANIC */ + + +/* --- /proc output --- */ + +static notrace int __ipipe_in_critical_trpath(long point_no) +{ + return ((WRAP_POINT_NO(point_no-print_path->begin) < + WRAP_POINT_NO(print_path->end-print_path->begin)) || + ((print_path->end == print_path->begin) && + (WRAP_POINT_NO(point_no-print_path->end) > + print_post_trace))); +} + +static long __ipipe_signed_tsc2us(long long tsc) +{ + unsigned long long abs_tsc; + long us; + + /* ipipe_tsc2us works on unsigned => handle sign separately */ + abs_tsc = (tsc >= 0) ? tsc : -tsc; + us = ipipe_tsc2us(abs_tsc); + if (tsc < 0) + return -us; + else + return us; +} + +static void +__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point) +{ + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + strcpy(buf, "func "); + break; + + case IPIPE_TRACE_BEGIN: + strcpy(buf, "begin "); + break; + + case IPIPE_TRACE_END: + strcpy(buf, "end "); + break; + + case IPIPE_TRACE_FREEZE: + strcpy(buf, "freeze "); + break; + + case IPIPE_TRACE_SPECIAL: + sprintf(buf, "(0x%02x) ", + point->type >> IPIPE_TYPE_BITS); + break; + + case IPIPE_TRACE_PID: + sprintf(buf, "[%5d] ", (pid_t)point->v); + break; + + case IPIPE_TRACE_EVENT: + sprintf(buf, "event "); + break; + } +} + +static void +__ipipe_print_pathmark(struct seq_file *m, struct ipipe_trace_point *point) +{ + char mark = ' '; + int point_no = point - print_path->point; + int i; + + if (print_path->end == point_no) + mark = '<'; + else if (print_path->begin == point_no) + mark = '>'; + else if (__ipipe_in_critical_trpath(point_no)) + mark = ':'; + seq_printf(m, "%c%c", mark, + (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' '); + + if (!verbose_trace) + return; + + for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--) + seq_printf(m, "%c", + (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ? + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? + '#' : '+') : + (IPIPE_TFLG_DOMAIN_STALLED(point, i) ? '*' : ' ')); +} + +static void +__ipipe_print_delay(struct seq_file *m, struct ipipe_trace_point *point) +{ + unsigned long delay = 0; + int next; + char *mark = " "; + + next = WRAP_POINT_NO(point+1 - print_path->point); + + if (next != print_path->trace_pos) + delay = ipipe_tsc2ns(print_path->point[next].timestamp - + point->timestamp); + + if (__ipipe_in_critical_trpath(point - print_path->point)) { + if (delay > IPIPE_DELAY_WARN) + mark = "! "; + else if (delay > IPIPE_DELAY_NOTE) + mark = "+ "; + } + seq_puts(m, mark); + + if (verbose_trace) + seq_printf(m, "%3lu.%03lu%c ", delay/1000, delay%1000, + (point->flags & IPIPE_TFLG_NMI_HIT) ? 'N' : ' '); + else + seq_puts(m, " "); +} + +static void __ipipe_print_symname(struct seq_file *m, unsigned long eip) +{ + char namebuf[KSYM_NAME_LEN+1]; + unsigned long size, offset; + const char *sym_name; + char *modname; + + sym_name = kallsyms_lookup(eip, &size, &offset, &modname, namebuf); + +#ifdef CONFIG_IPIPE_TRACE_PANIC + if (!m) { + /* panic dump */ + if (sym_name) { + printk("%s+0x%lx", sym_name, offset); + if (modname) + printk(" [%s]", modname); + } + } else +#endif /* CONFIG_IPIPE_TRACE_PANIC */ + { + if (sym_name) { + if (verbose_trace) { + seq_printf(m, "%s+0x%lx", sym_name, offset); + if (modname) + seq_printf(m, " [%s]", modname); + } else + seq_puts(m, sym_name); + } else + seq_printf(m, "<%08lx>", eip); + } +} + +static void __ipipe_print_headline(struct seq_file *m) +{ + seq_printf(m, "Calibrated minimum trace-point overhead: %lu.%03lu " + "us\n\n", trace_overhead/1000, trace_overhead%1000); + + if (verbose_trace) { + const char *name[4] = { [0 ... 3] = "<unused>" }; + struct list_head *pos; + int i = 0; + + list_for_each_prev(pos, &__ipipe_pipeline) { + struct ipipe_domain *ipd = + list_entry(pos, struct ipipe_domain, p_link); + + name[i] = ipd->name; + if (++i > 3) + break; + } + + seq_printf(m, + " +----- Hard IRQs ('|': locked)\n" + " |+---- %s\n" + " ||+--- %s\n" + " |||+-- %s\n" + " ||||+- %s%s\n" + " ||||| +---------- " + "Delay flag ('+': > %d us, '!': > %d us)\n" + " ||||| | +- " + "NMI noise ('N')\n" + " ||||| | |\n" + " Type User Val. Time Delay Function " + "(Parent)\n", + name[3], name[2], name[1], name[0], + name[0] ? " ('*': domain stalled, '+': current, " + "'#': current+stalled)" : "", + IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000); + } else + seq_printf(m, + " +--------------- Hard IRQs ('|': locked)\n" + " | +- Delay flag " + "('+': > %d us, '!': > %d us)\n" + " | |\n" + " Type Time Function (Parent)\n", + IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000); +} + +static void *__ipipe_max_prtrace_start(struct seq_file *m, loff_t *pos) +{ + loff_t n = *pos; + + mutex_lock(&out_mutex); + + if (!n) { + struct ipipe_trace_path *tp; + unsigned long length_usecs; + int points, cpu; + unsigned long flags; + + /* protect against max_path/frozen_path updates while we + * haven't locked our target path, also avoid recursively + * taking global_path_lock from NMI context */ + flags = __ipipe_global_path_lock(); + + /* find the longest of all per-cpu paths */ + print_path = NULL; + for_each_online_cpu(cpu) { + tp = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)]; + if ((print_path == NULL) || + (tp->length > print_path->length)) { + print_path = tp; + break; + } + } + print_path->dump_lock = 1; + + __ipipe_global_path_unlock(flags); + + /* does this path actually contain data? */ + if (print_path->end == print_path->begin) + return NULL; + + /* number of points inside the critical path */ + points = WRAP_POINT_NO(print_path->end-print_path->begin+1); + + /* pre- and post-tracing length, post-trace length was frozen + in __ipipe_trace, pre-trace may have to be reduced due to + buffer overrun */ + print_pre_trace = pre_trace; + print_post_trace = WRAP_POINT_NO(print_path->trace_pos - + print_path->end - 1); + if (points+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1) + print_pre_trace = IPIPE_TRACE_POINTS - 1 - points - + print_post_trace; + + length_usecs = ipipe_tsc2us(print_path->length); + seq_printf(m, "I-pipe worst-case tracing service on %s/ipipe-%s\n" + "------------------------------------------------------------\n", + UTS_RELEASE, IPIPE_ARCH_STRING); + seq_printf(m, "CPU: %d, Begin: %lld cycles, Trace Points: " + "%d (-%d/+%d), Length: %lu us\n", + cpu, print_path->point[print_path->begin].timestamp, + points, print_pre_trace, print_post_trace, length_usecs); + __ipipe_print_headline(m); + } + + /* check if we are inside the trace range */ + if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 + + print_pre_trace + print_post_trace)) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin - + print_pre_trace + n)]; +} + +static void *__ipipe_prtrace_next(struct seq_file *m, void *p, loff_t *pos) +{ + loff_t n = ++*pos; + + /* check if we are inside the trace range with the next entry */ + if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 + + print_pre_trace + print_post_trace)) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin - + print_pre_trace + *pos)]; +} + +static void __ipipe_prtrace_stop(struct seq_file *m, void *p) +{ + if (print_path) + print_path->dump_lock = 0; + mutex_unlock(&out_mutex); +} + +static int __ipipe_prtrace_show(struct seq_file *m, void *p) +{ + long time; + struct ipipe_trace_point *point = p; + char buf[16]; + + if (!point->eip) { + seq_puts(m, "-<invalid>-\n"); + return 0; + } + + __ipipe_print_pathmark(m, point); + __ipipe_trace_point_type(buf, point); + seq_puts(m, buf); + if (verbose_trace) + switch (point->type & IPIPE_TYPE_MASK) { + case IPIPE_TRACE_FUNC: + seq_puts(m, " "); + break; + + case IPIPE_TRACE_PID: + __ipipe_get_task_info(buf, point, 0); + seq_puts(m, buf); + break; + + case IPIPE_TRACE_EVENT: + __ipipe_get_event_date(buf, print_path, point); + seq_puts(m, buf); + break; + + default: + seq_printf(m, "0x%08lx ", point->v); + } + + time = __ipipe_signed_tsc2us(point->timestamp - + print_path->point[print_path->begin].timestamp); + seq_printf(m, "%5ld", time); + + __ipipe_print_delay(m, point); + __ipipe_print_symname(m, point->eip); + seq_puts(m, " ("); + __ipipe_print_symname(m, point->parent_eip); + seq_puts(m, ")\n"); + + return 0; +} + +static struct seq_operations __ipipe_max_ptrace_ops = { + .start = __ipipe_max_prtrace_start, + .next = __ipipe_prtrace_next, + .stop = __ipipe_prtrace_stop, + .show = __ipipe_prtrace_show +}; + +static int __ipipe_max_prtrace_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &__ipipe_max_ptrace_ops); +} + +static ssize_t +__ipipe_max_reset(struct file *file, const char __user *pbuffer, + size_t count, loff_t *data) +{ + mutex_lock(&out_mutex); + ipipe_trace_max_reset(); + mutex_unlock(&out_mutex); + + return count; +} + +struct file_operations __ipipe_max_prtrace_fops = { + .open = __ipipe_max_prtrace_open, + .read = seq_read, + .write = __ipipe_max_reset, + .llseek = seq_lseek, + .release = seq_release, +}; + +static void *__ipipe_frozen_prtrace_start(struct seq_file *m, loff_t *pos) +{ + loff_t n = *pos; + + mutex_lock(&out_mutex); + + if (!n) { + struct ipipe_trace_path *tp; + int cpu; + unsigned long flags; + + /* protect against max_path/frozen_path updates while we + * haven't locked our target path, also avoid recursively + * taking global_path_lock from NMI context */ + flags = __ipipe_global_path_lock(); + + /* find the first of all per-cpu frozen paths */ + print_path = NULL; + for_each_online_cpu(cpu) { + tp = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)]; + if (tp->end >= 0) { + print_path = tp; + break; + } + } + if (print_path) + print_path->dump_lock = 1; + + __ipipe_global_path_unlock(flags); + + if (!print_path) + return NULL; + + /* back- and post-tracing length, post-trace length was frozen + in __ipipe_trace, back-trace may have to be reduced due to + buffer overrun */ + print_pre_trace = back_trace-1; /* substract freeze point */ + print_post_trace = WRAP_POINT_NO(print_path->trace_pos - + print_path->end - 1); + if (1+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1) + print_pre_trace = IPIPE_TRACE_POINTS - 2 - + print_post_trace; + + seq_printf(m, "I-pipe frozen back-tracing service on %s/ipipe-%s\n" + "------------------------------------------------------" + "------\n", + UTS_RELEASE, IPIPE_ARCH_STRING); + seq_printf(m, "CPU: %d, Freeze: %lld cycles, Trace Points: %d (+%d)\n", + cpu, print_path->point[print_path->begin].timestamp, + print_pre_trace+1, print_post_trace); + __ipipe_print_headline(m); + } + + /* check if we are inside the trace range */ + if (n >= print_pre_trace + 1 + print_post_trace) + return NULL; + + /* return the next point to be shown */ + return &print_path->point[WRAP_POINT_NO(print_path->begin- + print_pre_trace+n)]; +} + +static struct seq_operations __ipipe_frozen_ptrace_ops = { + .start = __ipipe_frozen_prtrace_start, + .next = __ipipe_prtrace_next, + .stop = __ipipe_prtrace_stop, + .show = __ipipe_prtrace_show +}; + +static int __ipipe_frozen_prtrace_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &__ipipe_frozen_ptrace_ops); +} + +static ssize_t +__ipipe_frozen_ctrl(struct file *file, const char __user *pbuffer, + size_t count, loff_t *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, pbuffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + ipipe_trace_frozen_reset(); + if (val > 0) + ipipe_trace_freeze(-1); + mutex_unlock(&out_mutex); + + return count; +} + +struct file_operations __ipipe_frozen_prtrace_fops = { + .open = __ipipe_frozen_prtrace_open, + .read = seq_read, + .write = __ipipe_frozen_ctrl, + .llseek = seq_lseek, + .release = seq_release, +}; + +static int __ipipe_rd_proc_val(char *page, char **start, off_t off, + int count, int *eof, void *data) +{ + int len; + + len = sprintf(page, "%u\n", *(int *)data); + len -= off; + if (len <= off + count) + *eof = 1; + *start = page + off; + if (len > count) + len = count; + if (len < 0) + len = 0; + + return len; +} + +static int __ipipe_wr_proc_val(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, buffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + *(int *)data = val; + mutex_unlock(&out_mutex); + + return count; +} + +static int __ipipe_rd_trigger(char *page, char **start, off_t off, int count, + int *eof, void *data) +{ + int len; + + if (!trigger_begin) + return 0; + + len = sprint_symbol(page, trigger_begin); + page[len++] = '\n'; + + len -= off; + if (len <= off + count) + *eof = 1; + *start = page + off; + if (len > count) + len = count; + if (len < 0) + len = 0; + + return len; +} + +static int __ipipe_wr_trigger(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char buf[KSYM_SYMBOL_LEN]; + unsigned long begin, end; + + if (count > sizeof(buf) - 1) + count = sizeof(buf) - 1; + if (copy_from_user(buf, buffer, count)) + return -EFAULT; + buf[count] = 0; + if (buf[count-1] == '\n') + buf[count-1] = 0; + + begin = kallsyms_lookup_name(buf); + if (!begin || !kallsyms_lookup_size_offset(begin, &end, NULL)) + return -ENOENT; + end += begin - 1; + + mutex_lock(&out_mutex); + /* invalidate the current range before setting a new one */ + trigger_end = 0; + wmb(); + ipipe_trace_frozen_reset(); + + /* set new range */ + trigger_begin = begin; + wmb(); + trigger_end = end; + mutex_unlock(&out_mutex); + + return count; +} + +#ifdef CONFIG_IPIPE_TRACE_MCOUNT +static void notrace +ipipe_trace_function(unsigned long ip, unsigned long parent_ip) +{ + if (!ipipe_trace_enable) + return; + __ipipe_trace(IPIPE_TRACE_FUNC, ip, parent_ip, 0); +} + +static struct ftrace_ops ipipe_trace_ops = { + .func = ipipe_trace_function +}; + +static int __ipipe_wr_enable(struct file *file, const char __user *buffer, + unsigned long count, void *data) +{ + char *end, buf[16]; + int val; + int n; + + n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count; + + if (copy_from_user(buf, buffer, n)) + return -EFAULT; + + buf[n] = '\0'; + val = simple_strtol(buf, &end, 0); + + if (((*end != '\0') && !isspace(*end)) || (val < 0)) + return -EINVAL; + + mutex_lock(&out_mutex); + + if (ipipe_trace_enable) { + if (!val) + unregister_ftrace_function(&ipipe_trace_ops); + } else if (val) + register_ftrace_function(&ipipe_trace_ops); + + ipipe_trace_enable = val; + + mutex_unlock(&out_mutex); + + return count; +} +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ + +extern struct proc_dir_entry *ipipe_proc_root; + +static struct proc_dir_entry * __init +__ipipe_create_trace_proc_val(struct proc_dir_entry *trace_dir, + const char *name, int *value_ptr) +{ + struct proc_dir_entry *entry; + + entry = create_proc_entry(name, 0644, trace_dir); + if (entry) { + entry->data = value_ptr; + entry->read_proc = __ipipe_rd_proc_val; + entry->write_proc = __ipipe_wr_proc_val; + } + return entry; +} + +void __init __ipipe_init_tracer(void) +{ + struct proc_dir_entry *trace_dir; + struct proc_dir_entry *entry; + unsigned long long start, end, min = ULLONG_MAX; + int i; +#ifdef CONFIG_IPIPE_TRACE_VMALLOC + int cpu, path; + + for_each_possible_cpu(cpu) { + struct ipipe_trace_path *tp_buf; + + tp_buf = vmalloc_node(sizeof(struct ipipe_trace_path) * + IPIPE_TRACE_PATHS, cpu_to_node(cpu)); + if (!tp_buf) { + printk(KERN_ERR "I-pipe: " + "insufficient memory for trace buffer.\n"); + return; + } + memset(tp_buf, 0, + sizeof(struct ipipe_trace_path) * IPIPE_TRACE_PATHS); + for (path = 0; path < IPIPE_TRACE_PATHS; path++) { + tp_buf[path].begin = -1; + tp_buf[path].end = -1; + } + per_cpu(trace_path, cpu) = tp_buf; + } +#endif /* CONFIG_IPIPE_TRACE_VMALLOC */ + + /* Calculate minimum overhead of __ipipe_trace() */ + local_irq_disable_hw(); + for (i = 0; i < 100; i++) { + ipipe_read_tsc(start); + __ipipe_trace(IPIPE_TRACE_FUNC, __BUILTIN_RETURN_ADDRESS0, + __BUILTIN_RETURN_ADDRESS1, 0); + ipipe_read_tsc(end); + + end -= start; + if (end < min) + min = end; + } + local_irq_enable_hw(); + trace_overhead = ipipe_tsc2ns(min); + +#ifdef CONFIG_IPIPE_TRACE_ENABLE + ipipe_trace_enable = 1; +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + register_ftrace_function(&ipipe_trace_ops); +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ +#endif /* CONFIG_IPIPE_TRACE_ENABLE */ + + trace_dir = create_proc_entry("trace", S_IFDIR, ipipe_proc_root); + + entry = create_proc_entry("max", 0644, trace_dir); + if (entry) + entry->proc_fops = &__ipipe_max_prtrace_fops; + + entry = create_proc_entry("frozen", 0644, trace_dir); + if (entry) + entry->proc_fops = &__ipipe_frozen_prtrace_fops; + + entry = create_proc_entry("trigger", 0644, trace_dir); + if (entry) { + entry->read_proc = __ipipe_rd_trigger; + entry->write_proc = __ipipe_wr_trigger; + } + + __ipipe_create_trace_proc_val(trace_dir, "pre_trace_points", + &pre_trace); + __ipipe_create_trace_proc_val(trace_dir, "post_trace_points", + &post_trace); + __ipipe_create_trace_proc_val(trace_dir, "back_trace_points", + &back_trace); + __ipipe_create_trace_proc_val(trace_dir, "verbose", + &verbose_trace); + entry = __ipipe_create_trace_proc_val(trace_dir, "enable", + &ipipe_trace_enable); +#ifdef CONFIG_IPIPE_TRACE_MCOUNT + if (entry) + entry->write_proc = __ipipe_wr_enable; +#endif /* CONFIG_IPIPE_TRACE_MCOUNT */ +} diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c index ecc3fa2..e3e4263 100644 --- a/kernel/irq/chip.c +++ b/kernel/irq/chip.c @@ -15,6 +15,7 @@ #include <linux/module.h> #include <linux/interrupt.h> #include <linux/kernel_stat.h> +#include <linux/ipipe.h> #include "internals.h" @@ -425,7 +426,9 @@ handle_level_irq(unsigned int irq, struct irq_desc *desc) irqreturn_t action_ret; raw_spin_lock(&desc->lock); +#ifndef CONFIG_IPIPE mask_ack_irq(desc, irq); +#endif if (unlikely(desc->status & IRQ_INPROGRESS)) goto out_unlock; @@ -505,8 +508,13 @@ handle_fasteoi_irq(unsigned int irq, struct irq_desc *desc) raw_spin_lock(&desc->lock); desc->status &= ~IRQ_INPROGRESS; +#ifdef CONFIG_IPIPE + desc->chip->unmask(irq); +out: +#else out: desc->chip->eoi(irq); +#endif raw_spin_unlock(&desc->lock); } @@ -548,8 +556,10 @@ handle_edge_irq(unsigned int irq, struct irq_desc *desc) kstat_incr_irqs_this_cpu(irq, desc); /* Start handling the irq */ +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* Mark the IRQ currently in progress.*/ desc->status |= IRQ_INPROGRESS; @@ -603,8 +613,10 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc) kstat_incr_irqs_this_cpu(irq, desc); +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* CONFIG_IPIPE */ action_ret = handle_IRQ_event(irq, desc->action); if (!noirqdebug) @@ -614,6 +626,134 @@ handle_percpu_irq(unsigned int irq, struct irq_desc *desc) desc->chip->eoi(irq); } +#ifdef CONFIG_IPIPE + +void __ipipe_ack_simple_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_end_simple_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_ack_level_irq(unsigned irq, struct irq_desc *desc) +{ + mask_ack_irq(desc, irq); +} + +void __ipipe_end_level_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->unmask) + desc->chip->unmask(irq); +} + +void __ipipe_ack_fasteoi_irq(unsigned irq, struct irq_desc *desc) +{ + desc->chip->eoi(irq); +} + +void __ipipe_end_fasteoi_irq(unsigned irq, struct irq_desc *desc) +{ + /* + * Non-requestable IRQs should not be masked in EOI handler. + */ + if (!(desc->status & IRQ_NOREQUEST)) + desc->chip->unmask(irq); +} + +void __ipipe_ack_edge_irq(unsigned irq, struct irq_desc *desc) +{ + desc->chip->ack(irq); +} + +void __ipipe_ack_percpu_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->ack) + desc->chip->ack(irq); +} + +void __ipipe_end_percpu_irq(unsigned irq, struct irq_desc *desc) +{ + if (desc->chip->eoi) + desc->chip->eoi(irq); +} + +void __ipipe_end_edge_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_ack_bad_irq(unsigned irq, struct irq_desc *desc) +{ + static int done; + + handle_bad_irq(irq, desc); + + if (!done) { + printk(KERN_WARNING "%s: unknown flow handler for IRQ %d\n", + __FUNCTION__, irq); + done = 1; + } +} + +void __ipipe_noack_irq(unsigned irq, struct irq_desc *desc) +{ +} + +void __ipipe_noend_irq(unsigned irq, struct irq_desc *desc) +{ +} + +irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, int is_chained) +{ + if (unlikely(handle == NULL)) { + desc->ipipe_ack = &__ipipe_ack_bad_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } else { + if (is_chained) { + desc->ipipe_ack = handle; + desc->ipipe_end = &__ipipe_noend_irq; + handle = __ipipe_noack_irq; + } else if (handle == &handle_simple_irq) { + desc->ipipe_ack = &__ipipe_ack_simple_irq; + desc->ipipe_end = &__ipipe_end_simple_irq; + } else if (handle == &handle_level_irq) { + desc->ipipe_ack = &__ipipe_ack_level_irq; + desc->ipipe_end = &__ipipe_end_level_irq; + } else if (handle == &handle_edge_irq) { + desc->ipipe_ack = &__ipipe_ack_edge_irq; + desc->ipipe_end = &__ipipe_end_edge_irq; + } else if (handle == &handle_fasteoi_irq) { + desc->ipipe_ack = &__ipipe_ack_fasteoi_irq; + desc->ipipe_end = &__ipipe_end_fasteoi_irq; + } else if (handle == &handle_percpu_irq) { + desc->ipipe_ack = &__ipipe_ack_percpu_irq; + desc->ipipe_end = &__ipipe_end_percpu_irq; + } else if (desc->chip == &no_irq_chip) { + desc->ipipe_ack = &__ipipe_noack_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } else { + desc->ipipe_ack = &__ipipe_ack_bad_irq; + desc->ipipe_end = &__ipipe_noend_irq; + } + } + + /* Suppress intermediate trampoline routine. */ + ipipe_root_domain->irqs[desc->irq].acknowledge = desc->ipipe_ack; + + return handle; +} + +#else /* !CONFIG_IPIPE */ + +irq_flow_handler_t +__fixup_irq_handler(struct irq_desc *desc, irq_flow_handler_t handle, int is_chained) +{ + return handle; +} + +#endif /* !CONFIG_IPIPE */ + void __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, const char *name) @@ -645,6 +785,8 @@ __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained, chip_bus_lock(irq, desc); raw_spin_lock_irqsave(&desc->lock, flags); + handle = __fixup_irq_handler(desc, handle, is_chained); + /* Uninstall? */ if (handle == handle_bad_irq) { if (desc->chip != &no_irq_chip) diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c index 814940e..4c2e41d 100644 --- a/kernel/irq/handle.c +++ b/kernel/irq/handle.c @@ -462,8 +462,10 @@ unsigned int __do_IRQ(unsigned int irq) /* * No locking required for CPU-local interrupts: */ +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif if (likely(!(desc->status & IRQ_DISABLED))) { action_ret = handle_IRQ_event(irq, desc->action); if (!noirqdebug) @@ -474,8 +476,10 @@ unsigned int __do_IRQ(unsigned int irq) } raw_spin_lock(&desc->lock); +#ifndef CONFIG_IPIPE if (desc->chip->ack) desc->chip->ack(irq); +#endif /* * REPLAY is when Linux resends an IRQ that was dropped earlier * WAITING is used by probe to mark irqs that are being tested diff --git a/kernel/lockdep.c b/kernel/lockdep.c index c62ec14..6d41b73 100644 --- a/kernel/lockdep.c +++ b/kernel/lockdep.c @@ -2327,7 +2327,7 @@ void trace_hardirqs_on_caller(unsigned long ip) /* we'll do an OFF -> ON transition: */ curr->hardirqs_enabled = 1; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (DEBUG_LOCKS_WARN_ON(current->hardirq_context)) return; @@ -2370,7 +2370,7 @@ void trace_hardirqs_off_caller(unsigned long ip) if (unlikely(!debug_locks || current->lockdep_recursion)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->hardirqs_enabled) { @@ -2402,7 +2402,7 @@ void trace_softirqs_on(unsigned long ip) if (unlikely(!debug_locks)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->softirqs_enabled) { @@ -2436,7 +2436,7 @@ void trace_softirqs_off(unsigned long ip) if (unlikely(!debug_locks)) return; - if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) + if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw())) return; if (curr->softirqs_enabled) { diff --git a/kernel/panic.c b/kernel/panic.c index c787333..ba50967 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -23,6 +23,7 @@ #include <linux/init.h> #include <linux/nmi.h> #include <linux/dmi.h> +#include <linux/ipipe_trace.h> int panic_on_oops; static unsigned long tainted_mask; @@ -307,6 +308,8 @@ void oops_enter(void) { tracing_off(); /* can't trust the integrity of the kernel anymore: */ + ipipe_trace_panic_freeze(); + ipipe_disable_context_check(ipipe_processor_id()); debug_locks_off(); do_oops_enter_exit(); } diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c index bbfe472..be44776 100644 --- a/kernel/power/hibernate.c +++ b/kernel/power/hibernate.c @@ -268,6 +268,7 @@ static int create_image(int platform_mode) goto Enable_cpus; local_irq_disable(); + local_irq_disable_hw_cond(); error = sysdev_suspend(PMSG_FREEZE); if (error) { @@ -297,6 +298,7 @@ static int create_image(int platform_mode) */ Enable_irqs: + local_irq_enable_hw_cond(); local_irq_enable(); Enable_cpus: @@ -389,6 +391,7 @@ static int resume_target_kernel(bool platform_mode) goto Enable_cpus; local_irq_disable(); + local_irq_disable_hw_cond(); error = sysdev_suspend(PMSG_QUIESCE); if (error) @@ -420,6 +423,7 @@ static int resume_target_kernel(bool platform_mode) sysdev_resume(); Enable_irqs: + local_irq_enable_hw_cond(); local_irq_enable(); Enable_cpus: @@ -501,6 +505,7 @@ int hibernation_platform_enter(void) goto Platform_finish; local_irq_disable(); + local_irq_disable_hw_cond(); sysdev_suspend(PMSG_HIBERNATE); hibernation_ops->enter(); /* We should never get here */ diff --git a/kernel/printk.c b/kernel/printk.c index 1751c45..a1e027a 100644 --- a/kernel/printk.c +++ b/kernel/printk.c @@ -566,6 +566,41 @@ static int have_callable_console(void) return 0; } +#ifdef CONFIG_IPIPE + +static IPIPE_DEFINE_SPINLOCK(__ipipe_printk_lock); + +static int __ipipe_printk_fill; + +static char __ipipe_printk_buf[__LOG_BUF_LEN]; + +void __ipipe_flush_printk (unsigned virq, void *cookie) +{ + char *p = __ipipe_printk_buf; + int len, lmax, out = 0; + unsigned long flags; + + goto start; + + do { + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); + start: + lmax = __ipipe_printk_fill; + while (out < lmax) { + len = strlen(p) + 1; + printk("%s",p); + p += len; + out += len; + } + spin_lock_irqsave(&__ipipe_printk_lock, flags); + } + while (__ipipe_printk_fill != lmax); + + __ipipe_printk_fill = 0; + + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); +} + /** * printk - print a kernel message * @fmt: format string @@ -590,6 +625,65 @@ static int have_callable_console(void) asmlinkage int printk(const char *fmt, ...) { + int r, fbytes, oldcount; + unsigned long flags; + int sprintk = 1; + int cs = -1; + va_list args; + + va_start(args, fmt); + + local_irq_save_hw(flags); + + if (test_bit(IPIPE_SPRINTK_FLAG, &__ipipe_current_domain->flags) || + oops_in_progress) + cs = ipipe_disable_context_check(ipipe_processor_id()); + else if (__ipipe_current_domain == ipipe_root_domain) { + struct ipipe_domain *dom; + + list_for_each_entry(dom, &__ipipe_pipeline, p_link) { + if (dom == ipipe_root_domain) + break; + if (test_bit(IPIPE_STALL_FLAG, + &ipipe_cpudom_var(dom, status))) + sprintk = 0; + } + } else + sprintk = 0; + + local_irq_restore_hw(flags); + + if (sprintk) { + r = vprintk(fmt, args); + if (cs != -1) + ipipe_restore_context_check(ipipe_processor_id(), cs); + goto out; + } + + spin_lock_irqsave(&__ipipe_printk_lock, flags); + + oldcount = __ipipe_printk_fill; + fbytes = __LOG_BUF_LEN - oldcount; + + if (fbytes > 1) { + r = vscnprintf(__ipipe_printk_buf + __ipipe_printk_fill, + fbytes, fmt, args) + 1; /* account for the null byte */ + __ipipe_printk_fill += r; + } else + r = 0; + + spin_unlock_irqrestore(&__ipipe_printk_lock, flags); + + if (oldcount == 0) + ipipe_trigger_irq(__ipipe_printk_virq); +out: + va_end(args); + + return r; +} +#else /* !CONFIG_IPIPE */ +asmlinkage int printk(const char *fmt, ...) +{ va_list args; int r; @@ -599,6 +693,7 @@ asmlinkage int printk(const char *fmt, ...) return r; } +#endif /* CONFIG_IPIPE */ /* cpu currently holding logbuf_lock */ static volatile unsigned int printk_cpu = UINT_MAX; diff --git a/kernel/sched.c b/kernel/sched.c index 3a8fb30..cdb5bfd 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -2368,6 +2368,7 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) { int cpu, orig_cpu, this_cpu, success = 0; + unsigned int old_state; unsigned long flags; struct rq *rq, *orig_rq; @@ -2379,7 +2380,9 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state, smp_wmb(); rq = orig_rq = task_rq_lock(p, &flags); update_rq_clock(rq); - if (!(p->state & state)) + old_state = p->state; + if (!(old_state & state) || + (old_state & (TASK_NOWAKEUP|TASK_ATOMICSWITCH))) goto out; if (p->se.on_rq) @@ -2869,13 +2872,15 @@ asmlinkage void schedule_tail(struct task_struct *prev) #endif if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); + + ipipe_init_notify(current); } /* * context_switch - switch to the new MM and the new * thread's register state. */ -static inline void +static inline int context_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { @@ -2917,12 +2922,23 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_to(prev, next, prev); barrier(); + +#ifdef CONFIG_IPIPE_DELAYED_ATOMICSW + current->state &= ~TASK_ATOMICSWITCH; +#else + prev->state &= ~TASK_ATOMICSWITCH; +#endif + if (task_hijacked(prev)) + return 1; + /* * this_rq must be evaluated again because prev may have moved * CPUs since it called schedule(), thus the 'rq' on its stack * frame will be invalid. */ finish_task_switch(this_rq(), prev); + + return 0; } /* @@ -5332,6 +5348,7 @@ notrace unsigned long get_parent_ip(unsigned long addr) void __kprobes add_preempt_count(int val) { + ipipe_check_context(ipipe_root_domain); #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? @@ -5354,6 +5371,7 @@ EXPORT_SYMBOL(add_preempt_count); void __kprobes sub_preempt_count(int val) { + ipipe_check_context(ipipe_root_domain); #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? @@ -5402,6 +5420,7 @@ static noinline void __schedule_bug(struct task_struct *prev) */ static inline void schedule_debug(struct task_struct *prev) { + ipipe_check_context(ipipe_root_domain); /* * Test if we are atomic. Since do_exit() needs to call into * schedule() atomically, we ignore that path for now. @@ -5478,7 +5497,7 @@ pick_next_task(struct rq *rq) /* * schedule() is the main scheduler function. */ -asmlinkage void __sched schedule(void) +asmlinkage int __sched schedule(void) { struct task_struct *prev, *next; unsigned long *switch_count; @@ -5492,6 +5511,9 @@ need_resched: rcu_sched_qs(cpu); prev = rq->curr; switch_count = &prev->nivcsw; + if (unlikely(prev->state & TASK_ATOMICSWITCH)) + /* Pop one disable level -- one still remains. */ + preempt_enable(); release_kernel_lock(prev); need_resched_nonpreemptible: @@ -5529,15 +5551,18 @@ need_resched_nonpreemptible: rq->curr = next; ++*switch_count; - context_switch(rq, prev, next); /* unlocks the rq */ + if (context_switch(rq, prev, next)) /* unlocks the rq */ + return 1; /* task hijacked by higher domain */ /* * the context switch might have flipped the stack from under * us, hence refresh the local variables. */ cpu = smp_processor_id(); rq = cpu_rq(cpu); - } else + } else { + prev->state &= ~TASK_ATOMICSWITCH; raw_spin_unlock_irq(&rq->lock); + } post_schedule(rq); @@ -5550,6 +5575,8 @@ need_resched_nonpreemptible: preempt_enable_no_resched(); if (need_resched()) goto need_resched; + + return 0; } EXPORT_SYMBOL(schedule); @@ -5633,7 +5660,8 @@ asmlinkage void __sched preempt_schedule(void) do { add_preempt_count(PREEMPT_ACTIVE); - schedule(); + if (schedule()) + return; sub_preempt_count(PREEMPT_ACTIVE); /* @@ -6396,6 +6424,7 @@ recheck: oldprio = p->prio; __setscheduler(rq, p, policy, param->sched_priority); + ipipe_setsched_notify(p); if (running) p->sched_class->set_curr_task(rq); @@ -7048,6 +7077,7 @@ void __cpuinit init_idle(struct task_struct *idle, int cpu) #else task_thread_info(idle)->preempt_count = 0; #endif + ipipe_check_context(ipipe_root_domain); /* * The idle tasks have their own, simple scheduling class: */ @@ -11039,3 +11069,64 @@ void synchronize_sched_expedited(void) EXPORT_SYMBOL_GPL(synchronize_sched_expedited); #endif /* #else #ifndef CONFIG_SMP */ + +#ifdef CONFIG_IPIPE + +int ipipe_setscheduler_root(struct task_struct *p, int policy, int prio) +{ + const struct sched_class *prev_class = p->sched_class; + int oldprio, on_rq, running; + unsigned long flags; + struct rq *rq; + + raw_spin_lock_irqsave(&p->pi_lock, flags); + rq = __task_rq_lock(p); + update_rq_clock(rq); + on_rq = p->se.on_rq; + running = task_current(rq, p); + if (on_rq) + deactivate_task(rq, p, 0); + if (running) + p->sched_class->put_prev_task(rq, p); + + p->sched_reset_on_fork = 0; + + oldprio = p->prio; + __setscheduler(rq, p, policy, prio); + ipipe_setsched_notify(p); + + if (running) + p->sched_class->set_curr_task(rq); + if (on_rq) { + activate_task(rq, p, 0); + + check_class_changed(rq, p, prev_class, oldprio, running); + } + __task_rq_unlock(rq); + raw_spin_unlock_irqrestore(&p->pi_lock, flags); + + rt_mutex_adjust_pi(p); + + return 0; +} +EXPORT_SYMBOL_GPL(ipipe_setscheduler_root); + +int ipipe_reenter_root(struct task_struct *prev, int policy, int prio) +{ + struct rq *rq = this_rq(); + + finish_task_switch(rq, prev); + + post_schedule(rq); + + (void)reacquire_kernel_lock(current); + preempt_enable_no_resched(); + + if (current->policy != policy || current->rt_priority != prio) + return ipipe_setscheduler_root(current, policy, prio); + + return 0; +} +EXPORT_SYMBOL_GPL(ipipe_reenter_root); + +#endif /* CONFIG_IPIPE */ diff --git a/kernel/signal.c b/kernel/signal.c index 934ae5e..7181e5f 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -541,6 +541,7 @@ void signal_wake_up(struct task_struct *t, int resume) unsigned int mask; set_tsk_thread_flag(t, TIF_SIGPENDING); + ipipe_sigwake_notify(t); /* TIF_SIGPENDING must be set first. */ /* * For SIGKILL, we want to wake it up in the stopped/traced/killable diff --git a/kernel/spinlock.c b/kernel/spinlock.c index be6517f..862aed4 100644 --- a/kernel/spinlock.c +++ b/kernel/spinlock.c @@ -26,7 +26,9 @@ * even on CONFIG_PREEMPT, because lockdep assumes that interrupts are * not re-enabled during lock-acquire (which the preempt-spin-ops do): */ -#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) +#if !defined(CONFIG_GENERIC_LOCKBREAK) || \ + defined(CONFIG_DEBUG_LOCK_ALLOC) || \ + defined(CONFIG_IPIPE) /* * The __lock_function inlines are taken from * include/linux/spinlock_api_smp.h diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c index b6b898d..75c2031 100644 --- a/kernel/time/tick-common.c +++ b/kernel/time/tick-common.c @@ -69,7 +69,7 @@ static void tick_periodic(int cpu) write_sequnlock(&xtime_lock); } - update_process_times(user_mode(get_irq_regs())); + update_root_process_times(get_irq_regs()); profile_tick(CPU_PROFILING); } @@ -177,6 +177,10 @@ static void tick_setup_device(struct tick_device *td, td->evtdev = newdev; + /* I-pipe: derive global tick IRQ from CPU 0 */ + if (cpu == 0) + ipipe_update_tick_evtdev(newdev); + /* * When the device is not per cpu, pin the interrupt to the * current cpu: diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index f992762..a9314ea 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -551,7 +551,7 @@ static void tick_nohz_handler(struct clock_event_device *dev) ts->idle_jiffies++; } - update_process_times(user_mode(regs)); + update_root_process_times(regs); profile_tick(CPU_PROFILING); while (tick_nohz_reprogram(ts, now)) { @@ -711,7 +711,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer) touch_softlockup_watchdog(); ts->idle_jiffies++; } - update_process_times(user_mode(regs)); + update_root_process_times(regs); profile_tick(CPU_PROFILING); } diff --git a/kernel/timer.c b/kernel/timer.c index c61a794..fb57c0c 100644 --- a/kernel/timer.c +++ b/kernel/timer.c @@ -1203,6 +1203,25 @@ void update_process_times(int user_tick) run_posix_cpu_timers(p); } +#ifdef CONFIG_IPIPE + +void update_root_process_times(struct pt_regs *regs) +{ + int cpu, user_tick = user_mode(regs); + + if (__ipipe_root_tick_p(regs)) { + update_process_times(user_tick); + return; + } + + run_local_timers(); + cpu = smp_processor_id(); + rcu_check_callbacks(cpu, user_tick); + run_posix_cpu_timers(current); +} + +#endif + /* * This function runs timers and the timer-tq in bottom half context. */ diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 60e2ce0..5b7a125 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -468,6 +468,7 @@ config DYNAMIC_FTRACE bool "enable/disable ftrace tracepoints dynamically" depends on FUNCTION_TRACER depends on HAVE_DYNAMIC_FTRACE + depends on !IPIPE_TRACE_MCOUNT default y help This option will modify all the calls to ftrace dynamically diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 1e6640f..5a3aa6b 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -28,6 +28,7 @@ #include <linux/ctype.h> #include <linux/list.h> #include <linux/hash.h> +#include <linux/ipipe.h> #include <trace/events/sched.h> @@ -1151,6 +1152,9 @@ static int __ftrace_modify_code(void *data) static void ftrace_run_update_code(int command) { +#ifdef CONFIG_IPIPE + unsigned long flags; +#endif /* CONFIG_IPIPE */ int ret; ret = ftrace_arch_code_modify_prepare(); @@ -1158,7 +1162,13 @@ static void ftrace_run_update_code(int command) if (ret) return; +#ifdef CONFIG_IPIPE + flags = ipipe_critical_enter(NULL); + __ftrace_modify_code(&command); + ipipe_critical_exit(flags); +#else /* !CONFIG_IPIPE */ stop_machine(__ftrace_modify_code, &command, NULL); +#endif /* !CONFIG_IPIPE */ ret = ftrace_arch_code_modify_post_process(); FTRACE_WARN_ON(ret); @@ -2663,9 +2673,9 @@ static int ftrace_process_locs(struct module *mod, } /* disable interrupts to prevent kstop machine */ - local_irq_save(flags); + local_irq_save_hw_notrace(flags); ftrace_update_code(mod); - local_irq_restore(flags); + local_irq_restore_hw_notrace(flags); mutex_unlock(&ftrace_lock); return 0; @@ -2744,9 +2754,9 @@ void __init ftrace_init(void) /* Keep the ftrace pointer to the stub */ addr = (unsigned long)ftrace_stub; - local_irq_save(flags); + local_irq_save_hw_notrace(flags); ftrace_dyn_arch_init(&addr); - local_irq_restore(flags); + local_irq_restore_hw_notrace(flags); /* ftrace_dyn_arch_init places the return code in addr */ if (addr) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 25c3ed5..6b9a5d0 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -136,6 +136,8 @@ config DEBUG_SECTION_MISMATCH - Enable verbose reporting from modpost to help solving the section mismatches reported. +source "kernel/ipipe/Kconfig.debug" + config DEBUG_KERNEL bool "Kernel debugging" help diff --git a/lib/bust_spinlocks.c b/lib/bust_spinlocks.c index 9681d54..2dba50c 100644 --- a/lib/bust_spinlocks.c +++ b/lib/bust_spinlocks.c @@ -13,6 +13,7 @@ #include <linux/wait.h> #include <linux/vt_kern.h> #include <linux/console.h> +#include <linux/ipipe_trace.h> void __attribute__((weak)) bust_spinlocks(int yes) @@ -24,6 +25,7 @@ void __attribute__((weak)) bust_spinlocks(int yes) unblank_screen(); #endif console_unblank(); + ipipe_trace_panic_dump(); if (--oops_in_progress == 0) wake_up_klogd(); } diff --git a/lib/ioremap.c b/lib/ioremap.c index 14c6078..a275469 100644 --- a/lib/ioremap.c +++ b/lib/ioremap.c @@ -85,8 +85,8 @@ int ioremap_page_range(unsigned long addr, if (err) break; } while (pgd++, addr = next, addr != end); - - flush_cache_vmap(start, end); + __ipipe_pin_range_globally(start, end); + flush_cache_vmap(start, end); return err; } diff --git a/lib/smp_processor_id.c b/lib/smp_processor_id.c index 4689cb0..3d12764 100644 --- a/lib/smp_processor_id.c +++ b/lib/smp_processor_id.c @@ -12,10 +12,13 @@ notrace unsigned int debug_smp_processor_id(void) unsigned long preempt_count = preempt_count(); int this_cpu = raw_smp_processor_id(); + if (!ipipe_root_domain_p) + goto out; + if (likely(preempt_count)) goto out; - if (irqs_disabled()) + if (irqs_disabled() || irqs_disabled_hw()) goto out; /* diff --git a/mm/memory.c b/mm/memory.c index 09e4b1b..bef0089 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -56,6 +56,7 @@ #include <linux/kallsyms.h> #include <linux/swapops.h> #include <linux/elf.h> +#include <linux/vmalloc.h> #include <asm/io.h> #include <asm/pgalloc.h> @@ -566,6 +567,32 @@ out: return pfn_to_page(pfn); } +static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) +{ + /* + * If the source page was a PFN mapping, we don't have + * a "struct page" for it. We do a best-effort copy by + * just copying from the original user address. If that + * fails, we just zero-fill it. Live with it. + */ + if (unlikely(!src)) { + void *kaddr = kmap_atomic(dst, KM_USER0); + void __user *uaddr = (void __user *)(va & PAGE_MASK); + + /* + * This really shouldn't fail, because the page is there + * in the page tables. But it might just be unreadable, + * in which case we just give up and fill the result with + * zeroes. + */ + if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) + memset(kaddr, 0, PAGE_SIZE); + kunmap_atomic(kaddr, KM_USER0); + flush_dcache_page(dst); + } else + copy_user_highpage(dst, src, va, vma); +} + /* * copy one vm_area from one task to the other. Assumes the page tables * already present in the new task to be cleared in the whole range @@ -574,8 +601,8 @@ out: static inline unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, - pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, - unsigned long addr, int *rss) + pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma, + unsigned long addr, int *rss, struct page *uncow_page) { unsigned long vm_flags = vma->vm_flags; pte_t pte = *src_pte; @@ -616,6 +643,21 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * in the parent and the child */ if (is_cow_mapping(vm_flags)) { +#ifdef CONFIG_IPIPE + if (uncow_page) { + struct page *old_page = vm_normal_page(vma, addr, pte); + cow_user_page(uncow_page, old_page, addr, vma); + pte = mk_pte(uncow_page, vma->vm_page_prot); + + if (vm_flags & VM_SHARED) + pte = pte_mkclean(pte); + pte = pte_mkold(pte); + + page_add_new_anon_rmap(uncow_page, vma, addr); + rss[!!PageAnon(uncow_page)]++; + goto out_set_pte; + } +#endif /* CONFIG_IPIPE */ ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -650,12 +692,26 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, int progress = 0; int rss[2]; swp_entry_t entry = (swp_entry_t){0}; - + struct page *uncow_page = NULL; +#ifdef CONFIG_IPIPE + int do_cow_break = 0; +again: + if (do_cow_break) { + uncow_page = alloc_page_vma(GFP_HIGHUSER, vma, addr); + if (uncow_page == NULL) + return -ENOMEM; + do_cow_break = 0; + } +#else again: rss[1] = rss[0] = 0; +#endif dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); - if (!dst_pte) + if (!dst_pte) { + if (uncow_page) + page_cache_release(uncow_page); return -ENOMEM; + } src_pte = pte_offset_map_nested(src_pmd, addr); src_ptl = pte_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -678,8 +734,25 @@ again: progress++; continue; } +#ifdef CONFIG_IPIPE + if (likely(uncow_page == NULL) && likely(pte_present(*src_pte))) { + if (is_cow_mapping(vma->vm_flags) && + test_bit(MMF_VM_PINNED, &src_mm->flags) && + ((vma->vm_flags|src_mm->def_flags) & VM_LOCKED)) { + arch_leave_lazy_mmu_mode(); + spin_unlock(src_ptl); + pte_unmap_nested(src_pte); + add_mm_rss(dst_mm, rss[0], rss[1]); + pte_unmap_unlock(dst_pte, dst_ptl); + cond_resched(); + do_cow_break = 1; + goto again; + } + } +#endif entry.val = copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, - vma, addr, rss); + vma, addr, rss, uncow_page); + uncow_page = NULL; if (entry.val) break; progress += 8; @@ -1956,32 +2029,6 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) return pte; } -static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) -{ - /* - * If the source page was a PFN mapping, we don't have - * a "struct page" for it. We do a best-effort copy by - * just copying from the original user address. If that - * fails, we just zero-fill it. Live with it. - */ - if (unlikely(!src)) { - void *kaddr = kmap_atomic(dst, KM_USER0); - void __user *uaddr = (void __user *)(va & PAGE_MASK); - - /* - * This really shouldn't fail, because the page is there - * in the page tables. But it might just be unreadable, - * in which case we just give up and fill the result with - * zeroes. - */ - if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) - memset(kaddr, 0, PAGE_SIZE); - kunmap_atomic(kaddr, KM_USER0); - flush_dcache_page(dst); - } else - copy_user_highpage(dst, src, va, vma); -} - /* * This routine handles present pages, when users try to write * to a shared page. It is done by copying the page to a new address @@ -3402,3 +3449,111 @@ void might_fault(void) } EXPORT_SYMBOL(might_fault); #endif + +#ifdef CONFIG_IPIPE + +static inline int ipipe_pin_pte_range(struct mm_struct *mm, pmd_t *pmd, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + spinlock_t *ptl; + pte_t *pte; + + do { + pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + continue; + + if (!pte_present(*pte) || pte_write(*pte)) { + pte_unmap_unlock(pte, ptl); + continue; + } + + if (do_wp_page(mm, vma, addr, pte, pmd, ptl, *pte) == VM_FAULT_OOM) + return -ENOMEM; + } while (addr += PAGE_SIZE, addr != end); + return 0; +} + +static inline int ipipe_pin_pmd_range(struct mm_struct *mm, pud_t *pud, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + unsigned long next; + pmd_t *pmd; + + pmd = pmd_offset(pud, addr); + do { + next = pmd_addr_end(addr, end); + if (pmd_none_or_clear_bad(pmd)) + continue; + if (ipipe_pin_pte_range(mm, pmd, vma, addr, next)) + return -ENOMEM; + } while (pmd++, addr = next, addr != end); + return 0; +} + +static inline int ipipe_pin_pud_range(struct mm_struct *mm, pgd_t *pgd, + struct vm_area_struct *vma, + unsigned long addr, unsigned long end) +{ + unsigned long next; + pud_t *pud; + + pud = pud_offset(pgd, addr); + do { + next = pud_addr_end(addr, end); + if (pud_none_or_clear_bad(pud)) + continue; + if (ipipe_pin_pmd_range(mm, pud, vma, addr, next)) + return -ENOMEM; + } while (pud++, addr = next, addr != end); + return 0; +} + +int ipipe_disable_ondemand_mappings(struct task_struct *tsk) +{ + unsigned long addr, next, end; + struct vm_area_struct *vma; + struct mm_struct *mm; + int result = 0; + pgd_t *pgd; + + mm = get_task_mm(tsk); + if (!mm) + return -EPERM; + + down_write(&mm->mmap_sem); + if (test_bit(MMF_VM_PINNED, &mm->flags)) + goto done_mm; + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (!is_cow_mapping(vma->vm_flags) + || !(vma->vm_flags & VM_WRITE)) + continue; + + addr = vma->vm_start; + end = vma->vm_end; + + pgd = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none_or_clear_bad(pgd)) + continue; + if (ipipe_pin_pud_range(mm, pgd, vma, addr, next)) { + result = -ENOMEM; + goto done_mm; + } + } while (pgd++, addr = next, addr != end); + } + set_bit(MMF_VM_PINNED, &mm->flags); + + done_mm: + up_write(&mm->mmap_sem); + mmput(mm); + return result; +} + +EXPORT_SYMBOL(ipipe_disable_ondemand_mappings); + +#endif diff --git a/mm/mmu_context.c b/mm/mmu_context.c index ded9081..cb2ac0e 100644 --- a/mm/mmu_context.c +++ b/mm/mmu_context.c @@ -23,15 +23,18 @@ void use_mm(struct mm_struct *mm) { struct mm_struct *active_mm; struct task_struct *tsk = current; + unsigned long flags; task_lock(tsk); active_mm = tsk->active_mm; + ipipe_mm_switch_protect(flags); if (active_mm != mm) { atomic_inc(&mm->mm_count); tsk->active_mm = mm; } tsk->mm = mm; - switch_mm(active_mm, mm, tsk); + __switch_mm(active_mm, mm, tsk); + ipipe_mm_switch_unprotect(flags); task_unlock(tsk); if (active_mm != mm) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ae00746..ca1597c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -172,6 +172,8 @@ static int vmap_page_range_noflush(unsigned long start, unsigned long end, return err; } while (pgd++, addr = next, addr != end); + __ipipe_pin_range_globally(start, end); + return nr; } [-- Attachment #6: flush.c --] [-- Type: text/x-csrc, Size: 7630 bytes --] /* * linux/arch/arm/mm/flush.c * * Copyright (C) 1995-2002 Russell King * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. */ #include <linux/module.h> #include <linux/mm.h> #include <linux/pagemap.h> #include <asm/cacheflush.h> #include <asm/cachetype.h> #include <asm/smp_plat.h> #include <asm/system.h> #include <asm/tlbflush.h> #include <asm/fcse.h> #include "mm.h" #ifdef CONFIG_CPU_CACHE_VIPT #define ALIAS_FLUSH_START 0xffff4000 static void flush_pfn_alias(unsigned long pfn, unsigned long vaddr) { unsigned long to = ALIAS_FLUSH_START + (CACHE_COLOUR(vaddr) << PAGE_SHIFT); const int zero = 0; set_pte_ext(TOP_PTE(to), pfn_pte(pfn, PAGE_KERNEL), 0); flush_tlb_kernel_page(to); asm( "mcrr p15, 0, %1, %0, c14\n" " mcr p15, 0, %2, c7, c10, 4" : : "r" (to), "r" (to + PAGE_SIZE - L1_CACHE_BYTES), "r" (zero) : "cc"); } void flush_cache_mm(struct mm_struct *mm) { if (cache_is_vivt()) { vivt_flush_cache_mm(mm); return; } if (cache_is_vipt_aliasing()) { asm( "mcr p15, 0, %0, c7, c14, 0\n" " mcr p15, 0, %0, c7, c10, 4" : : "r" (0) : "cc"); } } void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { if (cache_is_vivt()) { vivt_flush_cache_range(vma, start, end); return; } if (cache_is_vipt_aliasing()) { asm( "mcr p15, 0, %0, c7, c14, 0\n" " mcr p15, 0, %0, c7, c10, 4" : : "r" (0) : "cc"); } if (vma->vm_flags & VM_EXEC) __flush_icache_all(); } void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn) { if (cache_is_vivt()) { vivt_flush_cache_page(vma, user_addr, pfn); return; } if (cache_is_vipt_aliasing()) { flush_pfn_alias(pfn, user_addr); __flush_icache_all(); } if (vma->vm_flags & VM_EXEC && icache_is_vivt_asid_tagged()) __flush_icache_all(); } #else #define flush_pfn_alias(pfn,vaddr) do { } while (0) #endif #ifdef CONFIG_SMP static void flush_ptrace_access_other(void *args) { __flush_icache_all(); } #endif static void flush_ptrace_access(struct vm_area_struct *vma, struct page *page, unsigned long uaddr, void *kaddr, unsigned long len) { if (cache_is_vivt()) { if (cpumask_test_cpu(smp_processor_id(), mm_cpumask(vma->vm_mm))) { unsigned long addr = (unsigned long)kaddr; __cpuc_coherent_kern_range(addr, addr + len); } return; } if (cache_is_vipt_aliasing()) { flush_pfn_alias(page_to_pfn(page), uaddr); __flush_icache_all(); return; } /* VIPT non-aliasing cache */ if (vma->vm_flags & VM_EXEC) { unsigned long addr = (unsigned long)kaddr; __cpuc_coherent_kern_range(addr, addr + len); #ifdef CONFIG_SMP if (cache_ops_need_broadcast()) smp_call_function(flush_ptrace_access_other, NULL, 1); #endif } } /* * Copy user data from/to a page which is mapped into a different * processes address space. Really, we want to allow our "user * space" model to handle this. * * Note that this code needs to run on the current CPU. */ void copy_to_user_page(struct vm_area_struct *vma, struct page *page, unsigned long uaddr, void *dst, const void *src, unsigned long len) { #ifdef CONFIG_SMP preempt_disable(); #endif fcse_flush_cache_user_range(vma, uaddr, uaddr + len); memcpy(dst, src, len); flush_ptrace_access(vma, page, uaddr, dst, len); #ifdef CONFIG_SMP preempt_enable(); #endif } void __flush_dcache_page(struct address_space *mapping, struct page *page) { void *addr = page_address(page); /* * Writeback any data associated with the kernel mapping of this * page. This ensures that data in the physical page is mutually * coherent with the kernels mapping. */ #ifdef CONFIG_HIGHMEM /* * kmap_atomic() doesn't set the page virtual address, and * kunmap_atomic() takes care of cache flushing already. */ if (addr) #endif __cpuc_flush_dcache_area(addr, PAGE_SIZE); /* * If this is a page cache page, and we have an aliasing VIPT cache, * we only need to do one flush - which would be at the relevant * userspace colour, which is congruent with page->index. */ if (mapping && cache_is_vipt_aliasing()) flush_pfn_alias(page_to_pfn(page), page->index << PAGE_CACHE_SHIFT); } static void __flush_dcache_aliases(struct address_space *mapping, struct page *page) { struct mm_struct *mm = current->active_mm; struct vm_area_struct *mpnt; struct prio_tree_iter iter; pgoff_t pgoff; /* * There are possible user space mappings of this page: * - VIVT cache: we need to also write back and invalidate all user * data in the current VM view associated with this page. * - aliasing VIPT: we only need to find one mapping of this page. */ pgoff = page->index << (PAGE_CACHE_SHIFT - PAGE_SHIFT); flush_dcache_mmap_lock(mapping); vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) { unsigned long offset; /* * If this VMA is not in our MM, we can ignore it. */ if (mpnt->vm_mm != mm) continue; if (!(mpnt->vm_flags & VM_MAYSHARE)) continue; offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT; flush_cache_page(mpnt, mpnt->vm_start + offset, page_to_pfn(page)); } flush_dcache_mmap_unlock(mapping); } /* * Ensure cache coherency between kernel mapping and userspace mapping * of this page. * * We have three cases to consider: * - VIPT non-aliasing cache: fully coherent so nothing required. * - VIVT: fully aliasing, so we need to handle every alias in our * current VM view. * - VIPT aliasing: need to handle one alias in our current VM view. * * If we need to handle aliasing: * If the page only exists in the page cache and there are no user * space mappings, we can be lazy and remember that we may have dirty * kernel cache lines for later. Otherwise, we assume we have * aliasing mappings. * * Note that we disable the lazy flush for SMP. */ void flush_dcache_page(struct page *page) { struct address_space *mapping; /* * The zero page is never written to, so never has any dirty * cache lines, and therefore never needs to be flushed. */ if (page == ZERO_PAGE(0)) return; mapping = page_mapping(page); #ifndef CONFIG_SMP if (!PageHighMem(page) && mapping && !mapping_mapped(mapping)) set_bit(PG_dcache_dirty, &page->flags); else #endif { __flush_dcache_page(mapping, page); if (mapping && cache_is_vivt()) __flush_dcache_aliases(mapping, page); else if (mapping) __flush_icache_all(); } } EXPORT_SYMBOL(flush_dcache_page); /* * Flush an anonymous page so that users of get_user_pages() * can safely access the data. The expected sequence is: * * get_user_pages() * -> flush_anon_page * memcpy() to/from page * if written to page, flush_dcache_page() */ void __flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr) { unsigned long pfn; /* VIPT non-aliasing caches need do nothing */ if (cache_is_vipt_nonaliasing()) return; /* * Write back and invalidate userspace mapping. */ pfn = page_to_pfn(page); if (cache_is_vivt()) { flush_cache_page(vma, vmaddr, pfn); } else { /* * For aliasing VIPT, we can flush an alias of the * userspace address only. */ flush_pfn_alias(pfn, vmaddr); __flush_icache_all(); } /* * Invalidate kernel mapping. No data should be contained * in this mapping of the page. FIXME: this is overkill * since we actually ask for a write-back and invalidate. */ __cpuc_flush_dcache_area(page_address(page), PAGE_SIZE); } ^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 23:25 ` Felipe Brandão Cavalcanti @ 2010-07-02 7:33 ` Fredrik Asplund 2010-07-02 12:29 ` Fredrik Asplund 2010-07-02 8:14 ` Gilles Chanteperdrix 1 sibling, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-02 7:33 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org >I've edited the bitbake files for the kernel, but what I usually do is to >run the unpack script, replace the flush.c file, and keep the process going. >The defconfig is pretty much standard, except that I've enabled several >debugging options and got rid of a few troublesome pieces of code (power >management, CPU Freq. scaling, etc...). As for the edits, they seem very >similar to yours, but I didn't look at them very carefully. >Fredrik, did you managed to get a kernel compiled at all? Yes, no problems in compiling, but I have not yet had the time to actually run the resulting image. I will check where my setup crashes (no doubt it will... :/) and compare our files to see if I can figure something out. / Fredrik ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 7:33 ` Fredrik Asplund @ 2010-07-02 12:29 ` Fredrik Asplund 2010-07-02 12:35 ` Felipe Brandão Cavalcanti 2010-07-02 12:40 ` Gilles Chanteperdrix 0 siblings, 2 replies; 32+ messages in thread From: Fredrik Asplund @ 2010-07-02 12:29 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org >>I've edited the bitbake files for the kernel, but what I usually do is to >>run the unpack script, replace the flush.c file, and keep the process going. >>The defconfig is pretty much standard, except that I've enabled several >>debugging options and got rid of a few troublesome pieces of code (power >>management, CPU Freq. scaling, etc...). As for the edits, they seem very >>similar to yours, but I didn't look at them very carefully. >>Fredrik, did you managed to get a kernel compiled at all? > >Yes, no problems in compiling, but I have not yet had the time to actually run the resulting image. I will >check where my setup crashes (no doubt it will... :/) and compare our files to see if I can figure something >out. Now I had time to test the resulting image. It boots up without obvious problems and does not seem to be unstable over time either. However, I am unsure of how to test if Xenomai *actually* works (I found xeno-test in /usr/bin/, but I am unsure of how to judge the output... there does not seem to be any ERROR messages in it though). Anyone who can tell me how to do this? (BTW - I will add a description on what I did + the files to my project wiki. I will send you a link when it is done, sometime this weekend I guess...) / Fredrik ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 12:29 ` Fredrik Asplund @ 2010-07-02 12:35 ` Felipe Brandão Cavalcanti 2010-07-02 12:40 ` Gilles Chanteperdrix 1 sibling, 0 replies; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-02 12:35 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 1676 bytes --] 2010/7/2 Fredrik Asplund <fasplund@domain.hid> > >>I've edited the bitbake files for the kernel, but what I usually do is to > >>run the unpack script, replace the flush.c file, and keep the process > going. > >>The defconfig is pretty much standard, except that I've enabled several > >>debugging options and got rid of a few troublesome pieces of code (power > >>management, CPU Freq. scaling, etc...). As for the edits, they seem very > >>similar to yours, but I didn't look at them very carefully. > >>Fredrik, did you managed to get a kernel compiled at all? > > > >Yes, no problems in compiling, but I have not yet had the time to actually > run the resulting image. I will >check where my setup crashes (no doubt it > will... :/) and compare our files to see if I can figure something >out. > > Now I had time to test the resulting image. It boots up without obvious > problems and does not seem to be unstable over time either. > > However, I am unsure of how to test if Xenomai *actually* works (I found > xeno-test in /usr/bin/, but I am unsure of how to judge the output... there > does not seem to be any ERROR messages in it though). Anyone who can tell me > how to do this? > > (BTW - I will add a description on what I did + the files to my project > wiki. I will send you a link when it is done, sometime this weekend I > guess...) > > Great! Can you send me the uImage for testing? > / Fredrik > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 2355 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 12:29 ` Fredrik Asplund 2010-07-02 12:35 ` Felipe Brandão Cavalcanti @ 2010-07-02 12:40 ` Gilles Chanteperdrix 2010-07-02 12:45 ` Fredrik Asplund 1 sibling, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-02 12:40 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org Fredrik Asplund wrote: >>> I've edited the bitbake files for the kernel, but what I usually >>> do is to run the unpack script, replace the flush.c file, and >>> keep the process going. The defconfig is pretty much standard, >>> except that I've enabled several debugging options and got rid of >>> a few troublesome pieces of code (power management, CPU Freq. >>> scaling, etc...). As for the edits, they seem very similar to >>> yours, but I didn't look at them very carefully. Fredrik, did you >>> managed to get a kernel compiled at all? >> Yes, no problems in compiling, but I have not yet had the time to >> actually run the resulting image. I will >check where my setup >> crashes (no doubt it will... :/) and compare our files to see if I >> can figure something >out. > > Now I had time to test the resulting image. It boots up without > obvious problems and does not seem to be unstable over time either. > > However, I am unsure of how to test if Xenomai *actually* works (I > found xeno-test in /usr/bin/, but I am unsure of how to judge the > output... there does not seem to be any ERROR messages in it though). > Anyone who can tell me how to do this? Run the latency test. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 12:40 ` Gilles Chanteperdrix @ 2010-07-02 12:45 ` Fredrik Asplund 2010-07-02 12:59 ` Felipe Brandão Cavalcanti 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-02 12:45 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai@xenomai.org >Run the latency test. Ok. Well, the latencies output by the test seem within reasonable limits. Thanks. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 12:45 ` Fredrik Asplund @ 2010-07-02 12:59 ` Felipe Brandão Cavalcanti 2010-07-03 9:07 ` Fredrik Asplund 0 siblings, 1 reply; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-02 12:59 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 563 bytes --] OK, just tried your kernel - same problem. That means that something is wrong with my user-space build.... Will try to enable EABI now as suggested by Gilles. 2010/7/2 Fredrik Asplund <fasplund@domain.hid> > >Run the latency test. > > Ok. Well, the latencies output by the test seem within reasonable limits. > Thanks. > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 915 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 12:59 ` Felipe Brandão Cavalcanti @ 2010-07-03 9:07 ` Fredrik Asplund 2010-07-03 19:53 ` Felipe Brandão Cavalcanti 2010-07-03 20:04 ` Gilles Chanteperdrix 0 siblings, 2 replies; 32+ messages in thread From: Fredrik Asplund @ 2010-07-03 9:07 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org >OK, just tried your kernel - same problem. That means that something is wrong with my user-space build.... >Will try to enable EABI now as suggested by Gilles. Ok. You can find a description of what I did, the images and the build files at "http://flyg.sagar.se/node/17". Perhaps you can find the problem by comparing that with your setup. / Fredrik ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-03 9:07 ` Fredrik Asplund @ 2010-07-03 19:53 ` Felipe Brandão Cavalcanti 2010-07-05 7:44 ` Fredrik Asplund 2010-07-03 20:04 ` Gilles Chanteperdrix 1 sibling, 1 reply; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-03 19:53 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 1011 bytes --] Just tried your build... it seems that the available image doesn't include Xenomai! As for the procedures, they look a lot like mine, but I am having some very strange errors after updating my repositories (it compiled cleanly before...). I will try to take a look and fix this ASAP, but for now, I guess your image might do just what I need! 2010/7/3 Fredrik Asplund <fasplund@domain.hid> > >OK, just tried your kernel - same problem. That means that something is > wrong with my user-space build.... > >Will try to enable EABI now as suggested by Gilles. > > Ok. You can find a description of what I did, the images and the build > files at "http://flyg.sagar.se/node/17". Perhaps you can find the problem > by comparing that with your setup. > > / Fredrik > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 1487 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-03 19:53 ` Felipe Brandão Cavalcanti @ 2010-07-05 7:44 ` Fredrik Asplund 2010-07-05 13:13 ` Felipe Brandão Cavalcanti 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-05 7:44 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org > Just tried your build... it seems that the available image doesn't include Xenomai! Strange. How did you deploy it (i.e. if you deployed it on a SD Card, did you check that it actually followed that boot path?)? ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-05 7:44 ` Fredrik Asplund @ 2010-07-05 13:13 ` Felipe Brandão Cavalcanti 2010-07-05 13:21 ` Fredrik Asplund 0 siblings, 1 reply; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-05 13:13 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 754 bytes --] 2010/7/5 Fredrik Asplund <fasplund@domain.hid> > > Just tried your build... it seems that the available image doesn't > include Xenomai! > > Strange. How did you deploy it (i.e. if you deployed it on a SD Card, did > you check that it actually followed that boot path?)? > > I downloaded the uImage and RootFS from your website and deployed to the SD Card - and it did boot from the SD (I saw the clean startup and the logs). The kernel has Xenomai enabled, but the user-space portion doesn't seem to be installed. -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 1135 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-05 13:13 ` Felipe Brandão Cavalcanti @ 2010-07-05 13:21 ` Fredrik Asplund 2010-07-05 13:34 ` Fredrik Asplund 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-05 13:21 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org >I downloaded the uImage and RootFS from your website and deployed to the >SD Card - and it did boot from the SD (I saw the clean startup and the logs). >The kernel has Xenomai enabled, but the user-space portion doesn't seem to be installed. Ok. I must have copied the wrong file to DropBox. Try this one instead: http://dl.dropbox.com/u/8307741/Angstrom-gumstixwithxenomai-glibc-ipk-2010.4-test-20100701-overo.rootfs.tar.bz2 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-05 13:21 ` Fredrik Asplund @ 2010-07-05 13:34 ` Fredrik Asplund 2010-07-05 20:15 ` Felipe Brandão Cavalcanti 0 siblings, 1 reply; 32+ messages in thread From: Fredrik Asplund @ 2010-07-05 13:34 UTC (permalink / raw) To: Fredrik Asplund, Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org >>I downloaded the uImage and RootFS from your website and deployed to the >>SD Card - and it did boot from the SD (I saw the clean startup and the logs). >>The kernel has Xenomai enabled, but the user-space portion doesn't seem to be installed. >Ok. I must have copied the wrong file to DropBox. Try this one instead: >http://dl.dropbox.com/u/8307741/Angstrom-gumstixwithxenomai >-glibc-ipk-2010.4-test-20100701-overo.rootfs.tar.bz2 I checked my SD Card and the file above is the one I used when testing Xenomai last week (and also the one which has a name that corresponds to the build file I used). The other one must have been some old rootfs which I built earlier (it was around 23-01 when I updated the Wiki). Sorry. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-05 13:34 ` Fredrik Asplund @ 2010-07-05 20:15 ` Felipe Brandão Cavalcanti 0 siblings, 0 replies; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-05 20:15 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 1342 bytes --] Great! The entire system worked very well. Tested for 4 hours under load, best latency around 17us, and worst 47us. No lock-ups or instability. Thanks for the help! I will try to automate and document the process a bit, and put it in my lab's wiki (giving proper credit, of course). 2010/7/5 Fredrik Asplund <fasplund@domain.hid> > >>I downloaded the uImage and RootFS from your website and deployed to the > >>SD Card - and it did boot from the SD (I saw the clean startup and the > logs). > >>The kernel has Xenomai enabled, but the user-space portion doesn't seem > to be installed. > >Ok. I must have copied the wrong file to DropBox. Try this one instead: > >http://dl.dropbox.com/u/8307741/Angstrom-gumstixwithxenomai > >-glibc-ipk-2010.4-test-20100701-overo.rootfs.tar.bz2 > > I checked my SD Card and the file above is the one I used when testing > Xenomai last week (and also the one which has a name that corresponds to the > build file I used). The other one must have been some old rootfs which I > built earlier (it was around 23-01 when I updated the Wiki). > > Sorry. > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 1877 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-03 9:07 ` Fredrik Asplund 2010-07-03 19:53 ` Felipe Brandão Cavalcanti @ 2010-07-03 20:04 ` Gilles Chanteperdrix 2010-07-05 7:52 ` Fredrik Asplund 1 sibling, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-03 20:04 UTC (permalink / raw) To: Fredrik Asplund; +Cc: xenomai@xenomai.org Fredrik Asplund wrote: >> OK, just tried your kernel - same problem. That means that >> something is wrong with my user-space build.... Will try to enable >> EABI now as suggested by Gilles. > > Ok. You can find a description of what I did, the images and the > build files at "http://flyg.sagar.se/node/17". I read this page and would have a few remarks. When using prepare-kernel, you can pass all arguments on the command line to the script so that it does not ask any question. The Adeos project has a git: git://git.denx.de/ipipe-2.6.git You can use it to merge the I-pipe with your vendor branch. About the QA issues: if there is something wrong with Xenomai which causes these QA issues, please tell us so that we can fix it, for verifying Xenomai sources integrity, you can use the gpg signature. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-03 20:04 ` Gilles Chanteperdrix @ 2010-07-05 7:52 ` Fredrik Asplund 0 siblings, 0 replies; 32+ messages in thread From: Fredrik Asplund @ 2010-07-05 7:52 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai@xenomai.org >About the QA issues: if there is something wrong >with Xenomai which causes these QA issues, please >tell us so that we can fix it, for verifying Xenomai >sources integrity, you can use the gpg signature. I am no expert on these issues, but I think they have more to do with the build environment for Overo (and in some of the cases, the recipe files I created). I will look into it though. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-01 23:25 ` Felipe Brandão Cavalcanti 2010-07-02 7:33 ` Fredrik Asplund @ 2010-07-02 8:14 ` Gilles Chanteperdrix 2010-07-02 12:34 ` Felipe Brandão Cavalcanti 1 sibling, 1 reply; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-02 8:14 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org Felipe Brandão Cavalcanti wrote: > Sorry for the late reply. > > Anyway, I did enable Xenomai in the kernel itself, so no modules > involved - can't modprobe anything. > > As for the vanilla kernel, I've tried for a while to get it working > right on my board, but no luck. I am not sure if it has support to all > the peripherals, or if there is something that I am doing wrong (used > the defconfig, no modifications). > > I've edited the bitbake files for the kernel, but what I usually do is > to run the unpack script, replace the flush.c file, and keep the process > going. The defconfig is pretty much standard, except that I've enabled > several debugging options and got rid of a few troublesome pieces of > code (power management, CPU Freq. scaling, etc...). As for the edits, > they seem very similar to yours, but I didn't look at them very carefully. > > Fredrik, did you managed to get a kernel compiled at all? What about the EABI? -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 8:14 ` Gilles Chanteperdrix @ 2010-07-02 12:34 ` Felipe Brandão Cavalcanti 2010-07-02 12:39 ` Gilles Chanteperdrix 0 siblings, 1 reply; 32+ messages in thread From: Felipe Brandão Cavalcanti @ 2010-07-02 12:34 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai@xenomai.org [-- Attachment #1: Type: text/plain, Size: 1498 bytes --] 2010/7/2 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> > Felipe Brandão Cavalcanti wrote: > > Sorry for the late reply. > > > > Anyway, I did enable Xenomai in the kernel itself, so no modules > > involved - can't modprobe anything. > > > > As for the vanilla kernel, I've tried for a while to get it working > > right on my board, but no luck. I am not sure if it has support to all > > the peripherals, or if there is something that I am doing wrong (used > > the defconfig, no modifications). > > > > I've edited the bitbake files for the kernel, but what I usually do is > > to run the unpack script, replace the flush.c file, and keep the process > > going. The defconfig is pretty much standard, except that I've enabled > > several debugging options and got rid of a few troublesome pieces of > > code (power management, CPU Freq. scaling, etc...). As for the edits, > > they seem very similar to yours, but I didn't look at them very > carefully. > > > > Fredrik, did you managed to get a kernel compiled at all? > > What about the EABI? > EABI is disabled in both the kernel-space and the user-space portion. Will try with it later, just to be sure. > > -- > Gilles. > > -- -Felipe Brandão Cavalcanti LARA - Automation and Robotics Laboratory Department of Electrical Engineering UnB - University of Brasília, Brazil http://www.lara.unb.br/~fbcavalcanti/ [-- Attachment #2: Type: text/html, Size: 2172 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [Xenomai-help] Xenomai and Gumstix Overo 2010-07-02 12:34 ` Felipe Brandão Cavalcanti @ 2010-07-02 12:39 ` Gilles Chanteperdrix 0 siblings, 0 replies; 32+ messages in thread From: Gilles Chanteperdrix @ 2010-07-02 12:39 UTC (permalink / raw) To: Felipe Brandão Cavalcanti; +Cc: xenomai@xenomai.org Felipe Brandão Cavalcanti wrote: > 2010/7/2 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org > <mailto:gilles.chanteperdrix@xenomai.org>> > > Felipe Brandão Cavalcanti wrote: > > Sorry for the late reply. > > > > Anyway, I did enable Xenomai in the kernel itself, so no modules > > involved - can't modprobe anything. > > > > As for the vanilla kernel, I've tried for a while to get it working > > right on my board, but no luck. I am not sure if it has support to all > > the peripherals, or if there is something that I am doing wrong (used > > the defconfig, no modifications). > > > > I've edited the bitbake files for the kernel, but what I usually do is > > to run the unpack script, replace the flush.c file, and keep the > process > > going. The defconfig is pretty much standard, except that I've enabled > > several debugging options and got rid of a few troublesome pieces of > > code (power management, CPU Freq. scaling, etc...). As for the edits, > > they seem very similar to yours, but I didn't look at them very > carefully. > > > > Fredrik, did you managed to get a kernel compiled at all? > > What about the EABI? > > > EABI is disabled in both the kernel-space and the user-space portion. > Will try with it later, just to be sure. Are you sure your toolchain is not an EABI toolchain? Most recent toolchains are. -- Gilles. ^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2010-07-05 20:15 UTC | newest] Thread overview: 32+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-06-29 19:40 [Xenomai-help] Xenomai and Gumstix Overo Fredrik Asplund 2010-06-29 20:03 ` Gilles Chanteperdrix 2010-06-30 8:17 ` Fredrik Asplund 2010-06-30 8:28 ` Gilles Chanteperdrix 2010-06-30 21:58 ` Felipe Brandão Cavalcanti 2010-07-01 3:01 ` Thomas Lockhart 2010-07-01 6:49 ` Gilles Chanteperdrix 2010-07-01 7:17 ` Gilles Chanteperdrix 2010-07-01 8:11 ` Fredrik Asplund 2010-07-01 8:59 ` Fredrik Asplund 2010-07-01 9:54 ` Gilles Chanteperdrix 2010-07-01 10:16 ` Fredrik Asplund 2010-07-01 12:22 ` Fredrik Asplund 2010-07-01 23:25 ` Felipe Brandão Cavalcanti 2010-07-02 7:33 ` Fredrik Asplund 2010-07-02 12:29 ` Fredrik Asplund 2010-07-02 12:35 ` Felipe Brandão Cavalcanti 2010-07-02 12:40 ` Gilles Chanteperdrix 2010-07-02 12:45 ` Fredrik Asplund 2010-07-02 12:59 ` Felipe Brandão Cavalcanti 2010-07-03 9:07 ` Fredrik Asplund 2010-07-03 19:53 ` Felipe Brandão Cavalcanti 2010-07-05 7:44 ` Fredrik Asplund 2010-07-05 13:13 ` Felipe Brandão Cavalcanti 2010-07-05 13:21 ` Fredrik Asplund 2010-07-05 13:34 ` Fredrik Asplund 2010-07-05 20:15 ` Felipe Brandão Cavalcanti 2010-07-03 20:04 ` Gilles Chanteperdrix 2010-07-05 7:52 ` Fredrik Asplund 2010-07-02 8:14 ` Gilles Chanteperdrix 2010-07-02 12:34 ` Felipe Brandão Cavalcanti 2010-07-02 12:39 ` Gilles Chanteperdrix
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.