Feedback requested on switching the exception wrapper used for the PMU interrupt on ppc64

[Corey Ashford <cjashfor@us.ibm.com>]

Hello,

One of the things I've been working on is porting perfmon2 to ppc64. 
We've made a fair amount of progress on it, and support is available in 
libpfm and the perfmon2 kernel patch.

One of the things we had to work around was the "lazy interrupt 
disabling" mechanism in ppc64 Linux.  The problem was that the PMU 
exception handler (0xf00) uses the STD_EXCEPTION_PSERIES wrapper, which 
does not support lazy interrupt disabling.

This is desirable for Oprofile's use of the PMU since its handler is 
fairly simple and being able to profile interrupt protected code is 
desirable.  However, it causes problems for perfmon2, since the 
operations it performs on the thread of its PMU interrupt handler can 
cause a deadlock condition (it can end up calling spin_lock, for example).

Initially, to work around this, we created special spin_lock_irqsave and 
spin_unlock_irqrestore macros for perfmon2 which we could override for 
POWER to define them as functions which do hard disables and restores.

However, not all of the places that we need to disable interrupts were 
occurring from within the perfmon2 code.  Specifically, getting PMU 
interrupts in the middle of a schedule() call (where interrupts were 
expected to be disabled) was causing kernel hangs.

To fix this, I've gone back and removed the special spin_lock macros we 
defined in perfmon2 and have ifdef'd arch/powerpc/kernel/head_64.S file 
as follows:

/*** pSeries interrupt support ***/

         /* moved from 0xf00 */
+ #ifdef CONFIG_PERFMON
+         MASKABLE_EXCEPTION_PSERIES(., performance_monitor)
+ #else
         STD_EXCEPTION_PSERIES(., performance_monitor)
+ #endif

/*
  * An interrupt came in while soft-disabled; clear EE in SRR1,
  * clear paca->hard_enabled and return.

The downside of this change is that if someone is using Oprofile and 
they have enabled perfmon in their kernel, they will not get profile 
samples that occur in interrupt-protected regions of the kernel. 
However, they still can by configuring perfmon out of their kernel.

What do you think of this idea?  Is this something that you would object 
to when perfmon2 does go upstream to LKML?  I think we'd want to add 
some documentation somewhere that describes this side-effect of enabling 
perfmon in the ppc64 Linux kernel.  Do you have suggestions on where 
that should go?  I'm thinking perhaps in both the basic_profiling.txt 
and perfmon2.txt files in the Documentation subdirectory.

Thanks for your consideration,

- Corey

-- 
Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
cjashfor@us.ibm.com