performance monitor exceptions

performance monitor exceptions

[Werner Almesberger]

Hi,

I'm banging my head at a problem with exceptions. What I want to
do is to get the performance monitor to give me exceptions at
roughly 1 kHz, to drive the kernel profiler. The problem is that
my changes usually kill the system within a few minutes :-(

I'm quite new to the PPC, so I'll include a bit more details, in
case I made some trivial mistake. My kernel is 2.4.17, my CPU is a
MPC7410.

First of all, I changed the vector to:

        . = 0xf00
        b       PerformanceMonitor

Then, I added a tiny little handler that does nothing but re-enable
the PE exception. For getting a register, I mimicked EXCEPTION_PROLOG:

PerformanceMonitor:
        // 955 is SIAR
        mtspr   955,r20
        mfspr   r20,SPRN_MMCR0
        oris    r20,r20,MMCR0_PMXE@h
        mtspr   SPRN_MMCR0,r20
        mfspr   r20,955
        SYNC
        RFI

And then I added of course a bit of code to enable all this:

int __init setup_perf_mon(void)
{
        unsigned long flags;

        local_irq_save(flags);
        local_irq_disable();
        /* ca. 3 kHz */
        mtspr(SPRN_MMCR0,(mfspr(SPRN_MMCR0) & ~MMCR0_TBSEL_MASK) |
          MMCR0_PMXE | (MMCR0_TBSEL_TBL15 << MMCR0_TBSEL_SHIFT) | MMCR0_TBEE);
        local_irq_restore(flags);
        return 0;
}

It seems to work (I had a handler there that does something a little
more useful), but when I start kernel builds, I get a dead system
(rebooted by hardware watchdog) within a few minutes.

Instead of SIAR, I also tried SPRG0 and SPRG1, but got exactly the
same behaviour. Time to crash varies from one to ten minutes. The
time increases proportionally if I use TLB[19] (~200 Hz) instead.

Any ideas of what I could be trampling over (or be trampled over
by) ? This code seems rather bullet-proof to me, and I don't think
anything else is using SIAR, or could change it while I'm using it.

Thanks,
- Werner

--
  _________________________________________________________________________
 / Werner Almesberger, Lausanne, CH                    wa@almesberger.net /
/_http://icawww.epfl.ch/almesberger/_____________________________________/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: performance monitor exceptions

[benh]

>It seems to work (I had a handler there that does something a little
>more useful), but when I start kernel builds, I get a dead system
>(rebooted by hardware watchdog) within a few minutes.
>
>Instead of SIAR, I also tried SPRG0 and SPRG1, but got exactly the
>same behaviour. Time to crash varies from one to ten minutes. The
>time increases proportionally if I use TLB[19] (~200 Hz) instead.
>
>Any ideas of what I could be trampling over (or be trampled over
>by) ? This code seems rather bullet-proof to me, and I don't think
>anything else is using SIAR, or could change it while I'm using it.

A first quick look doesn't seem to show any problem with your code
I have two clues at hand:

 - I don't know how the perf. mon stuff works, but couldn't you
enter a condition were the CPU is continuously sending your the
pef monitor exception until you clear something else ?

 - I encountered a problem with the TAU stuff and the 7400 where
the CPU could actually get mixed up when getting both a normal
interrupt and a TAU exception and scew up itself (by taking both
exceptions and screwing up SRR0/1)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: performance monitor exceptions

[Werner Almesberger]

benh@kernel.crashing.org wrote:
>  - I don't know how the perf. mon stuff works, but couldn't you
> enter a condition were the CPU is continuously sending your the
> pef monitor exception until you clear something else ?

No, in a performance monitor exception, PMXE (PM exception enable)
is cleared in MMCR0, and I need to set it explicitly to get another
exception. The only performance event source that's enabled is the
one using the timer base, so this gives me exceptions at 200 Hz,
more if I want.

>  - I encountered a problem with the TAU stuff and the 7400 where
> the CPU could actually get mixed up when getting both a normal
> interrupt and a TAU exception and scew up itself (by taking both
> exceptions and screwing up SRR0/1)

Bingo ! That seems to be the one:

http://www.mot.co.jp/SPS/html/lit/pdf/mpc7410pns_rev0.pdf

  #5:
  Description: Consecutive Performance Monitor, Decrementer,
  or Thermal Management Interrupts may become non-recoverable.

  Work-Around: Avoid enabling any combination of
  Performance Monitor, Decrementer, or Thermal Management
  Interrupts at the same time.

Hmm, working around this will be tricky ...

Thanks a lot !

- Werner

--
  _________________________________________________________________________
 / Werner Almesberger, Lausanne, CH                    wa@almesberger.net /
/_http://icawww.epfl.ch/almesberger/_____________________________________/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: performance monitor exceptions

[Anton Blanchard]

> Bingo ! That seems to be the one:
>
> http://www.mot.co.jp/SPS/html/lit/pdf/mpc7410pns_rev0.pdf
>
>   #5:
>   Description: Consecutive Performance Monitor, Decrementer,
>   or Thermal Management Interrupts may become non-recoverable.
>
>   Work-Around: Avoid enabling any combination of
>   Performance Monitor, Decrementer, or Thermal Management
>   Interrupts at the same time.

Thats rather ugly :( BTW I have run kernel profiling off the performance
monitor exception on ppc64 which has been very useful (it allowed us
to profile inside interrupt disabled regions)

Anton

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: performance monitor exceptions

[benh]

>Thats rather ugly :( BTW I have run kernel profiling off the performance
>monitor exception on ppc64 which has been very useful (it allowed us
>to profile inside interrupt disabled regions)

A workaround for pmac or openpic ppc32 is to use the keylargo or the
openpic timer instead of the DEC. That could be a "kernel hacking" option

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

G4 boot crash problems with 2.4 "stable"

[David A. Gatwood]

On Wed, 13 Feb 2002 benh@kernel.crashing.org wrote:

> A workaround for pmac or openpic ppc32 is to use the keylargo or the
> openpic timer instead of the DEC. That could be a "kernel hacking" option

Speaking of OpenPIC....  I tried to build a kernel from BK TOT last night
for a PowerMac G4, and it hung about a half screen into booting after some
yammering about OpenPIC.

openpic enter
openpic timer
openpic ext
(hang)


Is there a 2.4 kernel source tarball around that is known to boot on
Sawtooth (using BootX)?  Is there a way to yank the OpenPIC support?


David

---------------------------------------------------------------------
                    Check out my weekly web comic:
                     http://www.techmagazine.org


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: G4 boot crash problems with 2.4 "stable"

[benh]

>> A workaround for pmac or openpic ppc32 is to use the keylargo or the
>> openpic timer instead of the DEC. That could be a "kernel hacking" option
>
>Speaking of OpenPIC....  I tried to build a kernel from BK TOT last night
>for a PowerMac G4, and it hung about a half screen into booting after some
>yammering about OpenPIC.
>
>openpic enter
>openpic timer
>openpic ext
>(hang)
>
>
>Is there a 2.4 kernel source tarball around that is known to boot on
>Sawtooth (using BootX)?  Is there a way to yank the OpenPIC support?

BootX isn't supported on any newworld mac. You should boot from OF,
using yaboot.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: G4 boot crash problems with 2.4 "stable"

[David A. Gatwood]

On Wed, 13 Feb 2002 benh@kernel.crashing.org wrote:

> >Is there a 2.4 kernel source tarball around that is known to boot on
> >Sawtooth (using BootX)?  Is there a way to yank the OpenPIC support?
>
> BootX isn't supported on any newworld mac. You should boot from OF,
> using yaboot.

BootX works fine for me with every 2.2 kernel I've tried.  It only fails
for 2.4 TOT builds.  I very much doubt the boot mechanism is at fault.


Later,
David

---------------------------------------------------------------------
                    Check out my weekly web comic:
                     http://www.techmagazine.org


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: G4 boot crash problems with 2.4 "stable"

[Benjamin Herrenschmidt]

>> >Is there a 2.4 kernel source tarball around that is known to boot on
>> >Sawtooth (using BootX)?  Is there a way to yank the OpenPIC support?
>>
>> BootX isn't supported on any newworld mac. You should boot from OF,
>> using yaboot.
>
>BootX works fine for me with every 2.2 kernel I've tried.  It only fails
>for 2.4 TOT builds.  I very much doubt the boot mechanism is at fault.

Well, it can be for various reasons. For example, the interrupt tree
cannot be properly parsed, we don't get the OF "phandles" and we now
rely on them (MacOS don't keep them when it gets the device-tree).
Also, some bus mastering devices aren't properly shut down, potentially
screwing things up in memory while the kernel loads.

I would really recommend you give a try to yaboot before investigating
much further.

(Note: I do have a sawtooth here which just works with latest kernels)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: G4 boot crash problems with 2.4 "stable"

[Ethan Benson]

On Wed, Feb 13, 2002 at 11:01:09AM -0800, David A. Gatwood wrote:
>
> On Wed, 13 Feb 2002 benh@kernel.crashing.org wrote:
>
> > A workaround for pmac or openpic ppc32 is to use the keylargo or the
> > openpic timer instead of the DEC. That could be a "kernel hacking" option
>
> Speaking of OpenPIC....  I tried to build a kernel from BK TOT last night
> for a PowerMac G4, and it hung about a half screen into booting after some
> yammering about OpenPIC.
>
> openpic enter
> openpic timer
> openpic ext
> (hang)
>
>
> Is there a 2.4 kernel source tarball around that is known to boot on
> Sawtooth (using BootX)?  Is there a way to yank the OpenPIC support?

http://penguinppc.org/projects/yaboot/doc/yaboot-howto.shtml

--
Ethan Benson
http://www.alaska.net/~erbenson/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/