high priority interrupts disabled

high priority interrupts disabled

[Steve Rossi]

Hi All,
I'm using linuxppc-2.4.16 on custom 855T based hardware. I've got an IDE
disk and I'm using the 8xx IDE driver in EXT_DIRECT mode. The ide
interface is using IRQ6, and I've got two other interrupts of interest
on IRQ1 and IRQ2. I have found that when I'm continuously streaming data
to the disk, one interrupt gets puts off for too long. I have observed
on a logic analyzer, IRQ2 gets asserted and remains asserted (without
being serviced) in excess of 10ms during which IRQ6 (hard disk) is
asserted and quickly serviced over 120 times, and IRQ1 is also asserted
and quickly serviced several times in that 10ms. IRQ2 doesn't get
serviced until the disk activity completes - perhaps at the end of
ide_multiwrite() - though I haven't verified this. So why is IRQ2 which
is higher priority than IRQ6 not being serviced yet IRQ1 is? I've set
drive->unmask=1 in the ide drive structure, so __cli() doesn't get
called in do_rw_disk(). IRQ1 is requested with the SA_INTERRUPT flag,
while IRQ2 is not - should this matter? I need for IRQ2 to be able to
preempt the disk write in the same way that IRQ1 is able to. If anyone
can offer any suggestions as to why its not, or where I should be
looking, that would be helpful.

Thanks,
Steve

--
-------------------------------------------------------
Steven K. Rossi                     srossi@labs.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled

[Dan Malek]

Steve Rossi wrote:

> .....IRQ2 gets asserted and remains asserted (without
> being serviced) in excess of 10ms during which IRQ6 (hard disk) is
> asserted and quickly serviced over 120 times, and IRQ1 is also asserted
> and quickly serviced several times in that 10ms. IRQ2 doesn't get
> serviced until the disk activity completes

That's interesting.  We read the SIVEC, which gives us the highest priority
unmasked pending interrupt.  We don't read and parse bitmasks.  Since IRQ1
is getting serviced, I wonder if IRQ2 is being masked.  It is possible
to mask an irq, service others, then unmask the irq again, so ensure your
driver isn't doing this.  Of course, we are also doing a mask_and_ack in
the 8xx interrupt handler, which may not be the right thing either.

You could try changing the mask_and_ack function in the ppc8xx_pic.c to
just ack the interrupt and let us know what happens.

> .... IRQ1 is requested with the SA_INTERRUPT flag,
> while IRQ2 is not - should this matter?

I didn't think this flag had any effect anymore.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled

[Steve Rossi]

Dan Malek wrote:

> Steve Rossi wrote:
>
> > .....IRQ2 gets asserted and remains asserted (without
> > being serviced) in excess of 10ms during which IRQ6 (hard disk) is
> > asserted and quickly serviced over 120 times, and IRQ1 is also asserted
> > and quickly serviced several times in that 10ms. IRQ2 doesn't get
> > serviced until the disk activity completes

>
> ...I wonder if IRQ2 is being masked.  It is possible
> to mask an irq, service others, then unmask the irq again, so ensure your
> driver isn't doing this.  Of course, we are also doing a mask_and_ack in
> the 8xx interrupt handler, which may not be the right thing either.

I verified that IRQ2 is not getting masked in the SIMASK register during the
disk I/O which is when it can get held off. I've also tried doing ack only
instead of mask_and_ack - I couldn't boot past the enabling of the FEC
ethernet when I did this, so I didn't spend much more time on it.

> > .... IRQ1 is requested with the SA_INTERRUPT flag,
> > while IRQ2 is not - should this matter?
>
> I didn't think this flag had any effect anymore.

SA_INTERRUPT definitely has an effect: in irq.c in handle_irq_event() :
if (!(action->flags & SA_INTERRUPT))
                __sti();

apparently, interrupts installed with SA_INTERRUPT flag will run with external
interrupts disabled (EE bit cleared in MSR). right? or am I missing something.

I've also found that if I install my IRQ2 using the SA_INTERRUPT flag, I do
not see the problem. But when IRQ2 is installed with SA_INTERRUPT, IRQ1 cannot
preempt it (which is not the behavior that I want) presumably because it it
running with external interrupt disabled.  Apparently the IDE interrupt is
installed with the SA_INTERRUPT flag set as well (documented in ide-probe.c).
Next I will experiment with installing the IDE interrupt without the
SA_INTERRUPT flag.

Another observation that I've made is that the problem doesn't occur all the
time, but it seems to be triggered when IRQ2 is asserted during IRQ6
processing (or while IRQ6 is asserted). If IRQ6 really is being installed with
SA_INTERRUPT, it would make sense that IRQ2 wouldn't get serviced until IRQ6
completes, but IRQ6 completes over 120 times before IRQ2 is serviced - that
doesn't make sense.

If anyone has any more information regarding how the SA_INTERRUPT flag works
that will be helpful. I suspect the problem that I am having is rooted in
something that is going on in the IDE driver code as opposed to the kernel
interrrupt processing - but I'm not ruling that out either. If anyone is more
familar with the IDE driver who can offer a suggestion as to why this is
happening I would appreciate that. If there are any other suggestions, I'd
love to hear them too.

Thanks!
Steve

--
-------------------------------------------------------
Steven K. Rossi                     srossi@labs.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled

[Wolfgang Denk]

In message <3C14F91F.F382ABE3@labs.mot.com> you wrote:
>
> Another observation that I've made is that the problem doesn't occur all the
> time, but it seems to be triggered when IRQ2 is asserted during IRQ6
> processing (or while IRQ6 is asserted). If IRQ6 really is being installed with
> SA_INTERRUPT, it would make sense that IRQ2 wouldn't get serviced until IRQ6
> completes, but IRQ6 completes over 120 times before IRQ2 is serviced - that
> doesn't make sense.

Maybe you can get more information using the Linux Trace Toolkit;  if
necessary,  add  some  custom  traces  in  the suspected areas of the
interrupt handlers. See http://www.opersys.com/LTT/index.html

Hope this helps,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
People are very flexible and learn to adjust to strange  surroundings
--  they can become accustomed to read Lisp and Fortran programs, for
example.   - Leon Sterling and Ehud Shapiro, Art of Prolog, MIT Press

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled

[Steve Rossi]

Hi All,
Here's an update

> Dan Malek wrote:
>
> > ...I wonder if IRQ2 is being masked.  ...
>
> I verified that IRQ2 is not getting masked in the SIMASK register during the
> disk I/O which is when it can get held off ...

Well I lied - sort of - turns out that under normal circumstances IRQ2 doesn't
get masked but under some condition (which I have not yet determined) IRQ2 does
get masked during IRQ6.
Since IRQ6 - the hard disk interrupt - does data transfer in the ISR, then
interrupts again, the processor ends up exiting from the ISR, then jumping right
back in hundreds of times in a row. I'm a newbie to writing Linux drivers, but
this seems to me like poor practice - to do the work of data transfer to the disk
in the ISR itself, but I guess its only really a problem in PIO modes. I would be
interested in knowing why its done this way. Regardless, my IRQ2 doesn't get
unmasked again until the full number of sectors is written to the disk.

I tried to use the Linux Trace Toolkit - as Wolfgang suggested, but after
applying the patches to my 2.4.16 kernel and hacking in the pieces that didn't
patch cleanly, the kernel wouldn't boot, so I didn't spend much more time on it.
I think it would only give me marginally more information than what I'm able to
gleen by looking at the bus activity.

Are there any other experiences with ide-disk driver causing other interrupts to
be held off for long periods of time?

Can someone who is familiar with the interrupt entry/exit mechanisms comment on
if it is possible for a normal IRQ2 isr to run (note it runs with interrupts
enabled MSR EE bit =1), then as its exiting, but before it is re-enabled for IRQ6
to run? Since IRQ6 fires one right after the other hundreds of times, the
un-masking of IRQ2 wouldn't happen until IRQ6 stops firing?? Is this plausable or
is there sufficient protection to guarantee that IRQ2 will exit completely
(thereby getting un-masked) before IRQ6 can run? Is there a mechanism by which
all lower priority interrupts are disabled when a higher priority one is running?
If not, it seems to me this could happen.

Thanks,
Steve


--
-------------------------------------------------------
Steven K. Rossi                     srossi@labs.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled - problem found

[Steve Rossi]

Here's another update for anyone interested:

Steve Rossi wrote:

> ...Is there a mechanism by which
> all lower priority interrupts are disabled when a higher priority one is running?

This is exactly the problem. I was able to run with data show-cycles enabled so
I could trace exactly what was happening with the SIVEC, SIMASK & SIPEND. Here's the
situation that causes the problem I described in earlier posts:
The hard disk interrupt (IRQ6) does data transfer in the ISR itself - as a result
the ISRs are very long, and furthermore, the interrupt signal is asserted again
before the service routine is exited and IRQ6 is unmasked - so as a result, as soon
as its unmasked, it causes a new interrupt to the core and IRQ6 is entered again.

Take the situation where IRQ2 becomes active between the time when IRQ6 is unmasked,
but before the next pending IRQ6 is entered. SIVEC is read, and indicates IRQ2, IRQ2
is masked & acked, then since IRQ2 is NOT flagged with SA_INTERRUPT, the interrupts
to the core are enabled. Immediately, the core recognizes that IRQ6 is pending and
unmasked so it begins to service it. Essentially IRQ6 pre-empted IRQ2, IRQ2 is
masked out. IRQ6 will run, unmask, interrupt again, mask, run, unmask, interrupt
again ... etc until the complete transfer to the disk is done. Only when there is no
longer any other interrupts pending will the IRQ2 which was preempted earlier finish
running - many milliseconds after IRQ2 was triggered.

So the problem is that mask & ack only masks the pending interrupts when I believe
it should mask the pending interrupts and all interrupts that are lower priority
than it. Conversely unmask should unmask the given interrupt and all interrupts of
lower priority that were previously unmasked when mask&ack was called. Is that right
or am I missing something here?


Steve

--
-------------------------------------------------------
Steven K. Rossi                     srossi@labs.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled - problem found

[Dan Malek]

Steve Rossi wrote:


> So the problem is that mask & ack only masks the pending interrupts....

You are right.  Sorry, but I didn't have time today to reply to your first
message.  The problem with the 8xx (and 8260, and 4xx) is there isn't any
notion of priorities within the interrupt nesting.  As you have seen, any
interrupt can interrupt another, and although SIVEC (or the software bit
search) gives us the highest priority pending interrupt, once we enable
them again we get the next one delivered.

We should mask all lower priority interrupts, and the challenge is keeping
the proper nesting of the masks so the nested interrupts can be "unwound"
properly.  I'm thinking about it, and since you have been looking at the
functions so closely, if you have any implementation details let us know :-).
Perhaps we could look at the pending and enabled masks and unmask up to the
next pending interrupt to be serviced.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled - problem found

[Steve Rossi]

Dan Malek wrote:

> We should mask all lower priority interrupts, and the challenge is keeping
> the proper nesting of the masks so the nested interrupts can be "unwound"
> properly ...

Thanks for your followup Dan. I've got a rather straightforward "dumb" approach
to fixing this problem - see the patch below. The disclaimer - I'm a hardware
guy
trying to do software, so there might be a much more elegant solution. The idea
here is that the ppc_cached_irq_mask is maintained as is, but in addition to
masking
the interrupt being serviced, all lower priority interrupts are also masked.
This is un-done
in the unmask function, where ppc_cached_irq_mask is also used to ensure we're
not
un-masking interrupts that were masked prior to the call to mask_and_ack. This
doesn't
affect the nesting of interrupts.  I might be completely missing something here,
so any comments?

I've tested this and it seems to work fine - and it even fixes my problem!

Steve


--- ppc8xx_pic.c.orig   Thu Dec 13 08:35:02 2001
+++ ppc8xx_pic.c        Thu Dec 13 08:39:45 2001
@@ -20,6 +20,24 @@
  * but they are overkill for us.
  */

+static unsigned int ppc_irq_priority_mask[NR_IRQS] = { 0x0,
+                                           0x80000000,
+                                           0xC0000000,
+                                           0xE0000000,
+                                           0xF0000000,
+                                           0xF8000000,
+                                           0xFC000000,
+                                           0xFE000000,
+                                           0xFF000000,
+                                           0xFF800000,
+                                           0xFFC00000,
+                                           0xFFE00000,
+                                           0xFFF00000,
+                                           0xFFF80000,
+                                           0xFFFC0000,
+                                           0xFFFE0000
+                                          };
+
 static void m8xx_mask_irq(unsigned int irq_nr)
 {
        int     bit, word;
@@ -41,7 +59,8 @@

        ppc_cached_irq_mask[word] |= (1 << (31-bit));
        ((immap_t *)IMAP_ADDR)->im_siu_conf.sc_simask =
-                                               ppc_cached_irq_mask[word];
+                                               ppc_cached_irq_mask[word] |
+                                                (~ppc_irq_priority_mask[bit] &
ppc_cached_irq_mask[word]);
 }

 static void m8xx_mask_and_ack(unsigned int irq_nr)
@@ -53,7 +72,7 @@

        ppc_cached_irq_mask[word] &= ~(1 << (31-bit));
        ((immap_t *)IMAP_ADDR)->im_siu_conf.sc_simask =
-                                               ppc_cached_irq_mask[word];
+                                               ppc_cached_irq_mask[word] &
ppc_irq_priority_mask[bit];
        ((immap_t *)IMAP_ADDR)->im_siu_conf.sc_sipend = 1 << (31-bit);
 }



--
-------------------------------------------------------
Steven K. Rossi                     srossi@labs.mot.com
Staff Engineer
Multimedia Communications Research Laboratory
Motorola Labs
-------------------------------------------------------


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: high priority interrupts disabled - problem found

[Dan Malek]

Steve Rossi wrote:


> Thanks for your followup Dan. I've got a rather straightforward "dumb" approach
> to fixing this problem - see the patch below.

Looks pretty good.  My last concern is the way we overload the semantics of
the functions depending upon a driver requesting the service and the interrupt
controller using these functions.  For example, a driver will request to
enable or disable one particular interrupt, while the interrupt handling functions
will need to honor the priority, as you have added.  Allowing a driver request
to affect the priority mask may not be appropriate.

Thanks for the effort understanding the problem and finding a solution that
works for you.  It appears to be "more right" than it was :-).


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/