8390.c workaround for interrupt storm

8390.c workaround for interrupt storm

[Marcelo Tosatti]

Hi netdev, 

We've been seeing an interrupt storm on our 
embedded PowerPC systems with Linksys PCMLM56 (network+modem) 
PCMCIA cards (network part of it driven by pcnet_cs.c) as soon as the network 
cable is  pluged/unpluged and after some activity on the serial
line. 

The card goes nuts, and doesnt stop sending interrupts. 

So I hacked up this patch which resets the 8390 chip
as soon as 256 empty interrupts are triggered. Works fine,
the card comes back to sanity after resetting.

Not sure if its specific to our boards, but I believe
it might happen on other archs also, so I'm posting it 
here in the hope it might be useful for someone with the
same problem.

Not likely to be included in mainline driver, but hey, why not,
its protection against hardware insanity.

--- 8390.c.orig 2004-08-13 11:54:28.000000000 -0300
+++ 8390.c      2004-08-13 11:54:07.000000000 -0300
@@ -411,6 +411,8 @@
  * needed.
  */
  
+int interrupt_cnt = 0;
+
 void ei_interrupt(int irq, void *dev_id, struct pt_regs * regs)
 {
        struct net_device *dev = dev_id;
@@ -493,6 +495,22 @@
  
                outb_p(E8390_NODMA+E8390_PAGE0+E8390_START, e8390_base + E8390_CMD);
        }
+
+       /*
+        * Workaround infinite interrupt storm caused by Linksys combocard
+        * on embedded PowerPC. Other cards/platform can suffer
+        * from the same. Reset the device if we get more than 256
+        * empty interrupts.
+        */
+       if (!nr_serviced)
+               interrupt_cnt++;
+       else
+               interrupt_cnt = 0;
+
+       if (interrupt_cnt > 256) {
+               ei_status.reset_8390(dev);
+               interrupt_cnt = 0;
+       }
      
        if (interrupts && ei_debug)
        {

Re: 8390.c workaround for interrupt storm

[Jeff Garzik]

Marcelo Tosatti wrote:
> Hi netdev, 
> 
> We've been seeing an interrupt storm on our 
> embedded PowerPC systems with Linksys PCMLM56 (network+modem) 
> PCMCIA cards (network part of it driven by pcnet_cs.c) as soon as the network 
> cable is  pluged/unpluged and after some activity on the serial
> line. 
> 
> The card goes nuts, and doesnt stop sending interrupts. 
> 
> So I hacked up this patch which resets the 8390 chip
> as soon as 256 empty interrupts are triggered. Works fine,
> the card comes back to sanity after resetting.
> 
> Not sure if its specific to our boards, but I believe
> it might happen on other archs also, so I'm posting it 
> here in the hope it might be useful for someone with the
> same problem.
> 
> Not likely to be included in mainline driver, but hey, why not,
> its protection against hardware insanity.


You have stumbled on a common class of problems, I think.

IMO interrupt storms are almost always caused by the platform or bus.  I 
see tons of bug reports for ethernet drivers which turn out to be 
interrupt routing, interrupt handling, or interrupt storms unrelated to 
the ethernet driver itself.

Yes, you can sometimes work around this in the driver, but that tends to 
wind up hiding the problem.

PCMCIA debugging isn't fun, to be sure, maybe RMK has some ideas?

	Jeff

Re: 8390.c workaround for interrupt storm

[Marcelo Tosatti]

On Fri, Aug 13, 2004 at 04:17:34PM -0400, Jeff Garzik wrote:
> Marcelo Tosatti wrote:
> >Hi netdev, 
> >
> >We've been seeing an interrupt storm on our 
> >embedded PowerPC systems with Linksys PCMLM56 (network+modem) 
> >PCMCIA cards (network part of it driven by pcnet_cs.c) as soon as the 
> >network cable is  pluged/unpluged and after some activity on the serial
> >line. 
> >
> >The card goes nuts, and doesnt stop sending interrupts. 
> >
> >So I hacked up this patch which resets the 8390 chip
> >as soon as 256 empty interrupts are triggered. Works fine,
> >the card comes back to sanity after resetting.
> >
> >Not sure if its specific to our boards, but I believe
> >it might happen on other archs also, so I'm posting it 
> >here in the hope it might be useful for someone with the
> >same problem.
> >
> >Not likely to be included in mainline driver, but hey, why not,
> >its protection against hardware insanity.

Hi Jeff!

> You have stumbled on a common class of problems, I think.
> 
> IMO interrupt storms are almost always caused by the platform or bus.  I 
> see tons of bug reports for ethernet drivers which turn out to be 
> interrupt routing, interrupt handling, or interrupt storms unrelated to 
> the ethernet driver itself.
> 
> Yes, you can sometimes work around this in the driver, but that tends to 
> wind up hiding the problem.

I see. So, following your thoughts, what is causing the interrupt storm 
here is not the card itself, but the bus/platform/irq controller? That is,
those interrupts are not coming from the card.

Or that something on the bus/platform/irq controller triggers the card
into insanity, and the card really is sending those interrupts? 

Since resetting the 8390 stops it from madness, its likely that 
the card itself is sending the interrupts, no?

Interesting...

> PCMCIA debugging isn't fun, to be sure, maybe RMK has some ideas?

This is a MontaVista v2.4.17 kernel patched up, so I doubt he will care much.

I'll ask him anyway.

Thanks for answering!!