FEC_IEVENT_RFIFO_ERROR

FEC_IEVENT_RFIFO_ERROR

[Babarovic Ivica]

Hi!

I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
for MPC5200 chip on a custom board that is almost lite5200 compatible.
I noticed a couple of times I have a strange error at bootup.
It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
went trough without problems but since today system just hangs.
Sometimes with several printouts of this error.
---boot sequence ------
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
....

I traced a problem a bit and found that this happenes at
mpc52xx_fec_probe() function in fec.c at this point:
-----------------------------------------------------------------------------------------
       /* Get the IRQ we need one by one */
                /* Control */
        dev->irq = ocp->def->irq;
-->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT, 
                        "mpc52xx_fec_ctrl", dev)) {
                printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request 
failed\n");
                ret = -EBUSY;
                dev->irq = -1;  /* Don't try to free it */
                goto probe_error;
        }
------------------------------------------------------------------------------------------

When fec_interrupt() is called also from fec.c.

------------------------------------------------------------------------------------------
static irqreturn_t fec_interrupt(int irq, void *dev_id, struct pt_regs 
*regs)
{
        struct net_device *dev = (struct net_device *)dev_id;
        struct fec_priv *priv = (struct fec_priv *)dev->priv;
        struct mpc52xx_fec *fec = priv->fec;
        int ievent;

        ievent = in_be32(&fec->ievent);
        out_be32(&fec->ievent, ievent);         /* clear pending events */

        if (ievent & (FEC_IEVENT_RFIFO_ERROR | FEC_IEVENT_XFIFO_ERROR)) {
                if (ievent & FEC_IEVENT_RFIFO_ERROR)
-->                        printk(KERN_WARNING "FEC_IEVENT_RFIFO_ERROR\n");
                if (ievent & FEC_IEVENT_XFIFO_ERROR)
                        printk(KERN_WARNING "FEC_IEVENT_XFIFO_ERROR\n");
                fec_reinit(dev);
        }
        else if (ievent & FEC_IEVENT_MII)
                fec_mii(dev);
        return IRQ_HANDLED;
}
-------------------------------------------------------------------------------------------------


This is what I found in MPC5200 Users Manual:
Receive FIFO Error--indicates error occurred within the forest green version
RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared,
halting FEC frame processing. When this occurs, software must ensure both
the FIFO Controller and BestComm are soft-reset.

Any ideas on what could be causing this?

Re: FEC_IEVENT_RFIFO_ERROR

[Sylvain Munaut]

Hi

> I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
> for MPC5200 chip on a custom board that is almost lite5200 compatible.
> I noticed a couple of times I have a strange error at bootup.
> It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
> went trough without problems but since today system just hangs.
> Sometimes with several printouts of this error.
> ---boot sequence ------
> FEC_IEVENT_RFIFO_ERROR
> FEC_IEVENT_RFIFO_ERROR
> FEC_IEVENT_RFIFO_ERROR
> ....

Theses are definitly not "normal" but you said "since today it just hangs",
did something change in the environment of the card ?

> I traced a problem a bit and found that this happenes at
> mpc52xx_fec_probe() function in fec.c at this point:
> ----------------------------------------------------------------------------------------- 
>
>       /* Get the IRQ we need one by one */
>                /* Control */
>        dev->irq = ocp->def->irq;
> -->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT, 
>                        "mpc52xx_fec_ctrl", dev)) {
>                printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request 
> failed\n");
>                ret = -EBUSY;
>                dev->irq = -1;  /* Don't try to free it */
>                goto probe_error;
>        }
> ------------------------------------------------------------------------------------------ 
>

It ovbiously can't happen before since the message it printed in that 
interrupt
handler. But it should not happen there either (not so early) !

This error globally says : "Somthing got wrong with the receive buffer". But
at this point, frame reception is not yet enabled, how could it go wrong 
? Unless
your bootloader don't take care of shutting down the fec, then frames 
may be
stuck in the fifo between the bootloader and the fec init ...

> This is what I found in MPC5200 Users Manual:
> Receive FIFO Error--indicates error occurred within the forest green 
> version
> RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared,
> halting FEC frame processing. When this occurs, software must ensure both
> the FIFO Controller and BestComm are soft-reset.
>
> Any ideas on what could be causing this?

I can't explain why this happen so early at init (as I said before) but 
other things that could
cause such an event :
 - We don't have enough buffer descriptors : The bescomm task just fill 
them all and runs out
   of them before the interrupt is handled.
 - The bestcomm engine don't flush the RX fifo quicly enough. Currently 
the only tasks
 - Bestcomm stopped processing for whatever reason ...
 - Something else that I don't see at the moment. I'll try to "stress 
test" network a little bit,
 see if I can reproduce the issue.

In the mean time, pull the latest change, I just pushed some fixes 
related to frame reception,
I don't think it's related to your issue but ...



    Sylvain

Re: FEC_IEVENT_RFIFO_ERROR

[Babarovic Ivica]

Sylvain Munaut wrote:

> Hi
>
>> I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
>> for MPC5200 chip on a custom board that is almost lite5200 compatible.
>> I noticed a couple of times I have a strange error at bootup.
>> It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
>> went trough without problems but since today system just hangs.
>> Sometimes with several printouts of this error.
>> ---boot sequence ------
>> FEC_IEVENT_RFIFO_ERROR
>> FEC_IEVENT_RFIFO_ERROR
>> FEC_IEVENT_RFIFO_ERROR
>> ....
>
>
> Theses are definitly not "normal" but you said "since today it just 
> hangs",
> did something change in the environment of the card ?


I changed the card yesterday and I got fresh bootloader on. I don't compile
bootloader, I receive it with the board. Maybe I should change this 
practice.
Anyway as I said I got the new board yesterday with some hardware fixes 
and I
managed to get it up after I set up bootloader environment. It worked 
until this mornig.
I was working on my ( noob :D ) custom drivers and executed another
cycle of make/copy image/reboot  and noticed that for some reason
fec.c and fec_phy.c got also recompiled. After that, things stoped working.
I really don't know why those two got into the make routine but that was
the end of fun. :D
I'm stuck at this error now and lot's of new stuff to learn. :D I don't 
mind that
though although I got completely of track with what I was doing before.

>
>> I traced a problem a bit and found that this happenes at
>> mpc52xx_fec_probe() function in fec.c at this point:
>> ----------------------------------------------------------------------------------------- 
>>
>>       /* Get the IRQ we need one by one */
>>                /* Control */
>>        dev->irq = ocp->def->irq;
>> -->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT, 
>>                        "mpc52xx_fec_ctrl", dev)) {
>>                printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request 
>> failed\n");
>>                ret = -EBUSY;
>>                dev->irq = -1;  /* Don't try to free it */
>>                goto probe_error;
>>        }
>> ------------------------------------------------------------------------------------------ 
>>
>
>
> It ovbiously can't happen before since the message it printed in that 
> interrupt
> handler. But it should not happen there either (not so early) !
>
> This error globally says : "Somthing got wrong with the receive 
> buffer". But
> at this point, frame reception is not yet enabled, how could it go 
> wrong ? Unless
> your bootloader don't take care of shutting down the fec, then frames 
> may be
> stuck in the fifo between the bootloader and the fec init ...

I think I understand what you're saying.
Hmmm. I really don't know what bootloader takes care of.
What should I do to check this? Should I try with another
version of bootloader? BTW I got the board with U-Boot 1.1.1.

>
>> This is what I found in MPC5200 Users Manual:
>> Receive FIFO Error--indicates error occurred within the forest green 
>> version
>> RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared,
>> halting FEC frame processing. When this occurs, software must ensure 
>> both
>> the FIFO Controller and BestComm are soft-reset.
>>
>> Any ideas on what could be causing this?
>
>
> I can't explain why this happen so early at init (as I said before) 
> but other things that could
> cause such an event :
> - We don't have enough buffer descriptors : The bescomm task just fill 
> them all and runs out
>   of them before the interrupt is handled.
> - The bestcomm engine don't flush the RX fifo quicly enough. Currently 
> the only tasks
> - Bestcomm stopped processing for whatever reason ...
> - Something else that I don't see at the moment. I'll try to "stress 
> test" network a little bit,
> see if I can reproduce the issue.
>
> In the mean time, pull the latest change, I just pushed some fixes 
> related to frame reception,
> I don't think it's related to your issue but ...
>
Ok. I will try with this now.

Re: FEC_IEVENT_RFIFO_ERROR

[Babarovic Ivica]

Sylvain Munaut wrote:

>
>
> In the mean time, pull the latest change, I just pushed some fixes 
> related to frame reception,
> I don't think it's related to your issue but ...
>
>
>
>    Sylvain
>
No luck unfortunatly. I pulled/recompiled and still have the same error. :(

Re: FEC_IEVENT_RFIFO_ERROR

[Dale Farnsworth]

On Thu, Mar 03, 2005 at 01:14:54PM +0000, Babarovic Ivica wrote:
> I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
> for MPC5200 chip on a custom board that is almost lite5200 compatible.
> I noticed a couple of times I have a strange error at bootup.
> It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
> went trough without problems but since today system just hangs.
> Sometimes with several printouts of this error.
> ---boot sequence ------
> FEC_IEVENT_RFIFO_ERROR
> FEC_IEVENT_RFIFO_ERROR
> FEC_IEVENT_RFIFO_ERROR
> ....
> 
> I traced a problem a bit and found that this happenes at
> mpc52xx_fec_probe() function in fec.c at this point:
> --------------------------------------------------------------------------
>       /* Get the IRQ we need one by one */
>                /* Control */
>        dev->irq = ocp->def->irq;
> -->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT, 
>                        "mpc52xx_fec_ctrl", dev)) {
>                printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request 
> failed\n");
>                ret = -EBUSY;
>                dev->irq = -1;  /* Don't try to free it */
>                goto probe_error;
>        }
> --------------------------------------------------------------------------

It looks like the bootloader left the FEC enabled.  You might try
the following (untested) patch which resets the FEC and disables
its interrupts before requesting the irq.

-Dale

===== drivers/net/fec_mpc52xx/fec.c 1.1 vs edited =====
--- 1.1/drivers/net/fec_mpc52xx/fec.c	2004-11-20 15:26:33 -07:00
+++ edited/drivers/net/fec_mpc52xx/fec.c	2005-03-03 09:43:08 -07:00
@@ -85,18 +85,13 @@
 	return 0;
 }
 
-/* This function is called to start or restart the FEC during a link
- * change.  This happens on fifo errors or when switching between half
- * and full duplex.
- */
-static void fec_restart(struct net_device *dev, int duplex)
+static void fec_reset(struct net_device *dev)
 {
 	struct fec_priv *priv = (struct fec_priv *)dev->priv;
 	struct mpc52xx_fec *fec = priv->fec;
-	u32 rcntrl;
-	u32 tcntrl;
 	int i;
 
+	out_be32(&priv->fec->imask, 0);		/* mask all interrupts */
 	out_be32(&fec->rfifo_status, in_be32(&fec->rfifo_status) & 0x700000);
 	out_be32(&fec->tfifo_status, in_be32(&fec->tfifo_status) & 0x700000);
 	out_be32(&fec->reset_cntrl, 0x1000000);
@@ -110,6 +105,20 @@
 	}
 	if (i == FEC_RESET_DELAY)
 		printk (KERN_ERR "FEC Reset timeout!\n");
+}
+
+/* This function is called to start or restart the FEC during a link
+ * change.  This happens on fifo errors or when switching between half
+ * and full duplex.
+ */
+static void fec_restart(struct net_device *dev, int duplex)
+{
+	struct fec_priv *priv = (struct fec_priv *)dev->priv;
+	struct mpc52xx_fec *fec = priv->fec;
+	u32 rcntrl;
+	u32 tcntrl;
+
+	fec_reset(dev);
 
 	/* Set station address. */
 	fec_set_paddr(dev, dev->dev_addr);
@@ -645,6 +654,8 @@
 	ret = sdma_fec_tx_init(priv->tx_sdma, priv->tx_fifo);
 	if (ret < 0)
 		goto probe_error;
+
+	fec_reset(dev);
 
 	/* Get the IRQ we need one by one */
 		/* Control */

Re: FEC_IEVENT_RFIFO_ERROR

[Babarovic Ivica]

Hi!

I tried the patch you've sent and It's still the same.
FEC_IEVENT_RFIFO_ERROR is still here.
Can you/anyone think of something that I could do to make
analyzing this problem easier?



Dale Farnsworth wrote:

>On Thu, Mar 03, 2005 at 01:14:54PM +0000, Babarovic Ivica wrote:
>  
>
>>I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
>>for MPC5200 chip on a custom board that is almost lite5200 compatible.
>>I noticed a couple of times I have a strange error at bootup.
>>It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
>>went trough without problems but since today system just hangs.
>>Sometimes with several printouts of this error.
>>---boot sequence ------
>>FEC_IEVENT_RFIFO_ERROR
>>FEC_IEVENT_RFIFO_ERROR
>>FEC_IEVENT_RFIFO_ERROR
>>....
>>
>>I traced a problem a bit and found that this happenes at
>>mpc52xx_fec_probe() function in fec.c at this point:
>>--------------------------------------------------------------------------
>>      /* Get the IRQ we need one by one */
>>               /* Control */
>>       dev->irq = ocp->def->irq;
>>-->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT, 
>>                       "mpc52xx_fec_ctrl", dev)) {
>>               printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request 
>>failed\n");
>>               ret = -EBUSY;
>>               dev->irq = -1;  /* Don't try to free it */
>>               goto probe_error;
>>       }
>>--------------------------------------------------------------------------
>>    
>>
>
>It looks like the bootloader left the FEC enabled.  You might try
>the following (untested) patch which resets the FEC and disables
>its interrupts before requesting the irq.
>
>-Dale
>
>===== drivers/net/fec_mpc52xx/fec.c 1.1 vs edited =====
>--- 1.1/drivers/net/fec_mpc52xx/fec.c	2004-11-20 15:26:33 -07:00
>+++ edited/drivers/net/fec_mpc52xx/fec.c	2005-03-03 09:43:08 -07:00
>@@ -85,18 +85,13 @@
> 	return 0;
> }
> 
>-/* This function is called to start or restart the FEC during a link
>- * change.  This happens on fifo errors or when switching between half
>- * and full duplex.
>- */
>-static void fec_restart(struct net_device *dev, int duplex)
>+static void fec_reset(struct net_device *dev)
> {
> 	struct fec_priv *priv = (struct fec_priv *)dev->priv;
> 	struct mpc52xx_fec *fec = priv->fec;
>-	u32 rcntrl;
>-	u32 tcntrl;
> 	int i;
> 
>+	out_be32(&priv->fec->imask, 0);		/* mask all interrupts */
> 	out_be32(&fec->rfifo_status, in_be32(&fec->rfifo_status) & 0x700000);
> 	out_be32(&fec->tfifo_status, in_be32(&fec->tfifo_status) & 0x700000);
> 	out_be32(&fec->reset_cntrl, 0x1000000);
>@@ -110,6 +105,20 @@
> 	}
> 	if (i == FEC_RESET_DELAY)
> 		printk (KERN_ERR "FEC Reset timeout!\n");
>+}
>+
>+/* This function is called to start or restart the FEC during a link
>+ * change.  This happens on fifo errors or when switching between half
>+ * and full duplex.
>+ */
>+static void fec_restart(struct net_device *dev, int duplex)
>+{
>+	struct fec_priv *priv = (struct fec_priv *)dev->priv;
>+	struct mpc52xx_fec *fec = priv->fec;
>+	u32 rcntrl;
>+	u32 tcntrl;
>+
>+	fec_reset(dev);
> 
> 	/* Set station address. */
> 	fec_set_paddr(dev, dev->dev_addr);
>@@ -645,6 +654,8 @@
> 	ret = sdma_fec_tx_init(priv->tx_sdma, priv->tx_fifo);
> 	if (ret < 0)
> 		goto probe_error;
>+
>+	fec_reset(dev);
> 
> 	/* Get the IRQ we need one by one */
> 		/* Control */
>
>
>
>  
>

Re: FEC_IEVENT_RFIFO_ERROR

[Sylvain Munaut]

Hi

>
> I tried the patch you've sent and It's still the same.
> FEC_IEVENT_RFIFO_ERROR is still here.
> Can you/anyone think of something that I could do to make
> analyzing this problem easier?
>
Resetting the FEC early is a good thing so that patch should be there 
anyway, so we don't setup irq handlers for a hardware in unknow state.

However looking at fec code, there is another problem with it : The FEC 
is enabled in probe, once for all but the DMA tasks that should empty
the buffers are only enabled when the interface is up, so during it's 
down time, any received packed ends up in filling the FIFO ...

This problem goes away when the interface is up tough, so you should not 
experience 'crash' ... Do you nfs boot ? Try with a static image, just
to see, maybe having the error at boot screw something. If it doesn't 
boot with a static image then there is still another problem.


    Sylvain

Re: FEC_IEVENT_RFIFO_ERROR

[Babarovic Ivica]

Hi Sylvain, Dale and others!

Well, I went for a 5 minute walk to get some fresh air and gather my 
thoughts.
To put it shortly I'm mad as hell. I can't believe what a noob mistake 
I've made.
In one of my drivers I had a for loop that loops trough some IO range  
to discover
if hardware is there and instead  of  this for-expression  (i=0; i<31; 
i+=2), I wrote
this one (i=0; i<31; i=+2) . The difference is in =+ and += . Variable i 
never got
incremented past 2 and of course there was chaos because loop was 
running to infinity. :)
Function I was calling in 'infinite' runaway loop did some input and 
output with a custom
bus and read something from a GPIO register. Here it is:
--------------------------------------------------------------
static inline int ebus_echo(char * io)
{
       struct mpc52xx_gpio *gpio;
       volatile unsigned char dummy;

       gpio = (struct mpc52xx_gpio *)MPC52xx_GPIO;

       out_8(RESET_ECHO, 0);
       dummy=in_8(io);
       if (in_8(&gpio->sint_ival) & 0x02)
               return 1;
       return 0;
}
-----------------------------------------------------------------

This driver was initializing itself just before the NIC. I wonder how 
ethernet init
even started and got to the RFIFO part since there was an infinite loop 
running before it?
What a productive day I just had. A stupid typo cost me 12 hours of work 
and I have
nothing to show for. I apologize and I only hope something good comes 
out of this.

/me ducks

Best regards
I.


Sylvain Munaut wrote:

Hi

>
> I tried the patch you've sent and It's still the same.
> FEC_IEVENT_RFIFO_ERROR is still here.
> Can you/anyone think of something that I could do to make
> analyzing this problem easier?
>
Resetting the FEC early is a good thing so that patch should be there 
anyway, so we don't setup irq handlers for a hardware in unknow state.

However looking at fec code, there is another problem with it : The FEC 
is enabled in probe, once for all but the DMA tasks that should empty
the buffers are only enabled when the interface is up, so during it's 
down time, any received packed ends up in filling the FIFO ...

This problem goes away when the interface is up tough, so you should not 
experience 'crash' ... Do you nfs boot ? Try with a static image, just
to see, maybe having the error at boot screw something. If it doesn't 
boot with a static image then there is still another problem.


   Sylvain