XLlTemac soft lockup BUG

XLlTemac soft lockup BUG

[khollan]

Hi all,

I am trying to switch my design from the PLB Temac to the local link temac,
and i have a problem when trying to run ifconfig in my init script.  It
gives me a BUG: soft lockup - CPU#0 stuck for 11s! [ifconfig:208]
Here is the rest of the dump, thanks for your help in advance.
[  233.114960] eth0: XLlTemac: Options: 0x3fa

[  233.118449] eth0: XLlTemac: allocating interrupt 8 for fifo mode.

[  237.135029] eth0: XLlTemac: We renegotiated the speed to: 100

[  237.150339] eth0: XLlTemac: speed set to 100Mb/s

[  244.792140] BUG: soft lockup - CPU#0 stuck for 11s! [ifconfig:204]

[  244.797813] NIP: c0188a5c LR: c012d3d0 CTR: c016c3f8

[  244.803533] REGS: ceb1fbf0 TRAP: 1001   Not tainted  (2.6.24-rc8-xlnx)

[  244.811039] MSR: 00029030 <EE,ME,IR,DR>  CR: 22000444  XER: 20000000

[  244.818363] TASK = cf82f7f0[204] 'ifconfig' THREAD: ceb1e000

[  244.824673] GPR00: cebca800 ceb1fca0 cf82f7f0 cebc84a0 cf86c000 00000010
cebc8520 00000000 

[  244.834297] GPR08: 00000000 00000040 cebc84a0 00000001 c0248780 

[  244.841416] NIP [c0188a5c] eth_type_trans+0x10/0x114

[  244.847132] LR [c012d3d0] FifoRecvHandler+0x154/0x1dc

[  244.852945] Call Trace:

[  244.855758] [ceb1fca0] [c012d368] FifoRecvHandler+0xec/0x1dc (unreliable)

[  244.863579] [ceb1fcc0] [c0023b4c] tasklet_action+0x88/0xe4

[  244.869894] [ceb1fcd0] [c0023c20] __do_softirq+0x64/0xd4

[  244.876010] [ceb1fcf0] [c00064f4] do_softirq+0x40/0x58

[  244.881925] [ceb1fd00] [c0023a50] irq_exit+0x38/0x48

[  244.887640] [ceb1fd10] [c000673c] do_IRQ+0x88/0xa4

[  244.893154] [ceb1fd20] [c000325c] ret_from_except+0x0/0x18

[  244.899470] [ceb1fde0] [c0130804] xenet_open+0x24c/0x41c

[  244.905586] [ceb1fe10] [c017aa60] dev_open+0x88/0xf0

[  244.911300] [ceb1fe20] [c017a904] dev_change_flags+0xcc/0x1a0

[  244.917917] [ceb1fe40] [c01b9110] devinet_ioctl+0x298/0x6c0

[  244.924334] [ceb1fea0] [c01b965c] inet_ioctl+0xd0/0x104

[  244.930349] [ceb1feb0] [c016d12c] sock_ioctl+0x22c/0x268

[  244.936465] [ceb1fed0] [c006eb78] do_ioctl+0x38/0x84

[  244.942179] [ceb1fee0] [c006ef48] vfs_ioctl+0x384/0x3b8

[  244.948195] [ceb1ff10] [c006efbc] sys_ioctl+0x40/0x70

[  244.954010] [ceb1ff40] [c0002c10] ret_from_syscall+0x0/0x3c

[  244.960423] Instruction dump:

[  244.963832] 7d005379 3863012c 41820010 7d2534aa 7d2335aa 38c00000
7cc33378 4e800020 

[  244.972752] 81230054 80030090 2b89000d 7c6a1b78 <90030080> 90830018
409d002c 80030058 
-- 
View this message in context: http://www.nabble.com/XLlTemac--soft-lockup-BUG-tp16711066p16711066.html
Sent from the linuxppc-embedded mailing list archive at Nabble.com.

RE: XLlTemac soft lockup BUG

[Hugo Villeneuve]

linuxppc-embedded-bounces+hugo.villeneuve=3Dlyrtech.com@ozlabs.org =
wrote:
> Hi all,
>=20
> I am trying to switch my design from the PLB Temac to the local link
> temac, and i have a problem when trying to run ifconfig in my init
> script.  It gives me a BUG: soft lockup - CPU#0 stuck for 11s!
> [ifconfig:208] Here is the rest of the dump, thanks for your help in
> advance. [  233.114960] eth0: XLlTemac: Options: 0x3fa  =20
>=20
> [  233.118449] eth0: XLlTemac: allocating interrupt 8 for fifo mode.
>=20
> [  237.135029] eth0: XLlTemac: We renegotiated the speed to: 100
>=20
> [  237.150339] eth0: XLlTemac: speed set to 100Mb/s
>=20
> [  244.792140] BUG: soft lockup - CPU#0 stuck for 11s! [ifconfig:204]

Hi,
we had a similar error message, which was caused by us selecting the =
wrong PHY type in the kernel configuration menu (latest =
linux-2.6-xlnx-git tree). In fact, we had to modify the lltemac driver =
to support our PHY (BCM5466). Once we did that, the error message went =
away.

Hugo V.


Hugo Villeneuve
Hardware developer | Concepteur mat=E9riel
Lyrtech
Phone/T=E9l. : (1) (418) 877-4644 #2395
Toll-free/Sans frais - Canada & USA : (1) (888) 922-4644 #2395
Fax/T=E9l=E9c. : (1) (418) 877-7710
www.lyrtech.com
Infinite possibilities...TM

THIS MESSAGE AND ALL ATTACHED DOCUMENTS ARE EXCLUSIVELY INTENDED
TO THE INDICATED RECIPIENTS AND ITS CONTENTS MAY BE CONFIDENTIAL.
IT IS STRICTLY FORBIDDEN TO ANYONE TO TAKE COGNIZANCE, USE, OR
DIVULGE THE INFORMATION CONTAINED HEREIN. IF YOU MISTAKENLY
RECEIVE THIS MESSAGE, IMMEDIATELY INFORM LYRTECH AND DESTROY THE
MESSAGE AND ATTACHMENTS FORTHWITH.THANK YOU.

LE PRESENT MESSAGE ET LES DOCUMENTS QUI Y SONT JOINTS S'ADRESSENT
EXCLUSIVEMENT AU(X)DESTINATAIRE(S) INDIQUE(S) ET LEUR TENEUR PEUT
ETRE CONFIDENTIELLE. IL EST STRICTEMENT INTERDIT A QUICONQUE D'EN
PRENDRE CONNAISSANCE, DE LES UTILISER OU DE LES DIVULGUER. SI
VOUS RECEVEZ LE PRESENT MESSAGE PAR ERREUR, VEUILLEZ EN AVISER
LYRTECH IMMEDIATEMENT ET DETRUIRE LE MESSAGE SEANCE TENANTE,
AINSI QUE LES DOCUMENTS QUI Y SONT JOINTS.

RE: XLlTemac soft lockup BUG

[khollan]

Hugo Villeneuve-3 wrote:
> 
> Hi,
> we had a similar error message, which was caused by us selecting the wrong
> PHY type in the kernel configuration menu (latest linux-2.6-xlnx-git
> tree). In fact, we had to modify the lltemac driver to support our PHY
> (BCM5466). Once we did that, the error message went away.
> 
> Hugo V.
> 
I tried out all 3 configurations and it didn't seem to help.  We are using
the Marvell Phy on our prototype.  The weird thing is it seems to work
perfectly every once in awhile, but not very often.  I seem to have tracked
the problem to the FifoRecvHandler tasklet function in xlltemac_main.c it
gets stuck in this while loop:
while (XLlFifo_RxOccupancy(&lp->Fifo) != 0) {
the XLlFifo_RxOccupancy always reads the same value.
Any ideas?

Thanks
Kevin
-- 
View this message in context: http://www.nabble.com/XLlTemac--soft-lockup-BUG-tp16711066p16734685.html
Sent from the linuxppc-embedded mailing list archive at Nabble.com.

Re: XLlTemac soft lockup BUG

[John Bonesio]

Hi Kevin,

Does xlltemac_main.c have
    #define MARVELL_88E1111_PHY
in it? If not, try setting that.

It's odd that sometimes it works. Does your system meet timing?

We've found that when the PHY is being poked, it can take a while before the PHY settles down and starts working. If the code tries to start using the PHY too soon, then the PHY never seems to get to a good state. So, you might experiement with different values in the udelay() call in _XLlTemac_SetOperatingSpeed().

If you really think there's something wrong in the fifo code, you can enable debug (#define DEBUG) in the following files:
drivers/xilinx_common/xdebug.h
drivers/xilinx_common/xstreamer.h

Perhaps that will shed some light on the problem.

- John


On Wednesday 16 April 2008 15:08, khollan wrote:
> 
> Hugo Villeneuve-3 wrote:
> > 
> > Hi,
> > we had a similar error message, which was caused by us selecting the wrong
> > PHY type in the kernel configuration menu (latest linux-2.6-xlnx-git
> > tree). In fact, we had to modify the lltemac driver to support our PHY
> > (BCM5466). Once we did that, the error message went away.
> > 
> > Hugo V.
> > 
> I tried out all 3 configurations and it didn't seem to help.  We are using
> the Marvell Phy on our prototype.  The weird thing is it seems to work
> perfectly every once in awhile, but not very often.  I seem to have tracked
> the problem to the FifoRecvHandler tasklet function in xlltemac_main.c it
> gets stuck in this while loop:
> while (XLlFifo_RxOccupancy(&lp->Fifo) != 0) {
> the XLlFifo_RxOccupancy always reads the same value.
> Any ideas?
> 
> Thanks
> Kevin
> -- 
> View this message in context: http://www.nabble.com/XLlTemac--soft-lockup-BUG-tp16711066p16734685.html
> Sent from the linuxppc-embedded mailing list archive at Nabble.com.
> 
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> 
> 

Re: XLlTemac soft lockup BUG

[Brian Silverman]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

Hugo,

We have a design with the BCM5482, and will be trying it out shortly.  
Any chance you could post a patch of your BCM5466 mods?

-bri

Hugo Villeneuve wrote:
> linuxppc-embedded-bounces+hugo.villeneuve=lyrtech.com@ozlabs.org wrote:
>   
>> Hi all,
>>
>> I am trying to switch my design from the PLB Temac to the local link
>> temac, and i have a problem when trying to run ifconfig in my init
>> script.  It gives me a BUG: soft lockup - CPU#0 stuck for 11s!
>> [ifconfig:208] Here is the rest of the dump, thanks for your help in
>> advance. [  233.114960] eth0: XLlTemac: Options: 0x3fa   
>>
>> [  233.118449] eth0: XLlTemac: allocating interrupt 8 for fifo mode.
>>
>> [  237.135029] eth0: XLlTemac: We renegotiated the speed to: 100
>>
>> [  237.150339] eth0: XLlTemac: speed set to 100Mb/s
>>
>> [  244.792140] BUG: soft lockup - CPU#0 stuck for 11s! [ifconfig:204]
>>     
>
> Hi,
> we had a similar error message, which was caused by us selecting the wrong PHY type in the kernel configuration menu (latest linux-2.6-xlnx-git tree). In fact, we had to modify the lltemac driver to support our PHY (BCM5466). Once we did that, the error message went away.
>
> Hugo V.
>
>
> Hugo Villeneuve
> Hardware developer | Concepteur matériel
> Lyrtech
> Phone/Tél. : (1) (418) 877-4644 #2395
> Toll-free/Sans frais - Canada & USA : (1) (888) 922-4644 #2395
> Fax/Téléc. : (1) (418) 877-7710
> www.lyrtech.com
> Infinite possibilities...TM
>
> THIS MESSAGE AND ALL ATTACHED DOCUMENTS ARE EXCLUSIVELY INTENDED
> TO THE INDICATED RECIPIENTS AND ITS CONTENTS MAY BE CONFIDENTIAL.
> IT IS STRICTLY FORBIDDEN TO ANYONE TO TAKE COGNIZANCE, USE, OR
> DIVULGE THE INFORMATION CONTAINED HEREIN. IF YOU MISTAKENLY
> RECEIVE THIS MESSAGE, IMMEDIATELY INFORM LYRTECH AND DESTROY THE
> MESSAGE AND ATTACHMENTS FORTHWITH.THANK YOU.
>
> LE PRESENT MESSAGE ET LES DOCUMENTS QUI Y SONT JOINTS S'ADRESSENT
> EXCLUSIVEMENT AU(X)DESTINATAIRE(S) INDIQUE(S) ET LEUR TENEUR PEUT
> ETRE CONFIDENTIELLE. IL EST STRICTEMENT INTERDIT A QUICONQUE D'EN
> PRENDRE CONNAISSANCE, DE LES UTILISER OU DE LES DIVULGUER. SI
> VOUS RECEVEZ LE PRESENT MESSAGE PAR ERREUR, VEUILLEZ EN AVISER
> LYRTECH IMMEDIATEMENT ET DETRUIRE LE MESSAGE SEANCE TENANTE,
> AINSI QUE LES DOCUMENTS QUI Y SONT JOINTS.
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>
>   


-- 
Brian Silverman
Concept X, LLC


[-- Attachment #2: Type: text/html, Size: 3295 bytes --]

RE: XLlTemac soft lockup BUG

[Hugo Villeneuve]

Brian Silverman wrote:
> Hugo,
> We have a design with the BCM5482, and will be trying it out shortly.
> Any chance you could post a patch of your BCM5466 mods?=20

Hi,
The main issue here is that you need to tell the LLTEMAC interface about =
the speed of your PHY. You can do this by adding this code example in =
the function=20
set_mac_speed():

---
#define BROADCOM_BCM5466_RGMII_AUX_STATUS 0x19

#define BCM5466_SPEED_1000BASE_T_FULL_DUPLEX 7
#define BCM5466_SPEED_1000BASE_T_HALF_DUPLEX 6
#define BCM5466_SPEED_100BASE_T_FULL_DUPLEX  5
#define BCM5466_SPEED_100BASE_T4             4
#define BCM5466_SPEED_100BASE_T_HALF_DUPLEX  3
#define BCM5466_SPEED_10BASE_T_FULL_DUPLEX   2
#define BCM5466_SPEED_10BASE_T_HALF_DUPLEX   1
#define BCM5466_SPEED_AN_NOT_COMPLETE        0

        u16 RegValue;

        _XLlTemac_PhyRead(&lp->Emac, lp->gmii_addr, =
BROADCOM_BCM5466_RGMII_AUX_STATUS, &RegValue);
        phylinkspeed =3D ( RegValue & 0x0700 ) >> 8; /* Keeping only =
bits 10-8 */

        /* Update TEMAC speed accordingly */
        switch (phylinkspeed) {
        case BCM5466_SPEED_1000BASE_T_FULL_DUPLEX:
        case BCM5466_SPEED_1000BASE_T_HALF_DUPLEX:
                lp->cur_speed =3D 1000;
                break;
        case BCM5466_SPEED_100BASE_T_FULL_DUPLEX:
        case BCM5466_SPEED_100BASE_T_HALF_DUPLEX:
        case BCM5466_SPEED_100BASE_T4:
                lp->cur_speed =3D 100;
                break;
        case BCM5466_SPEED_10BASE_T_FULL_DUPLEX:
        case BCM5466_SPEED_10BASE_T_HALF_DUPLEX:
                lp->cur_speed =3D 10;
                break;
        default:
                lp->cur_speed =3D 1000;
                break;
        }

        _XLlTemac_SetOperatingSpeed(&lp->Emac, lp->cur_speed);
---

Hope it helps, Hugo.


> Hugo Villeneuve wrote:
>=20
> 	linuxppc-embedded-bounces+hugo.villeneuve=3Dlyrtech.com@ozlabs.org
> wrote:=20
>=20
>=20
> 		Hi all,
>=20
> 		I am trying to switch my design from the PLB Temac to the local link
> 		temac, and i have a problem when trying to run ifconfig in my init
> 		script.  It gives me a BUG: soft lockup - CPU#0 stuck for 11s!
> 		[ifconfig:208] Here is the rest of the dump, thanks for your help in
> 		advance. [  233.114960] eth0: XLlTemac: Options: 0x3fa
>=20
> 		[  233.118449] eth0: XLlTemac: allocating interrupt 8 for fifo mode.
>=20
> 		[  237.135029] eth0: XLlTemac: We renegotiated the speed to: 100
>=20
> 		[  237.150339] eth0: XLlTemac: speed set to 100Mb/s
>=20
> 		[  244.792140] BUG: soft lockup - CPU#0 stuck for 11s!
> [ifconfig:204]=20
>=20
>=20
>=20
> 	Hi,
> 	we had a similar error message, which was caused by us selecting the
> wrong PHY type in the kernel configuration menu (latest
> linux-2.6-xlnx-git tree). In fact, we had to modify the lltemac
> driver to support our PHY (BCM5466). Once we did that, the error
> message went away.   =20
>=20
> 	Hugo V.
>=20
>=20
> 	Hugo Villeneuve
> 	Hardware developer | Concepteur mat=E9riel
> 	Lyrtech
> 	Phone/T=E9l. : (1) (418) 877-4644 #2395
> 	Toll-free/Sans frais - Canada & USA : (1) (888) 922-4644 #2395
> 	Fax/T=E9l=E9c. : (1) (418) 877-7710
> 	www.lyrtech.com
> 	Infinite possibilities...TM
>=20
> 	THIS MESSAGE AND ALL ATTACHED DOCUMENTS ARE EXCLUSIVELY INTENDED
> 	TO THE INDICATED RECIPIENTS AND ITS CONTENTS MAY BE CONFIDENTIAL.
> 	IT IS STRICTLY FORBIDDEN TO ANYONE TO TAKE COGNIZANCE, USE, OR
> 	DIVULGE THE INFORMATION CONTAINED HEREIN. IF YOU MISTAKENLY
> 	RECEIVE THIS MESSAGE, IMMEDIATELY INFORM LYRTECH AND DESTROY THE
> 	MESSAGE AND ATTACHMENTS FORTHWITH.THANK YOU.
>=20
> 	LE PRESENT MESSAGE ET LES DOCUMENTS QUI Y SONT JOINTS S'ADRESSENT
> 	EXCLUSIVEMENT AU(X)DESTINATAIRE(S) INDIQUE(S) ET LEUR TENEUR PEUT
> 	ETRE CONFIDENTIELLE. IL EST STRICTEMENT INTERDIT A QUICONQUE D'EN
> 	PRENDRE CONNAISSANCE, DE LES UTILISER OU DE LES DIVULGUER. SI
> 	VOUS RECEVEZ LE PRESENT MESSAGE PAR ERREUR, VEUILLEZ EN AVISER
> 	LYRTECH IMMEDIATEMENT ET DETRUIRE LE MESSAGE SEANCE TENANTE,
> 	AINSI QUE LES DOCUMENTS QUI Y SONT JOINTS.
> 	_______________________________________________
> 	Linuxppc-embedded mailing list
> 	Linuxppc-embedded@ozlabs.org
> 	https://ozlabs.org/mailman/listinfo/linuxppc-embedded



Hugo Villeneuve
Hardware developer | Concepteur mat=E9riel
Lyrtech
Phone/T=E9l. : (1) (418) 877-4644 #2395
Toll-free/Sans frais - Canada & USA : (1) (888) 922-4644 #2395
Fax/T=E9l=E9c. : (1) (418) 877-7710
www.lyrtech.com
Infinite possibilities...TM

THIS MESSAGE AND ALL ATTACHED DOCUMENTS ARE EXCLUSIVELY INTENDED
TO THE INDICATED RECIPIENTS AND ITS CONTENTS MAY BE CONFIDENTIAL.
IT IS STRICTLY FORBIDDEN TO ANYONE TO TAKE COGNIZANCE, USE, OR
DIVULGE THE INFORMATION CONTAINED HEREIN. IF YOU MISTAKENLY
RECEIVE THIS MESSAGE, IMMEDIATELY INFORM LYRTECH AND DESTROY THE
MESSAGE AND ATTACHMENTS FORTHWITH.THANK YOU.

LE PRESENT MESSAGE ET LES DOCUMENTS QUI Y SONT JOINTS S'ADRESSENT
EXCLUSIVEMENT AU(X)DESTINATAIRE(S) INDIQUE(S) ET LEUR TENEUR PEUT
ETRE CONFIDENTIELLE. IL EST STRICTEMENT INTERDIT A QUICONQUE D'EN
PRENDRE CONNAISSANCE, DE LES UTILISER OU DE LES DIVULGUER. SI
VOUS RECEVEZ LE PRESENT MESSAGE PAR ERREUR, VEUILLEZ EN AVISER
LYRTECH IMMEDIATEMENT ET DETRUIRE LE MESSAGE SEANCE TENANTE,
AINSI QUE LES DOCUMENTS QUI Y SONT JOINTS.

Re: XLlTemac soft lockup BUG

[khollan]

Hi,

I have done some more digging and it seems like whenever the system works
the command XLlFifo_IsRxEmpty(&lp->Fifo)) returns a 1 meaning that the
RxFifo is empty.  I stuck this test in the xenet_open function right before
the fifo interupt gets enabled.

When ever this test comes back as 0 the ifconfig will get into the soft
lockup I described above.

Is there a function that empties out the fifo, I tried XLlFifo_RxReset but
that doesn't seem to work.

Thanks

Kevin
-- 
View this message in context: http://www.nabble.com/XLlTemac--soft-lockup-BUG-tp16711066p16834738.html
Sent from the linuxppc-embedded mailing list archive at Nabble.com.

Re: XLlTemac soft lockup BUG

[John Bonesio]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

Hi Kevin,

When the system locks up, does it get stuck in the FifoRecvHandler()
routine?

How many bytes are returned from XLlFifo_RxGetLen(&lp->Fifo) in that
routine?

You can try calling XLlFifo_Reset(&lp->Fifo); to see if that clears the
fifo. Though, I suspect that this wont fix the root cause of the problem
you're seeing.

- John


On Wed, 2008-04-23 at 11:09 -0700, khollan wrote:

> Hi,
> 
> I have done some more digging and it seems like whenever the system works
> the command XLlFifo_IsRxEmpty(&lp->Fifo)) returns a 1 meaning that the
> RxFifo is empty.  I stuck this test in the xenet_open function right before
> the fifo interupt gets enabled.
> 
> When ever this test comes back as 0 the ifconfig will get into the soft
> lockup I described above.
> 
> Is there a function that empties out the fifo, I tried XLlFifo_RxReset but
> that doesn't seem to work.
> 
> Thanks
> 
> Kevin



[-- Attachment #2: Type: text/html, Size: 1616 bytes --]