OMAP2430: networking errors

OMAP2430: networking errors

[Budhee Jamaich]

Hello list,

I'm having networking problems with my OMAP.

I tried a few booting schemes, but all failed.
I tried DHCP auto-IP configurations and
NFS root-fs, but they fail to connect to the server.

When I'm running tcpdump on the server,
I can see that the OMAP sends the "arp who-has x.x.x.x",
looking for my server's MAC address,
and I see the server's response: "arp reply x.x.x.x is-at ZZ:ZZ:ZZ:ZZ:ZZ:ZZ (oui
Unknown)",

but the OMAP doesn't see the arp reply,
and keep sending the arp who-has packets... :(

anyone knows why this may be happening ?
or any ideas how to debug this further ?

Thank You All,
Budhee

RE: OMAP2430: networking errors

[Woodruff, Richard]

> but the OMAP doesn't see the arp reply,
> and keep sending the arp who-has packets... :(
> 
> anyone knows why this may be happening ?
> or any ideas how to debug this further ?

Is this a custom board?  What etherent chip is in use?

Does networking work at the u-boot level?

If it is using the standard etherent on SDP's I'd say the gpmc timing
was wrong on your instance.  If its not the same chip add possible
driver error.

Regards,
Richard W.

Re: OMAP2430: networking errors

[Budhee Jamaich]

On 5/17/07, Woodruff, Richard <r-woodruff2@ti.com> wrote:
> > but the OMAP doesn't see the arp reply,
> > and keep sending the arp who-has packets... :(
> Is this a custom board?  What etherent chip is in use?

it's a development board. how do I know which ethernet chip is on it ?

>
> Does networking work at the u-boot level?

Have the same problems, but just sometimes.
I do use tftpboot to load the kernel and fs,
and sometimes I encounter the same problems
(arp-reply packets get ignored).
feels pretty much random behaviour.

Sometimes the DHCP succeeds, too.

>
> If it is using the standard etherent on SDP's I'd say the gpmc timing
> was wrong on your instance.  If its not the same chip add possible
> driver error.

Can you please guide me how do i check the ethernet chip ?
and if it's standard, how do I change the gpmc timing ?

Thank you !
Budhee

Re: OMAP2430: networking errors

[Steve Poulsen]

Budhee,

I suggest you make sure that response is making it to the driver 
properly.  Find out which module is being used for your Ethernet 
controller and rebuild it with debug information.   This is likely a 
#define at the top of the file to set the debug level.  You may need to 
add a bit of code to print out all received packets' data.   I had a 
similar problem and doing this allowed me to trace the problem.  Sorry, 
but my problem was due to a custom board issue so it won't be yours.

Steve

Budhee Jamaich wrote:
> Hello list,
>
> I'm having networking problems with my OMAP.
>
> I tried a few booting schemes, but all failed.
> I tried DHCP auto-IP configurations and
> NFS root-fs, but they fail to connect to the server.
>
> When I'm running tcpdump on the server,
> I can see that the OMAP sends the "arp who-has x.x.x.x",
> looking for my server's MAC address,
> and I see the server's response: "arp reply x.x.x.x is-at 
> ZZ:ZZ:ZZ:ZZ:ZZ:ZZ (oui
> Unknown)",
>
> but the OMAP doesn't see the arp reply,
> and keep sending the arp who-has packets... :(
>
> anyone knows why this may be happening ?
> or any ideas how to debug this further ?
>
> Thank You All,
> Budhee
> _______________________________________________
> Linux-omap-open-source mailing list
> Linux-omap-open-source@linux.omap.com
> http://linux.omap.com/mailman/listinfo/linux-omap-open-source
>

Re: OMAP2430: networking errors

[Budhee Jamaich]

Thanks Richard and Steve,

On 5/17/07, Budhee Jamaich <budheej@gmail.com> wrote:
> Hello list,
>
> the OMAP doesn't see the arp reply,
> and keep sending the arp who-has packets... :(

this is from the kernel boot messages:
smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre <nico@cam.org>
eth0: SMC91C94 (rev 9) at 0xc8864300 IRQ 245 [nowait]
eth0: Ethernet addr: yy:yy:yy:yy:yy:yy

Thanks All
Budhee

Re: OMAP2430: networking errors

[Budhee Jamaich]

On 5/17/07, Woodruff, Richard <r-woodruff2@ti.com> wrote:
> > but the OMAP doesn't see the arp reply,
> > and keep sending the arp who-has packets... :(
> >
> > anyone knows why this may be happening ?
> > or any ideas how to debug this further ?
>
> Is this a custom board?  What etherent chip is in use?

It is SMC LAN91C96. is it standard ?
the driver which is loaded is smc91x.c

>
> Does networking work at the u-boot level?
>
> If it is using the standard etherent on SDP's I'd say the gpmc timing
> was wrong on your instance.  If its not the same chip add possible
> driver error.
>
> Regards,
> Richard W.
>

RE: OMAP2430: networking errors

[Woodruff, Richard]

> it's a development board. how do I know which ethernet chip is on it ?

Ok.  Is it the TI-SDP2430 development board?

If so, what images are you using?  Did you build them or use pre-built
ones.

Why do you report your MAC address as  yy:yy:yy:yy:yy:yy.  Is that what
the driver said or did you remove it for some reason.

If it works sometimes and not other times that is odd.  Are you using
the same MAC address on multiple board or something?

Regards,
Richard W.

> 
> >
> > Does networking work at the u-boot level?
> 
> Have the same problems, but just sometimes.
> I do use tftpboot to load the kernel and fs, and sometimes I 
> encounter the same problems (arp-reply packets get ignored).
> feels pretty much random behaviour.
> 
> Sometimes the DHCP succeeds, too.
> 
> >
> > If it is using the standard etherent on SDP's I'd say the 
> gpmc timing 
> > was wrong on your instance.  If its not the same chip add possible 
> > driver error.
> 
> Can you please guide me how do i check the ethernet chip ?
> and if it's standard, how do I change the gpmc timing ?
> 
> Thank you !
> Budhee
> 

Re: OMAP2430: networking errors

[Budhee Jamaich]

On 5/17/07, Woodruff, Richard <r-woodruff2@ti.com> wrote:
>
> > it's a development board. how do I know which ethernet chip is on it ?
>
> Ok.  Is it the TI-SDP2430 development board?
yes. ethernet chip: SMC 91C96.

>
> If so, what images are you using?  Did you build them or use pre-built
> ones.
I built it. used TI's BSP. it is based on kernel 2.6.10.
I am now cloning myself a copy of Tony's git tree.
I will try to build it, maybe that will help.

otherwise I guess I would have no choice but debug smc91x.c ?

>
> Why do you report your MAC address as  yy:yy:yy:yy:yy:yy.  Is that what
> the driver said or did you remove it for some reason.
oh, you will have to forgive me here.
I usually remove personal data from posts to public mailing lists.

>
> If it works sometimes and not other times that is odd.  Are you using
> the same MAC address on multiple board or something?
hmm, the MAC address is the real hardware one. AFAIK it is
unique for that specific board.

Thank You !
Budhee

Re: OMAP2430: networking errors

[Steve Poulsen]

Budhee,

If you are building your own kernel, I would just open smc91x.c and 
change the

#define SMC_DEBUG 0

to

#define SMC_DEBUG 4

I think that is enough to cause it to print out the packets it 
receives.   When you boot, you should get some printing when it tries to 
DHCP.  It is kind of a one-shot process since you can't mount a 
filesystem via NFS until the network is working.

I cannot help you with that specific board.  Richard may help more 
there.  I can just give you some ideas as to what I have done to track 
these down...the hard way.

Also, I would suggest checking your cables and hubs.  One thing I have 
noticed is that if you mess around changing MAC addresses, the ARP 
tables of various devices will end up wrong.   Unplugging and plugging 
your cable into a new slot of your hub (or power cycling your hub), 
should correct that.  The reason I bring this up is that you said U-boot 
sometimes will not reply to the ARP packet.   This is more disturbing 
than a simple case where a driver always fails.

Finally, I noticed my smc91x.h is patched for OMAP.  Around line 191 it 
looks like this:

#define    SMC_IRQ_FLAGS (( \
           machine_is_omap_h2() \
        || machine_is_omap_h3() \
        || machine_is_omap_h4() \
        || (machine_is_omap_innovator() && !cpu_is_omap1510()) \
    ) ? SA_TRIGGER_FALLING : SA_TRIGGER_RISING)

The patch added the machine_is_omap_h4() line.  This was the only thing 
patched in the driver that seemed relevant.  To be honest, I am not sure 
what is considered omap_h4 (h2, h3 show up in menuconfig and have a 
descriptions as to what they cover), but I would make sure yours looks 
like above, to confirm the proper patching.

I am using 2.6.16 patched for omap:

http://ftp.kernel.org/pub/linux/kernel/v2.6/
http://www.muru.com/linux/omap/patches/old/

Good luck!

Steve


Budhee Jamaich wrote:
> On 5/17/07, Woodruff, Richard <r-woodruff2@ti.com> wrote:
>>
>> > it's a development board. how do I know which ethernet chip is on it ?
>>
>> Ok.  Is it the TI-SDP2430 development board?
> yes. ethernet chip: SMC 91C96.
>
>>
>> If so, what images are you using?  Did you build them or use pre-built
>> ones.
> I built it. used TI's BSP. it is based on kernel 2.6.10.
> I am now cloning myself a copy of Tony's git tree.
> I will try to build it, maybe that will help.
>
> otherwise I guess I would have no choice but debug smc91x.c ?
>
>>
>> Why do you report your MAC address as  yy:yy:yy:yy:yy:yy.  Is that what
>> the driver said or did you remove it for some reason.
> oh, you will have to forgive me here.
> I usually remove personal data from posts to public mailing lists.
>
>>
>> If it works sometimes and not other times that is odd.  Are you using
>> the same MAC address on multiple board or something?
> hmm, the MAC address is the real hardware one. AFAIK it is
> unique for that specific board.
>
> Thank You !
> Budhee
> _______________________________________________
> Linux-omap-open-source mailing list
> Linux-omap-open-source@linux.omap.com
> http://linux.omap.com/mailman/listinfo/linux-omap-open-source
>

Re: OMAP2430: networking errors

[Budhee Jamaich]

On 5/17/07, Steve Poulsen <spoulsen@css-design.us> wrote:
> Also, I would suggest checking your cables and hubs.  One thing I have
> noticed is that if you mess around changing MAC addresses, the ARP
> tables of various devices will end up wrong.   Unplugging and plugging
> your cable into a new slot of your hub (or power cycling your hub),
> should correct that.  The reason I bring this up is that you said U-boot
> sometimes will not reply to the ARP packet.   This is more disturbing
> than a simple case where a driver always fails.

Thank you so much.

Ahh. I wish I was using a hub. somehow those stone-age repeaters always
work. I was using a cheap switch.

I recalled we already had switch problems in our company with another
OMAP. we dismissed that case as a hardware failure. but now I just
guess these boards are somehow fundamentally different than standard
PCs.   I have changed my switch, from some 20$ old one into a decent
piece, and now everything works like a charm (I must say that the
cheap switch works years for me with standard PCs, so I didn't
suspect it).

It'd be interesting to understand why the OMAP has trouble working
with those low cost switches (or the opposite), while standard PCs
seem to work fine with the...   but I guess I won't have the time
to explore it too much... :(

But I'm happy now everything works !

Again, Thank you Steve and Richard for the insights and help !
I learnt a lot all around.
It's so good to have people like you two around.

Bye,
Budhee

RE: OMAP2430: networking errors

[Syed Mohammed, Khasim]

>
>It'd be interesting to understand why the OMAP has trouble working
>with those low cost switches (or the opposite), while standard PCs
>seem to work fine with the...   but I guess I won't have the time
>to explore it too much... :(
>

As I know, the SMC91 driver for OMAP configures the chip in half duplex. Generally PCs will be configured to work Full duplex. Don't know if this is the problem, but giving you some start for reasoning :)

Regards,
Khasim

Re: OMAP2430: networking errors

[Steve Poulsen]

Budhee,

Here are some differences that occur during development and many 
~should~ go away during release.

 - The board is getting powered off and on multiple times. 
 - Each power-on often sets up the SMC91 through u-boot, uses it, then 
it disappears and gets connected again via the kernel (which may have a 
different setup).
 - The developer often has multiple boards with the same MAC address, or 
some boards that have a MAC address that has changed for some reason.
 - Switches maintain the ARP table and I imagine cheap switches don't 
expect the above strange occurrences.

In a product, the booting is often flash-based and the network is 
brought up once by the kernel, with a unique non-changing MAC address.

Add this to Khasim's response below (which would indicate a problem 
all-around), and there's plenty of fun to be had by someone (not me) in 
figuring this out.

Good Luck!

Steve

Syed Mohammed, Khasim wrote:
>> It'd be interesting to understand why the OMAP has trouble working
>> with those low cost switches (or the opposite), while standard PCs
>> seem to work fine with the...   but I guess I won't have the time
>> to explore it too much... :(
>>
>>     
>
> As I know, the SMC91 driver for OMAP configures the chip in half duplex. Generally PCs will be configured to work Full duplex. Don't know if this is the problem, but giving you some start for reasoning :)
>
> Regards,
> Khasim
>
>   

Re: OMAP2430: networking errors

[Budhee Jamaich]

> >> It'd be interesting to understand why the OMAP has trouble working
> >> with those low cost switches (or the opposite), while standard PCs
> >> seem to work fine with the...   but I guess I won't have the time
> >> to explore it too much... :(

> Syed Mohammed, Khasim wrote:
> > As I know, the SMC91 driver for OMAP configures the chip in half duplex. Generally PCs will be configured to work Full duplex. Don't know if this is the problem, but giving you some start for reasoning :)

Hmm. interesting. although, AFAIK, duplex mismatch can cause PHY collisions,
which result in slow down of the PHY channel. it cannot result in anomalies
such as I had (one side sees a packet, the other side ignoring it).
unless I have misfigured something about it ?

On 5/18/07, Steve Poulsen <spoulsen@css-design.us> wrote:
>  - The board is getting powered off and on multiple times.
>  - Each power-on often sets up the SMC91 through u-boot, uses it, then
> it disappears and gets connected again via the kernel (which may have a
> different setup).
>  - The developer often has multiple boards with the same MAC address, or
> some boards that have a MAC address that has changed for some reason.
>  - Switches maintain the ARP table and I imagine cheap switches don't
> expect the above strange occurrences.

True, all of this definately cause problems. But, in my case, I've tried even
to reset the cheap switch after I finished the u-boot session, and just before
I bootm'ed my kernel. now the differences between the two switches were
overwhelming - so this must be something deterministic and consistent.

Thanks all for the ideas !!
Budhee