Advice needed WRT very slow nfs in new port...

Advice needed WRT very slow nfs in new port...

[David Daney]

I am trying to port 2.6.12 to a 32 bit SOC (ATI Xilleon).  The same 
system is current running 2.4.29 with no problems.

I have an Intel Pro/100M Ethernet adapter on the PCI bus that I use to 
mount the root file system via nfs.  There is also an ohci usb adapter 
on the bus.

My problem is that with my new 2.6.12 port the NFS is very slow.  It 
takes about 10 minutes to mount the root filesystem, run init and 
finally run a shell on the serial port and present the shell prompt. 
Running commands like 'ls' works but can take 15 seconds in a small 
directory.

On the same hardware running 2.4.29 the system boots in 15 seconds and 
there is no delay for the 'ls'.

One theory I have is that there is a problem in either the interrupt or 
timer code somewhere.  My evidence for this is that if I repeatedly plug 
and unplug a usb memory device things run more quickly.  I think the 
interrupts from the USB may be kicking things into action.

Any pointers about where to look would be most appreciated.

Thanks
David Daney

Problems with Intel e100 driver on new MIPS port, was: Advice needed WRT very slow nfs in new port...

[David Daney]

David Daney wrote:
> I am trying to port 2.6.12 to a 32 bit SOC (ATI Xilleon).  The same 
> system is current running 2.4.29 with no problems.
> 
> I have an Intel Pro/100M Ethernet adapter on the PCI bus that I use to 
> mount the root file system via nfs.  There is also an ohci usb adapter 
> on the bus.
> 
> My problem is that with my new 2.6.12 port the NFS is very slow.  It 
> takes about 10 minutes to mount the root filesystem, run init and 
> finally run a shell on the serial port and present the shell prompt. 
> Running commands like 'ls' works but can take 15 seconds in a small 
> directory.
> 
> On the same hardware running 2.4.29 the system boots in 15 seconds and 
> there is no delay for the 'ls'.
> 
> One theory I have is that there is a problem in either the interrupt or 
> timer code somewhere.  My evidence for this is that if I repeatedly plug 
> and unplug a usb memory device things run more quickly.  I think the 
> interrupts from the USB may be kicking things into action.
> 
> Any pointers about where to look would be most appreciated.

Here is an update:

The CPU is a 32 bit MIPS 4Ke core (little endian).

My problem is only experienced with the drivers/net/e100.c driver.  When 
I use an RTL8100 with the 8139too driver the network seems to work fine.

The Intel driver reports to be:
e100: Intel(R) PRO/100 Network Driver, 3.4.8-k2-NAPI

I am just bringing the 2.6.12 kernel up on the board so it is quite 
possible that I have screwed something up in the board specific portion 
of the kernel.

When using the Intel Pro/100M NIC the network seems to work fine in all 
respects except for the speed issue.  If i run ping (either from the 
board or to the board from an external host) I get round trip times of 
almost exactly 1000mS.

With the same Intel NIC and the 2.4.29 kernel using the e100 driver I 
have no problems, so I don't think it is a hardware issue.

Does anyone have any idea what would cause 1000mS delay?

Thanks in advance for any insight,
David Daney

Re: Problems with Intel e100 driver on new MIPS port, was: Advice needed WRT very slow nfs in new port...

[M. Warner Losh]

In message: <42C34C4D.9020902@avtrex.com>
            David Daney <ddaney@avtrex.com> writes:
: Does anyone have any idea what would cause 1000mS delay?

That's remarkably close to 1s.  This often indicates that the transmit
of your next packet is causing the receive buffer to empty.  This is
usually due to blocked interrupts, or a failure to enable interrupts.

Warner

Re: Problems with Intel e100 driver on new MIPS port, was: Advice needed WRT very slow nfs in new port...

[David Daney]

M. Warner Losh wrote:
> In message: <42C34C4D.9020902@avtrex.com>
>             David Daney <ddaney@avtrex.com> writes:
> : Does anyone have any idea what would cause 1000mS delay?
> 
> That's remarkably close to 1s.  This often indicates that the transmit
> of your next packet is causing the receive buffer to empty.  This is
> usually due to blocked interrupts, or a failure to enable interrupts.
> 

But I observe ever increasing counts for the device in /proc/interrupts. 
  So the interrupts are working somewhat.

David Daney

Re: Problems with Intel e100 driver on new MIPS port, was: Advice needed WRT very slow nfs in new port...

[M. Warner Losh]

In message: <42C359F8.4060000@avtrex.com>
            David Daney <ddaney@avtrex.com> writes:
: M. Warner Losh wrote:
: > In message: <42C34C4D.9020902@avtrex.com>
: >             David Daney <ddaney@avtrex.com> writes:
: > : Does anyone have any idea what would cause 1000mS delay?
: > 
: > That's remarkably close to 1s.  This often indicates that the transmit
: > of your next packet is causing the receive buffer to empty.  This is
: > usually due to blocked interrupts, or a failure to enable interrupts.
: > 
: 
: But I observe ever increasing counts for the device in /proc/interrupts. 
:   So the interrupts are working somewhat.

Are you sure that you've routed the interrupts correctly?  Maybe those
interrupts are 'really' for a different device....

Warner

Re: Problems with Intel e100 driver on new MIPS port, was: Advice needed WRT very slow nfs in new port...

[Michael Stickel]

M. Warner Losh wrote:

>In message: <42C359F8.4060000@avtrex.com>
>            David Daney <ddaney@avtrex.com> writes:
>: M. Warner Losh wrote:
>: > In message: <42C34C4D.9020902@avtrex.com>
>: >             David Daney <ddaney@avtrex.com> writes:
>: > : Does anyone have any idea what would cause 1000mS delay?
>: > 
>: > That's remarkably close to 1s.  This often indicates that the transmit
>: > of your next packet is causing the receive buffer to empty.  This is
>: > usually due to blocked interrupts, or a failure to enable interrupts.
>: > 
>: 
>: But I observe ever increasing counts for the device in /proc/interrupts. 
>:   So the interrupts are working somewhat.
>
>Are you sure that you've routed the interrupts correctly?  Maybe those
>interrupts are 'really' for a different device....
>  
>
Add some debugging to the interrupt routine of the e100 and see what 
happens.

Michael

Re: Problems with Intel e100 driver on new MIPS port, was: Advice needed WRT very slow nfs in new port...

[David Daney]

Michael Stickel wrote:
> M. Warner Losh wrote:
> 
>> In message: <42C359F8.4060000@avtrex.com>
>>            David Daney <ddaney@avtrex.com> writes:
>> : M. Warner Losh wrote:
>> : > In message: <42C34C4D.9020902@avtrex.com>
>> : >             David Daney <ddaney@avtrex.com> writes:
>> : > : Does anyone have any idea what would cause 1000mS delay?
>> : > : > That's remarkably close to 1s.  This often indicates that the 
>> transmit
>> : > of your next packet is causing the receive buffer to empty.  This is
>> : > usually due to blocked interrupts, or a failure to enable interrupts.
>> : > : : But I observe ever increasing counts for the device in 
>> /proc/interrupts. :   So the interrupts are working somewhat.
>>
>> Are you sure that you've routed the interrupts correctly?  Maybe those
>> interrupts are 'really' for a different device....
>>  
>>
> Add some debugging to the interrupt routine of the e100 and see what 
> happens.

The interrupt routine is getting called each time a packet is received.

It looks like packets are not being transmitted until the interrupt for 
the the received packet is received.

If I ping the board at different intervals the round trip time is always 
almost exactly equal to the ping interval.  So if I ping every 50mS the 
round trip time is 50mS, ping every 200mS gives a RTT of 200mS, etc.

Any more ideas?

I am thinking that perhaps the CPU write-back-queue is interfearing with 
writes to the NIC's registers.  Perhaps I will try to disable it.

David Daney.