Re: Realtek 8169 Lock-ups

Re: Realtek 8169 Lock-ups

[Jeff Garzik]

Hum... where are we on this?

Francois made a few key fixes before he left, which I've included in the 
latest netdev patch...

http://www.kernel.org/pub/linux/kernel/people/jgarzik/patchkits/2.6/2.6.1-bk1-netdev2.patch.bz2

Have those already been tested?

Thanks,

	Jeff

Re: Realtek 8169 Lock-ups

[Francois Romieu]

Jeff Garzik <jgarzik@pobox.com> :
[...]
> Have those already been tested?

Yes. Tester (damouse) included in Cc: list.
The patches improves the situation so that it is possible to bring the
interface up and send/receive traffic but something clearly smashes the
stack (if someone wants to check, .jpg panic and r8169 module available
at http://www.fr.zoreil.com/people/francois/misc/debug).

After the latest days of "patch/patch -R/change line foo into bar/...",
I have rediffed the whole serie of changes against 2.6.1 so M. Damouse can
test an identified set of patches and isolate the misbehaving one(s). It is
available at http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.1.

If people can reproduce the tests that M. Damouse made, it should look like:
apply r8169-dma-api-tx.patch
apply r8169-dma-api-rx-buffers.patch
apply r8169-dma-api-rx-buffers-ahum.patch
apply r8169-rx-fill-typo.patch
apply r8169-start-xmit-fixes.patch
apply r8169-dma-api-tx-buffers.patch
apply r8169-rx_copybreak.patch
-> So far, so good, no regression from classical driver
apply r8169-mac-phy-version.patch
apply r8169-buggy-devinitdata.patch
-> Seems fine but panics after a few ko of traffic

<blink>Confirmation, anyone ?</blink>

The set contains two extra/untested patches [*] wrt 2.6.1-bk1-netdev2. 
No need to push these further until the current issues are fixed imho.

M. Pollock's problems look different:
- they do not depend on the driver used;
- they seem timing/smp related (mobile computer + p4 HT).

[*]
r8169-addr-high.patch
r8169-ethtool-introduction.patch

--
Ueimor

Re: Realtek 8169 Lock-ups

[Douglas Pollock]

On Jan 11, Francois Romieu <romieu@fr.zoreil.com> wrote:
> M. Pollock's problems look different:
> - they do not depend on the driver used;

I can confirm that I do not see kernel panics, and that the driver (for all intents and 
purposes) seems to work well for a period of time.  


> - they seem timing/smp related (mobile computer + p4 HT).

I ran without SMP support in the kernel for about 1.5 days on my work network, and did 
not see any of the lock-ups described.  However, this evidence is somewhat 
circumstantial, as we don't have a reproducible test case.

It would be nice if someone could point me to a network stress testing tool (something 
that opens multiple simultaneous connections and keeps them at high load).  I'm hoping 
that such a test might provide a reproducible test case.  This way we could confirm or 
rule-out the SMP relationship.



Thanks,
Doug.

Re: Realtek 8169 Lock-ups

[Francois Romieu]

Re,

  latest pile of r8169 related patches against 2.6.1-bk1 available at
http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.1-bk1

If you want to help:
apply r8169-dma-api-tx.patch
apply r8169-dma-api-rx-buffers.patch
apply r8169-dma-api-rx-buffers-ahum.patch
apply r8169-dma-api-rx-buffers-argl.patch <-- bug removed, sing and rejoice !
apply r8169-rx-fill-typo.patch
apply r8169-start-xmit-fixes.patch
apply r8169-dma-api-tx-buffers.patch
apply r8169-rx_copybreak.patch

-> does it work here ?

apply r8169-mac-phy-version.patch
apply r8169-buggy-devinitdata.patch

-> and here ?

--
Ueimor

Re: Realtek 8169 Lock-ups

[pollockd]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

On Sun, 2004-01-11 at 14:29, Jeff Garzik wrote:
> Douglas Pollock wrote:
> The kernel comes with "pktgen" which is a free packet generator you can 
> use for stressing kernels and LANs.  There is also ttcp, nttcp, various 
> filesystems tests over NFS (such as bonnie++), various block device 
> tests over nbd (network block device), ...

Thanks for that.  I should have remembered the packet generator.  Using
pktgen, I am able to reproduce the problem.


HARDWARE:
Intel P4 with hyper-threading enabled
Realtek 8169 (rev. 10)
Switched 100Mbps network
A second machine on the same network (to receive test packets)

SOFTWARE:
Kernel 2.6.0 or 2.6.1-rc1 (possibly others), with SMP for 2 processors,
Realtek 8169 driver, and packet generator


STEPS TO REPRODUCE:
1.) Set up the ipg script to send to the second machine on the 8169 NIC.
2.) pgset pkt_size 9014
3.) pg


OBSERVATIONS:
Soon after step #3, the 8169 NIC will stop responding (note: the netdev
watchdog reports a timeout in dmesg).  No packets get through.  However,
everything else still works.  Disabling SMP in the kernel fixes this
problem.



Doug.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Realtek 8169 Lock-ups

[Francois Romieu]

Re,

  latest pile of r8169 related patches against 2.6.1-bk1 available at:
http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.1-bk1-a

Credits go to damouse@ntlworld.com for a bug removal in r8169-init_one.patch.

No change of behavior expected for SMP so far.

--
Ueimor

Re: Realtek 8169 Lock-ups

[Francois Romieu]

pollockd@magma.ca <pollockd@magma.ca> :
[...]
> With SMP enabled, I get the following, and the packet generator aborts
> quickly:
> 
> APIC error on CPU0: 40(40)
> eth0: Too much work at interrupt!
> eth0: Too much work at interrupt!

Well, it is not clear that the "Too muck work at interrupt" recovery logic
really recovers much (be it for vanilla or modified r8169 driver).

Does it behave differently if you make the "while (boguscnt > 0);" always
true in rtl8169_interrupt() and ask for a finite number of packets (say
pgset "count 1000" or more) ?

--
Ueimor

Re: Realtek 8169 Lock-ups

[Douglas Pollock]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Francois,

Many apologies.  It was my own stupidity.  I've sorted things out, and
am now running 2.6.1-bk1 with your patchset.

With SMP enabled, I get a timeout from the watchdog and my network
connection becomes unresponsive.  In fact, during boot the DHCPC server
times out looking for an IP address.

I tried changing (boguscnt > 0) to (1 > 0) and this made no difference.

Both of these tests were run with 1000 9014-byte packets.

Even with SMP disabled, and even with the patches not applied, there seems to be problems (this is on 2.6.1-bk1).  Perhaps I just wasn't seeing them earlier.  Basically, sometimes the packet generator completes normally the first time.  Sometimes, it does not.  Either way, by the second run of the packet generator, the network device stops responding.

I will try the most recent kernel and patch set, and see what problems I get.



Doug.



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]