r8169: hard freezes on TX

r8169: hard freezes on TX

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 3120 bytes --]

Hi all,

I often see freezes when I do much outgoing transfer. I have never seen this 
happening on incoming transfers. When this happens the system locks up hard, 
I don't see anything in the log. Since this is my laptop I have trouble 
debugging it: there is no serial console and debugging this via netconsole 
doesn't look like a good idea.

When I say "much outgoing transfer" this means "several megabytes". If I copy 
out 30 MB I almost everytime get this. I usually copy that much only at home 
when I feed my gentoo server. That host only has a 10 MBit connection. 
Nevertheless I've also seen that on different hosts using different files on 
different protocols (ftp, scp, smb).

This is the output of lspci for my NIC:

05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E PCI 
Express Fast Ethernet controller (rev 01)
        Subsystem: Toshiba America Info Systems Unknown device ff00
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: I/O ports at 4000 [size=256]
        Region 2: Memory at da000000 (64-bit, non-prefetchable) [size=4K]
        [virtual] Expansion ROM at d4000000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA 
PME(D0-,D1+,D2+,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Vital Product Data
        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/1 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
                Device: Latency L0s <1us, L1 unlimited
                Device: AtnBtn+ AtnInd+ PwrInd+
                Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
                Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
                Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
                Link: Latency L0s unlimited, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
                Link: Speed 2.5Gb/s, Width x1
        Capabilities: [84] Vendor Specific Information
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [12c] Virtual Channel
        Capabilities: [148] Device Serial Number xx-xx-xx-xx-xx-xx-xx-xx
        Capabilities: [154] Power Budgeting

Hm, is there a reason why we don't use MSI here?

Ah, one thing is missing: I've not tested it with current kernel, latest I 
tested was 2.6.21-rc7. But I've seen this on many previous version, although 
I thought it became better some versions ago. I wont bet on it, it might just 
have been luck.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: r8169: hard freezes on TX

[Francois Romieu]

Rolf Eike Beer <eike-kernel@sf-tec.de> :
[...]
> I often see freezes when I do much outgoing transfer. I have never seen this 
> happening on incoming transfers. When this happens the system locks up hard, 
> I don't see anything in the log. Since this is my laptop I have trouble 
> debugging it: there is no serial console and debugging this via netconsole 
> doesn't look like a good idea.

Keyboard leds are dead afterwards I guess, right ?

If you are experiencing bugs related to networking, I suggest to stay
away from netconsole. It is not funny to analyze several bugs at the same
time.

> When I say "much outgoing transfer" this means "several megabytes". If I copy 
> out 30 MB I almost everytime get this. I usually copy that much only at home 
> when I feed my gentoo server. That host only has a 10 MBit connection. 
> Nevertheless I've also seen that on different hosts using different files on 
> different protocols (ftp, scp, smb).

:o/

So it can be reproduced with a simple ftp put of several megabytes of
data completely cached in memory (no disk access) ?

[...]
> This is the output of lspci for my NIC:
> 
> 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E PCI 
> Express Fast Ethernet controller (rev 01)

I have not seen a lot of reports for this one. Either it is perfect or
it is barely used.

[...]
> Hm, is there a reason why we don't use MSI here?

A request for testers was posted (netdev + lk) on 16/03/2007 which contained
MSI code for the 8168. I did not enable it for the 8101 because it had only
received (positive) reports from 8168 users.

Afair, the RFT got no feedback.

> Ah, one thing is missing: I've not tested it with current kernel, latest I 
> tested was 2.6.21-rc7. But I've seen this on many previous version, although 
> I thought it became better some versions ago. I wont bet on it, it might just 
> have been luck.

You can/should try:
http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.22-rc2 (patch-kit)
or:
http://www.fr.zoreil.com/people/francois/misc/20070522-2.6.22-rc2-r8169.patch

If you are fluent with git and you do not mind rebasing, you can try
git://electric-eye.fr.zoreil.com/home/romieu/linux/linux-2.6-out r8169

(don't do the initial clone from here, thanks)

As an option, akpm includes the git branch for you in -mm.

-- 
Ueimor

Re: r8169: hard freezes on TX

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 2797 bytes --]

You wrote:
> Rolf Eike Beer <eike-kernel@sf-tec.de> :
> [...]
>
> > I often see freezes when I do much outgoing transfer. I have never seen
> > this happening on incoming transfers. When this happens the system locks
> > up hard, I don't see anything in the log. Since this is my laptop I have
> > trouble debugging it: there is no serial console and debugging this via
> > netconsole doesn't look like a good idea.
>
> Keyboard leds are dead afterwards I guess, right ?

I'm not absolutely sure but I can do a test later today.

> If you are experiencing bugs related to networking, I suggest to stay
> away from netconsole. It is not funny to analyze several bugs at the same
> time.

I did not expect anything coming through there, last time I tested even pings 
were unanswered. Because of this I didn't even try netconsole.

> > When I say "much outgoing transfer" this means "several megabytes". If I
> > copy out 30 MB I almost everytime get this. I usually copy that much only
> > at home when I feed my gentoo server. That host only has a 10 MBit
> > connection. Nevertheless I've also seen that on different hosts using
> > different files on different protocols (ftp, scp, smb).
> >
> :o/
>
> So it can be reproduced with a simple ftp put of several megabytes of
> data completely cached in memory (no disk access) ?

I can put it into RAM before next test to be absolutely sure.

> > This is the output of lspci for my NIC:
> >
> > 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E PCI
> > Express Fast Ethernet controller (rev 01)
>
> I have not seen a lot of reports for this one. Either it is perfect or
> it is barely used.
>
> [...]
>
> > Hm, is there a reason why we don't use MSI here?
>
> A request for testers was posted (netdev + lk) on 16/03/2007 which
> contained MSI code for the 8168. I did not enable it for the 8101 because
> it had only received (positive) reports from 8168 users.
>
> Afair, the RFT got no feedback.

I'll dig for it.

> > Ah, one thing is missing: I've not tested it with current kernel, latest
> > I tested was 2.6.21-rc7. But I've seen this on many previous version,
> > although I thought it became better some versions ago. I wont bet on it,
> > it might just have been luck.
>
> You can/should try:
> http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.22-rc2 (patch-kit)
> or:
> http://www.fr.zoreil.com/people/francois/misc/20070522-2.6.22-rc2-r8169.pat
>ch
>
> If you are fluent with git and you do not mind rebasing, you can try
> git://electric-eye.fr.zoreil.com/home/romieu/linux/linux-2.6-out r8169
>
> (don't do the initial clone from here, thanks)
>
> As an option, akpm includes the git branch for you in -mm.

I'll take a look, thanks.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: r8169: hard freezes on TX

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

Francois Romieu wrote:
> Rolf Eike Beer <eike-kernel@sf-tec.de> :
> [...]

Ok, just tested. I used a file of 200MB and copied it to another host on the 
LAN. If I used our 100 MBit switch nothing happened. When I put a 10 MBit hub 
in the middle it died at 77 MB.

> > I often see freezes when I do much outgoing transfer. I have never seen
> > this happening on incoming transfers. When this happens the system locks
> > up hard, I don't see anything in the log. Since this is my laptop I have
> > trouble debugging it: there is no serial console and debugging this via
> > netconsole doesn't look like a good idea.
>
> Keyboard leds are dead afterwards I guess, right ?

Yes.
> > When I say "much outgoing transfer" this means "several megabytes". If I
> > copy out 30 MB I almost everytime get this. I usually copy that much only
> > at home when I feed my gentoo server. That host only has a 10 MBit
> > connection. Nevertheless I've also seen that on different hosts using
> > different files on different protocols (ftp, scp, smb).
> >
> :o/
>
> So it can be reproduced with a simple ftp put of several megabytes of
> data completely cached in memory (no disk access) ?

I used scp, but: yes.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: r8169: hard freezes on TX

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Rolf Eike Beer wrote:
> Francois Romieu wrote:
> > Rolf Eike Beer <eike-kernel@sf-tec.de> :
> > [...]
>
> Ok, just tested. I used a file of 200MB and copied it to another host on
> the LAN. If I used our 100 MBit switch nothing happened. When I put a 10
> MBit hub in the middle it died at 77 MB.

I just had another freeze using your patches. After 512kB over smb it was 
dead.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: r8169: hard freezes on TX

[Francois Romieu]

Rolf Eike Beer <eike-kernel@sf-tec.de> :
[...]
> I just had another freeze using your patches. After 512kB over smb it was 
> dead.

In-kernel smb/cifs ?

-- 
Ueimor

Re: r8169: hard freezes on TX

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 303 bytes --]

You wrote:
> Rolf Eike Beer <eike-kernel@sf-tec.de> :
> [...]
>
> > I just had another freeze using your patches. After 512kB over smb it was
> > dead.
>
> In-kernel smb/cifs ?

Copying to a partition mounted via smb:// protocol in konqueror which uses 
kio_smb (userspace io slave).

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: r8169: hard freezes on TX

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

Rolf Eike Beer wrote:
> You wrote:
> > Rolf Eike Beer <eike-kernel@sf-tec.de> :
> > [...]
> >
> > > I just had another freeze using your patches. After 512kB over smb it
> > > was dead.
> >
> > In-kernel smb/cifs ?
>
> Copying to a partition mounted via smb:// protocol in konqueror which uses
> kio_smb (userspace io slave).

This start to get weird. I sent a single file (~4 MB) via FTP to a host 
connected via a 10 MBit Hub: freeze. Now I decided to beat it and tested a 
bit around. I copied some folders of my MP3 collection over there, all went 
fine. Then I created a tar file of ~400 MB and sent that which also went 
fine. Even the next try with this file didn't produce a crash. What was 
different on this tests was that I used the console instead of X (don't want 
to munch up too many file systems). Any ideas?

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: r8169: hard freezes on TX

[Francois Romieu]

Rolf Eike Beer <eike-kernel@sf-tec.de> :
[...]
> fine. Even the next try with this file didn't produce a crash. What was 
> different on this tests was that I used the console instead of X (don't want 
> to munch up too many file systems). Any ideas?

X and friends ?

Can you send the usual dmesg, lspci -vvx and lsmod ?

-- 
Ueimor