Extremely bad performance with RTL8111/8168B

Extremely bad performance with RTL8111/8168B

[Pavel Machek]

Hi!

I've small Intel desktop with RTL8111/8168B ethernet... but the
performance is extremely bad.

In 3.10, I only get about 200KB/sec:

pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme                                               100%   16MB
236.0KB/s   01:09    

In older version, I get 1MB/s, but network fails every 10 minutes or
so. Card is:

03:00.0 Ethernet controller: Realtek Semiconductor Co.,
Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)

I did install firmware-realtek package as suggested by debian
installer.

Dmesg says:

r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:03:00.0 eth0: RTL8168d/8111d at 0xf8010000,
00:27:0e:2a:43:71, XID 081000c0 IRQ 17
r8169 0000:03:00.0 eth0: jumbo features [frames: 9200 bytes, tx
checksumming: ko]

Any ideas what is wrong?

Thanks,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Extremely bad performance with RTL8111/8168B

[Eric Dumazet]

On Mon, 2013-07-08 at 11:54 +0200, Pavel Machek wrote:
> Hi!
> 
> I've small Intel desktop with RTL8111/8168B ethernet... but the
> performance is extremely bad.
> 
> In 3.10, I only get about 200KB/sec:
> 
> pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
> pavel@duo's password: 
> delme                                               100%   16MB
> 236.0KB/s   01:09    
> 
> In older version, I get 1MB/s, but network fails every 10 minutes or
> so. Card is:
> 
> 03:00.0 Ethernet controller: Realtek Semiconductor Co.,
> Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
> 
> I did install firmware-realtek package as suggested by debian
> installer.
> 
> Dmesg says:
> 
> r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> r8169 0000:03:00.0 eth0: RTL8168d/8111d at 0xf8010000,
> 00:27:0e:2a:43:71, XID 081000c0 IRQ 17
> r8169 0000:03:00.0 eth0: jumbo features [frames: 9200 bytes, tx
> checksumming: ko]
> 
> Any ideas what is wrong?

Hi Pavel

I guess receiving is OK ?

It looks like TSO packets are not transmitted, so TCP only 'works'
because retransmits use non TSO packets.

Could you report "ethtool -k eth0" ?

try : ethtool -K eth0 tso off

Re: Extremely bad performance with RTL8111/8168B

[Francois Romieu]

Eric Dumazet <eric.dumazet@gmail.com> :
[8168d failure]
> It looks like TSO packets are not transmitted, so TCP only 'works'
> because retransmits use non TSO packets.
> 
> Could you report "ethtool -k eth0" ?
> 
> try : ethtool -K eth0 tso off

If so it could be worth adding a test for RTL_GIGA_MAC_VER_25 in
rtl_test_hw_pad_bug as has been done for RTL_GIGA_MAC_VER_34 (see
b423e9ae49d78ea3f53b131c8d5a6087aed16fd6).

1a9646497b163a8b9da5e70008d809dc91b32855 may be reverted if
'ethtool -K eth0 tso off' does not make a difference.

-- 
Ueimor

Re: Extremely bad performance with RTL8111/8168B

[Pavel Machek]

Hi!

> Eric Dumazet <eric.dumazet@gmail.com> :
> [8168d failure]
> > It looks like TSO packets are not transmitted, so TCP only 'works'
> > because retransmits use non TSO packets.
> > 
> > Could you report "ethtool -k eth0" ?
> > 
> > try : ethtool -K eth0 tso off
> 
> If so it could be worth adding a test for RTL_GIGA_MAC_VER_25 in
> rtl_test_hw_pad_bug as has been done for RTL_GIGA_MAC_VER_34 (see
> b423e9ae49d78ea3f53b131c8d5a6087aed16fd6).
> 
> 1a9646497b163a8b9da5e70008d809dc91b32855 may be reverted if
> 'ethtool -K eth0 tso off' does not make a difference.

No, that did not seem to make any difference.

pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme                                         100%   34MB 653.6KB/s   00:53    
pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme                                         100%   34MB 509.4KB/s   01:08    
pavel@amd:~$ 

BTW I'm on 100mbit switch, so advanced stuff needed for gigabit speed
is not too important for me. I may want to connect it with cross cable
in future, but for now...

So I tried:

root@duo:~# ethtool  -K eth0 rx off gro off
root@duo:~# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
root@duo:~# 
root@duo:~# mii-tool eth0
eth0: negotiated 100baseTx-FD flow-control, link ok

But speed is still the same :-(:

pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme                                         100%   34MB 481.1KB/s   01:12    

Aha. I see abnormal ping times from duo to amd. (Cca 80msec range.)
amd has e1000e card, and apparently its interrupt does not
work. Fortunately it is shared with ahci so I can work around that
one... and when I do, transfer gets even slower. Hmm?

I tried tcpdump on duo. Sample:

20:06:44.063284 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 52128:53576, ack 1, win 151, options [nop,nop,TS va
l 162757452 ecr 218614], length 1448
20:06:44.063324 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 47784, win 516, options [nop,nop,TS val 218614 ecr 
162757452,nop,nop,sack 1 {49232:53576}], length 0
20:06:44.064267 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 47784:49232, ack 1, win 151, options [nop,nop,TS va
l 162757452 ecr 218614], length 1448
20:06:44.064308 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 53576, win 501, options [nop,nop,TS val 218614 ecr 
162757452], length 0
20:06:44.064325 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 53576:55024, ack 1, win 151, options [nop,nop,TS va
l 162757452 ecr 218614], length 1448
20:06:44.064357 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 55024, win 521, options [nop,nop,TS val 218614 ecr 
162757452], length 0
20:06:44.065057 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [P.], seq 1:49, ack 55024, win 521, options [nop,nop,TS val 
218614 ecr 162757452], length 48
20:06:44.065291 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 55024:56472, ack 1, win 151, options [nop,nop,TS va
l 162757453 ecr 218614], length 1448
20:06:44.065326 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 56472:57920, ack 1, win 151, options [nop,nop,TS va
l 162757453 ecr 218614], length 1448
20:06:44.065356 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 57920, win 521, options [nop,nop,TS val 218614 ecr 
162757453], length 0
20:06:44.066356 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 59368:60816, ack 49, win 151, options [nop,nop,TS v
al 162757453 ecr 218614], length 1448
20:06:44.066409 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 57920, win 521, options [nop,nop,TS val 218615 ecr 
162757453,nop,nop,sack 1 {59368:60816}], length 0
20:06:44.066426 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 60816:62264, ack 49, win 151, options [nop,nop,TS v
al 162757453 ecr 218614], length 1448
20:06:44.066462 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 57920, win 521, options [nop,nop,TS val 218615 ecr 
162757453,nop,nop,sack 1 {59368:62264}], length 0
20:06:44.067297 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 62264:63712, ack 49, win 151, options [nop,nop,TS v
al 162757453 ecr 218615], length 1448
20:06:44.067341 IP 10.0.0.9.22 > 10.0.0.6.45418: Flags [.], ack 57920, win 521, options [nop,nop,TS val 218615 ecr 
162757453,nop,nop,sack 1 {59368:63712}], length 0
20:06:44.067359 IP 10.0.0.6.45418 > 10.0.0.9.22: Flags [.], seq 63712:65160, ack 49, win 151, options [nop,nop,TS v
al 162757453 ecr 218615], length 1448

Any other ideas?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Extremely bad performance with RTL8111/8168B

[Eric Dumazet]

On Tue, 2013-07-09 at 02:09 +0200, Pavel Machek wrote:

> BTW I'm on 100mbit switch, so advanced stuff needed for gigabit speed
> is not too important for me. I may want to connect it with cross cable
> in future, but for now...



Check "ethtool eth0" and make sure you are in full-duplex mode.

Re: Extremely bad performance with RTL8111/8168B

[Pavel Machek]

On Mon 2013-07-08 17:12:32, Eric Dumazet wrote:
> On Tue, 2013-07-09 at 02:09 +0200, Pavel Machek wrote:
> 
> > BTW I'm on 100mbit switch, so advanced stuff needed for gigabit speed
> > is not too important for me. I may want to connect it with cross cable
> > in future, but for now...
> 
> Check "ethtool eth0" and make sure you are in full-duplex mode.

It seems that is not a problem... and I guess I have new speed record:

pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme                                          66%   22MB  11.7KB/s - stalled -
/tmp/delme: Broken pipe
pavel@amd:~$ 

(No, I did not terminate the transfer, it just died :-().

root@duo:~# ethtool eth0
Settings for eth0:
	Supported ports: [ TP MII ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Link partner advertised link modes:  10baseT/Half 10baseT/Full 
	                                     100baseT/Half 100baseT/Full 
	Link partner advertised pause frame use: Symmetric
	Link partner advertised auto-negotiation: Yes
	Speed: 100Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000033 (51)
	Link detected: yes
root@duo:~# 

Hmm, that is slightly interesting. What is the pause stuff? e1000e on
the same switch reports:

Settings for eth0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Speed: 100Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: on
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000007 (7)
	Link detected: yes
root@amd:~# 

I tried 

root@duo:~# ethtool -A eth0 rx off tx off
Cannot get device pause settings: Operation not supported
root@duo:~# 

Also... Why it reports Port: MII instead of "twisted pair"?

Any other ideas?

Thanks,
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Extremely bad performance with RTL8111/8168B

[Ben Hutchings]

On Tue, 2013-07-09 at 11:32 +0200, Pavel Machek wrote:
> On Mon 2013-07-08 17:12:32, Eric Dumazet wrote:
> > On Tue, 2013-07-09 at 02:09 +0200, Pavel Machek wrote:
> > 
> > > BTW I'm on 100mbit switch, so advanced stuff needed for gigabit speed
> > > is not too important for me. I may want to connect it with cross cable
> > > in future, but for now...
> > 
> > Check "ethtool eth0" and make sure you are in full-duplex mode.
> 
> It seems that is not a problem... and I guess I have new speed record:
> 
> pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
> pavel@duo's password: 
> delme                                          66%   22MB  11.7KB/s - stalled -
> /tmp/delme: Broken pipe
> pavel@amd:~$ 
> 
> (No, I did not terminate the transfer, it just died :-().
> 
> root@duo:~# ethtool eth0
> Settings for eth0:
> 	Supported ports: [ TP MII ]
> 	Supported link modes:   10baseT/Half 10baseT/Full 
> 	                        100baseT/Half 100baseT/Full 
> 	                        1000baseT/Half 1000baseT/Full 
> 	Supports auto-negotiation: Yes
> 	Advertised link modes:  10baseT/Half 10baseT/Full 
> 	                        100baseT/Half 100baseT/Full 
> 	                        1000baseT/Half 1000baseT/Full 
> 	Advertised pause frame use: Symmetric Receive-only
> 	Advertised auto-negotiation: Yes
> 	Link partner advertised link modes:  10baseT/Half 10baseT/Full 
> 	                                     100baseT/Half 100baseT/Full 
> 	Link partner advertised pause frame use: Symmetric
> 	Link partner advertised auto-negotiation: Yes
[...]
> Hmm, that is slightly interesting. What is the pause stuff? e1000e on
> the same switch reports:
>
> Settings for eth0:
> 	Supported ports: [ TP ]
> 	Supported link modes:   10baseT/Half 10baseT/Full 
> 	                        100baseT/Half 100baseT/Full 
> 	                        1000baseT/Full 
> 	Supports auto-negotiation: Yes
> 	Advertised link modes:  10baseT/Half 10baseT/Full 
> 	                        100baseT/Half 100baseT/Full 
> 	                        1000baseT/Full 
> 	Advertised pause frame use: No

e1000e doesn't report the advertised pause frame flags (or any link
partner advertising flags).

> 	Advertised auto-negotiation: Yes
> 	Speed: 100Mb/s
> 	Duplex: Full
> 	Port: Twisted Pair
> 	PHYAD: 1
> 	Transceiver: internal
> 	Auto-negotiation: on
> 	MDI-X: on
> 	Supports Wake-on: pumbg
> 	Wake-on: g
> 	Current message level: 0x00000007 (7)
> 	Link detected: yes
> root@amd:~# 
> 
> I tried 
> 
> root@duo:~# ethtool -A eth0 rx off tx off
> Cannot get device pause settings: Operation not supported
> root@duo:~# 
> 
> Also... Why it reports Port: MII instead of "twisted pair"?

Minor bug in r8169, not really relevant.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Extremely bad performance with RTL8111/8168B

[Pavel Machek]

Hi!

> pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
> pavel@duo's password: 
> delme                                         100%   34MB 481.1KB/s   01:12    
> 
> Aha. I see abnormal ping times from duo to amd. (Cca 80msec range.)
> amd has e1000e card, and apparently its interrupt does not
> work. Fortunately it is shared with ahci so I can work around that
> one... and when I do, transfer gets even slower. Hmm?

I verified that copy is slow even from Elf to duo. Elf is old desktop,
and its ethernet card is working correctly, AFAICT.

Copy is even slower from Elf->duo, cca 66KB/sec. It is also slower
when I work around interrupt problems on amd, amd->duo is then cca
200KB/sec.

IOW yes amd's e1000e has a problem, but duo's realtek has even bigger
problem.

Thanks,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

RE: Extremely bad performance with RTL8111/8168B

[hayeswang]

Francois Romieu [mailto:romieu@fr.zoreil.com] 
> Sent: Tuesday, July 09, 2013 6:02 AM
> To: Eric Dumazet
> Cc: Pavel Machek; Hayeswang; netdev@vger.kernel.org
> Subject: Re: Extremely bad performance with RTL8111/8168B
> 
> Eric Dumazet <eric.dumazet@gmail.com> :
> [8168d failure]
> > It looks like TSO packets are not transmitted, so TCP only 'works'
> > because retransmits use non TSO packets.
> > 
> > Could you report "ethtool -k eth0" ?
> > 
> > try : ethtool -K eth0 tso off
> 
> If so it could be worth adding a test for RTL_GIGA_MAC_VER_25 in
> rtl_test_hw_pad_bug as has been done for RTL_GIGA_MAC_VER_34 (see
> b423e9ae49d78ea3f53b131c8d5a6087aed16fd6).

I don't think it is necessary for RTL_GIGA_MAC_VER_25. I don't find any issue
about checksum or TSO for 8111d yet.
 
Best Regards,
Hayes

Re: Extremely bad performance with RTL8111/8168B

[Pavel Machek]

Hi!

> > I've small Intel desktop with RTL8111/8168B ethernet... but the
> > performance is extremely bad.
> > 
> > In 3.10, I only get about 200KB/sec:
> > 
> > pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
> > pavel@duo's password: 
> > delme                                               100%   16MB
> > 236.0KB/s   01:09    
> > 
> > In older version, I get 1MB/s, but network fails every 10 minutes or
> > so. Card is:
> > 
> > 03:00.0 Ethernet controller: Realtek Semiconductor Co.,
> > Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
> > 
> > I did install firmware-realtek package as suggested by debian
> > installer.
> > 
> > Dmesg says:
> > 
> > r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> > r8169 0000:03:00.0 eth0: RTL8168d/8111d at 0xf8010000,
> > 00:27:0e:2a:43:71, XID 081000c0 IRQ 17
> > r8169 0000:03:00.0 eth0: jumbo features [frames: 9200 bytes, tx
> > checksumming: ko]
> > 
> > Any ideas what is wrong?
> 
> Hi Pavel
> 
> I guess receiving is OK ?

Actually, receiving is slow. Transmit is ok. "duo" is "bad" machine
with RTL card.

pavel@amd:~/misc/hw/duo$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme 17%   35MB 759.1KB/s   03:39 ETA
(I aborted the copy).
pavel@amd:~/misc/hw/duo$ scp pavel@duo:/tmp/delme /tmp
pavel@duo's password: 
delme 100%   34MB  11.3MB/s   00:03    
pavel@amd:~/misc/hw/duo$ ping duo
PING duo (10.0.0.9) 56(84) bytes of data.
64 bytes from duo (10.0.0.9): icmp_req=1 ttl=64 time=1.17 ms
64 bytes from duo (10.0.0.9): icmp_req=2 ttl=64 time=0.737 ms
64 bytes from duo (10.0.0.9): icmp_req=3 ttl=64 time=1.17 ms
64 bytes from duo (10.0.0.9): icmp_req=4 ttl=64 time=0.608 ms

(I even out-of-tree driver for 2.6.32, and it seemed to have similar
problems.)


> It looks like TSO packets are not transmitted, so TCP only 'works'
> because retransmits use non TSO packets.
> 
> Could you report "ethtool -k eth0" ?

Yes:

root@duo:~# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

> try : ethtool -K eth0 tso off

No, that does not help:

pavel@amd:~$ scp /tmp/delme pavel@duo:/tmp
pavel@duo's password: 
delme 100%   34MB 706.9KB/s 00:49    

[Now... I plugged old J2585B 10/100VG PCI card into the duo...
full-height card into half-height slot, oops... only to discover that
it can't do 100mbit ethernet. I guess "100VG" means "not 100 mbit" :-(.]

Any other ideas? :-),

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Extremely bad performance with RTL8111/8168B

[Francois Romieu]

Pavel Machek <pavel@ucw.cz> :
[...]
> Actually, receiving is slow. Transmit is ok. "duo" is "bad" machine
> with RTL card.

No hint from the hardware statistics (ethtool -S eth0) ?

-- 
Ueimor

Re: Extremely bad performance with RTL8111/8168B

[Pavel Machek]

Hi!

> > Actually, receiving is slow. Transmit is ok. "duo" is "bad" machine
> > with RTL card.
> 
> No hint from the hardware statistics (ethtool -S eth0) ?

Nothing suspicious _now_.

For some reason network started going expected speed. I replugged
cable few times, replugged power, tried different ethernet card, went
back to RTL, and worked around e1000e problem on the other
machine. Not sure what helped, but now transfers are in 10MB/sec
range. ... strange.

It seems ok now.

Thanks,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html