r8169/r8168 slow internet upload

r8169/r8168 slow internet upload

[Marc Duponcheel]

 Hi all

 I have an issue with linux box acting as router (in fact forwarder)
with 2 Realtek RTL8111/8168/8411 cards.

 In short: large file uploads are way too slow and, during such
uploads, I see TcpRetransSegs and TcpOutRsts on sending host.

 System is bleeding edge gentoo.

 I replaced linux r8169 by vendor r8168 driver. That improved some
uploads (see below).


 - Details
  --------

Setup is as follows (VDSL2) with down/up 50Mbps/6Mbps


 intranet - server - router - modem - internet


modem =
 b-box3 technicolor box in bridge mode

router =
 The linux box in question doing pppoe on eth towards internet (where
 it receives an address for p-t-p to next hop) and having public /30
 network on eth towards intranet.

server =
 Also a bleeding edge gentoo gentoo box having one public address from
 the /30 network (router has the other one) and the doing NAT to
 devices on intranet.

  ** note **

  Before having the [router - modem] combo, I used a CISCO886VA-K9 box
 and I had expected 50/6 throughput. But provider, who migrates to
 VDSL2 vectoring, does not allow cisco on its network so I use b-box3
 as bridge (the box they allow) and put a linux router behind.


 - Symptoms
  ---------

 I test ftp between my server and clients few or several hops away.

* r8169:

 Incoming speed is OK (50Mbps) but outgoing is only 200Kbps. 30 times
too slow.

 Now I set eth speeds to 100Mbps (on linux-router, I never touch
linux-server who works fine anyway for years with cisco-router):

 Outgoing speed to few hops away is 2Mbps and to many hops away still 200Kbps.

* replace r8169 with r8168:

 Outgoing to few hops away is 3Mbps and to many hops away is 1.5Mbps.

 As said I always notice TcpRetransSegs and TcpOutRsts on server. I
 use
# watch -n 5 'nstat | grep ^Tcp'
 to inspect.


 I see few (really few) router drops on eth Rx towards intranet. In
fact I am not sure if those increase during upload.


 If I increase txqueuelen from 1000 to 10000 on router and flush tcp
metrics on router and server, outgoing speed starts 6Mbps but soon
drops back to steady 3Mbps and 1.5Mbps. I reverted back to 1000.

 FYI: ookla speedtests on all hosts show 50/6 (only with with r8168 on
   router), and, perhaps, for most work, everything is fine, but large
   file uploads, in particular from our ftp server to internet many
   hops away, is still 4 times too slow.

 If I do
watch 'ethtool --statistics iface | grep -v ' 0''
 on both router ethernets and server ethernet towards internet, I do not see
any error count increase.

 Note that realtek driver does not support setting eth flow control so
I could not try to turn it off (not that I expect it to help).

 Also PPP MTU/MRU are 1492 and TCPMSS clamp to PMTU is on server and router
(but that did not matter).

 - Conclusions
  ------------

 Setup is both driver and eth-speed sensitive. I also suspect TCP to
kick in to compensate for whatever issues.


 - Questions
  ----------

 Am I missing something obvious? Or are the RTL8111/8168/8411
cards/drivers not good for this setup? Any other hints? I am happy to
do whatever tests and provide whatever outputs that may help drive
discussion.


 *** thanks ****

--
 Marc Duponcheel
 Velodroomstraat 74 - 2600 Berchem - Belgium
 +32 (0)478 68.10.91 - marc@offline.be

Re: r8169/r8168 slow internet upload

[Francois Romieu]

Marc Duponcheel <marc@offline.be> :
[...]
>  I have an issue with linux box acting as router (in fact forwarder)
> with 2 Realtek RTL8111/8168/8411 cards.
> 
>  In short: large file uploads are way too slow and, during such
> uploads, I see TcpRetransSegs and TcpOutRsts on sending host.
> 
>  System is bleeding edge gentoo.

Please specify kernel version.

[...]
> * r8169:
> 
>  Incoming speed is OK (50Mbps) but outgoing is only 200Kbps. 30 times
> too slow.

Please send plain ethtool output and network capture file for the outgoing
traffic.

>  Now I set eth speeds to 100Mbps (on linux-router, I never touch
> linux-server who works fine anyway for years with cisco-router):
> 
>  Outgoing speed to few hops away is 2Mbps and to many hops away still 200Kbps.

A capture file for each of those would be welcome.

[...]
> * replace r8169 with r8168:
> 
>  Outgoing to few hops away is 3Mbps and to many hops away is 1.5Mbps.

Same capture as above will be welcome.

[...]
> Am I missing something obvious? Or are the RTL8111/8168/8411
> cards/drivers not good for this setup? Any other hints? I am happy to
> do whatever tests and provide whatever outputs that may help drive
> discussion.

Nothing obvious on the driver side. There could be an r816x hardware bug
thus some (small) packets loss.

Please send a complete dmesg including the 'XID' line from the r8169
driver.

Thanks.

-- 
Ueimor

Re: r8169/r8168 slow internet upload

[Francois Romieu]

Marc Duponcheel <marc@offline.be> :
[...]
> not something to do on a router)). That should be fine since r8168
> gave better results.

Is r8169 unusable even at 100 ?

[...]
> * case  100 (when router enp{2,3}s0 are  100Mbps)
> interestingly enough, speed took off at expected 6Mbps but dropped to
> stable 1.5Mbps

I haven't found a clear packet loss pattern yet.

Can you take the *same* (100) capture at enp3s0 ?

If the network speed test is fine as long as the client is near, I'd
seriously question your (c*gent ?) transit.

mtr output from far client to server would be welcome.

-- 
Ueimor

Re: r8169/r8168 slow internet upload

[Marc Duponcheel]

Here is TCP nstat after some time (days) of normal. mixed user traffic

 are TcpRetransSegs and TcpOutRsts reasonable ?

Thanks

opteron ~ # nstat | grep ^Tcp
TcpActiveOpens                  218771             0.0
TcpPassiveOpens                 245187             0.0
TcpAttemptFails                 961                0.0
TcpEstabResets                  1803               0.0
TcpInSegs                       5539041            0.0
TcpOutSegs                      5883556            0.0
TcpRetransSegs                  38828              0.0
TcpOutRsts                      369036             0.0
TcpExtSyncookiesFailed          112                0.0
TcpExtEmbryonicRsts             802                0.0
TcpExtTW                        218409             0.0
TcpExtPAWSEstab                 5                  0.0
TcpExtDelayedACKs               32173              0.0
TcpExtDelayedACKLocked          50                 0.0
TcpExtDelayedACKLost            1052               0.0
TcpExtListenDrops               7                  0.0
TcpExtTCPPrequeued              216710             0.0
TcpExtTCPDirectCopyFromBacklog  2218150            0.0
TcpExtTCPDirectCopyFromPrequeue 31007060           0.0
TcpExtTCPHPHits                 1189383            0.0
TcpExtTCPHPHitsToUser           23700              0.0
TcpExtTCPPureAcks               1265234            0.0
TcpExtTCPHPAcks                 1373386            0.0
TcpExtTCPSackRecovery           10161              0.0
TcpExtTCPSACKReneging           1                  0.0
TcpExtTCPFullUndo               2                  0.0
TcpExtTCPDSACKUndo              4                  0.0
TcpExtTCPLossUndo               103                0.0
TcpExtTCPLostRetransmit         23                 0.0
TcpExtTCPSackFailures           7                  0.0
TcpExtTCPLossFailures           46                 0.0
TcpExtTCPFastRetrans            29705              0.0
TcpExtTCPForwardRetrans         144                0.0
TcpExtTCPSlowStartRetrans       186                0.0
TcpExtTCPTimeouts               914                0.0
TcpExtTCPLossProbes             2094               0.0
TcpExtTCPLossProbeRecovery      250                0.0
TcpExtTCPSackRecoveryFail       63                 0.0
TcpExtTCPSchedulerFailed        1                  0.0
TcpExtTCPDSACKOldSent           1041               0.0
TcpExtTCPDSACKOfoSent           55                 0.0
TcpExtTCPDSACKRecv              967                0.0
TcpExtTCPAbortOnData            1520               0.0
TcpExtTCPAbortOnClose           185                0.0
TcpExtTCPAbortOnTimeout         256                0.0
TcpExtTCPDSACKIgnoredNoUndo     85                 0.0
TcpExtTCPSpuriousRTOs           11                 0.0
TcpExtTCPSackShifted            28438              0.0
TcpExtTCPSackMerged             26733              0.0
TcpExtTCPSackShiftFallback      32560              0.0
TcpExtTCPDeferAcceptDrop        335                0.0
TcpExtIPReversePathFilter       24                 0.0
TcpExtTCPRetransFail            114                0.0
TcpExtTCPRcvCoalesce            338862             0.0
TcpExtTCPOFOQueue               5712               0.0
TcpExtTCPOFOMerge               55                 0.0
TcpExtTCPChallengeACK           20                 0.0
TcpExtTCPSpuriousRtxHostQueues  463                0.0
TcpExtTCPAutoCorking            48213              0.0

--
 Marc Duponcheel
 Velodroomstraat 74 - 2600 Berchem - Belgium
 +32 (0)478 68.10.91 - marc@offline.be

Re: r8169/r8168 slow internet upload

[Marc Duponcheel]

 FYI

it seems upgrade to 3.14.5 -> 3.15.1 fixed issue ...

not sure which change implements fix but, in any case:

 THANKS

On 2014 May 12, Marc Duponcheel wrote:
>  Hi all
> 
>  I have an issue with linux box acting as router (in fact forwarder)
> with 2 Realtek RTL8111/8168/8411 cards.
> 
>  In short: large file uploads are way too slow and, during such
> uploads, I see TcpRetransSegs and TcpOutRsts on sending host.
> 
>  System is bleeding edge gentoo.
> 
>  I replaced linux r8169 by vendor r8168 driver. That improved some
> uploads (see below).
> 
> 
>  - Details
>   --------
> 
> Setup is as follows (VDSL2) with down/up 50Mbps/6Mbps
> 
> 
>  intranet - server - router - modem - internet
> 
> 
> modem =
>  b-box3 technicolor box in bridge mode
> 
> router =
>  The linux box in question doing pppoe on eth towards internet (where
>  it receives an address for p-t-p to next hop) and having public /30
>  network on eth towards intranet.
> 
> server =
>  Also a bleeding edge gentoo gentoo box having one public address from
>  the /30 network (router has the other one) and the doing NAT to
>  devices on intranet.
> 
>   ** note **
> 
>   Before having the [router - modem] combo, I used a CISCO886VA-K9 box
>  and I had expected 50/6 throughput. But provider, who migrates to
>  VDSL2 vectoring, does not allow cisco on its network so I use b-box3
>  as bridge (the box they allow) and put a linux router behind.
> 
> 
>  - Symptoms
>   ---------
> 
>  I test ftp between my server and clients few or several hops away.
> 
> * r8169:
> 
>  Incoming speed is OK (50Mbps) but outgoing is only 200Kbps. 30 times
> too slow.
> 
>  Now I set eth speeds to 100Mbps (on linux-router, I never touch
> linux-server who works fine anyway for years with cisco-router):
> 
>  Outgoing speed to few hops away is 2Mbps and to many hops away still 200Kbps.
> 
> * replace r8169 with r8168:
> 
>  Outgoing to few hops away is 3Mbps and to many hops away is 1.5Mbps.
> 
>  As said I always notice TcpRetransSegs and TcpOutRsts on server. I
>  use
> # watch -n 5 'nstat | grep ^Tcp'
>  to inspect.
> 
> 
>  I see few (really few) router drops on eth Rx towards intranet. In
> fact I am not sure if those increase during upload.
> 
> 
>  If I increase txqueuelen from 1000 to 10000 on router and flush tcp
> metrics on router and server, outgoing speed starts 6Mbps but soon
> drops back to steady 3Mbps and 1.5Mbps. I reverted back to 1000.
> 
>  FYI: ookla speedtests on all hosts show 50/6 (only with with r8168 on
>    router), and, perhaps, for most work, everything is fine, but large
>    file uploads, in particular from our ftp server to internet many
>    hops away, is still 4 times too slow.
> 
>  If I do
> watch 'ethtool --statistics iface | grep -v ' 0''
>  on both router ethernets and server ethernet towards internet, I do not see
> any error count increase.
> 
>  Note that realtek driver does not support setting eth flow control so
> I could not try to turn it off (not that I expect it to help).
> 
>  Also PPP MTU/MRU are 1492 and TCPMSS clamp to PMTU is on server and router
> (but that did not matter).
> 
>  - Conclusions
>   ------------
> 
>  Setup is both driver and eth-speed sensitive. I also suspect TCP to
> kick in to compensate for whatever issues.
> 
> 
>  - Questions
>   ----------
> 
>  Am I missing something obvious? Or are the RTL8111/8168/8411
> cards/drivers not good for this setup? Any other hints? I am happy to
> do whatever tests and provide whatever outputs that may help drive
> discussion.
> 
> 
>  *** thanks ****

--
 Marc Duponcheel
 Velodroomstraat 74 - 2600 Berchem - Belgium
 +32 (0)478 68.10.91 - marc@offline.be