From: Ben Greear <greearb@candelatech.com>
To: Josh Hunt <joshhunt00@gmail.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>, netdev <netdev@vger.kernel.org>
Subject: Re: TCP many-connection regression between 4.7 and 4.13 kernels.
Date: Tue, 23 Jan 2018 14:06:24 -0800 [thread overview]
Message-ID: <847d24f9-9bcc-6de3-fd58-7414c22eebeb@candelatech.com> (raw)
In-Reply-To: <CAKA=qzZ1xW3pD1q4MHpNRjXwbJPrh3YogO8X6rLAzZ7h6KS2Kg@mail.gmail.com>
On 01/22/2018 10:46 AM, Josh Hunt wrote:
> On Mon, Jan 22, 2018 at 10:30 AM, Ben Greear <greearb@candelatech.com> wrote:
>> On 01/22/2018 10:16 AM, Eric Dumazet wrote:
>>>
>>> On Mon, 2018-01-22 at 09:28 -0800, Ben Greear wrote:
>>>>
>>>> My test case is to have 6 processes each create 5000 TCP IPv4 connections
>>>> to each other
>>>> on a system with 16GB RAM and send slow-speed data. This works fine on a
>>>> 4.7 kernel, but
>>>> will not work at all on a 4.13. The 4.13 first complains about running
>>>> out of tcp memory,
>>>> but even after forcing those values higher, the max connections we can
>>>> get is around 15k.
>>>>
>>>> Both kernels have my out-of-tree patches applied, so it is possible it is
>>>> my fault
>>>> at this point.
>>>>
>>>> Any suggestions as to what this might be caused by, or if it is fixed in
>>>> more recent kernels?
>>>>
>>>> I will start bisecting in the meantime...
>>>>
>>>
>>> Hi Ben
>>>
>>> Unfortunately I have no idea.
>>>
>>> Are you using loopback flows, or have I misunderstood you ?
>>>
>>> How loopback connections can be slow-speed ?
>>>
>>
>> I am sending to self, but over external network interfaces, by using
>> routing tables and rules and such.
>>
>> On 4.13.16+, I see the Intel driver bouncing when I try to start 20k
>> connections. In this case, I have a pair of 10G ports doing 15k, and then
>> I try to start 5k on two of the 1G ports....
>>
>> Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Down
>> Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Down
>> Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Down
>> Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 22 10:15:43 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Down
>> Jan 22 10:15:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
>> Jan 22 10:15:51 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
>> (e1000e): transmit queue 0 timed out, trans_s...es: 1
>> Jan 22 10:15:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0
>> eth3: Reset adapter unexpectedly
>>
>
> Ben
>
> We had an interface doing this and grabbing these commits resolved it for us:
>
> 4aea7a5c5e94 e1000e: Avoid receiver overrun interrupt bursts
> 19110cfbb34d e1000e: Separate signaling for link check/link up
> d3509f8bc7b0 e1000e: Fix return value test
> 65a29da1f5fd e1000e: Fix wrong comment related to link detection
> c4c40e51f9c3 e1000e: Fix error path in link detection
>
> They are in the LTS kernels now, but don't believe they were when we
> first hit this problem.
Thanks a lot for the suggestions, I can confirm that these patches applied to my 4.13.16+
tree does indeed seem to fix the problem.
Thanks,
Ben
>
> Josh
>
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
next prev parent reply other threads:[~2018-01-23 22:06 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-22 17:28 TCP many-connection regression between 4.7 and 4.13 kernels Ben Greear
2018-01-22 18:16 ` Eric Dumazet
2018-01-22 18:27 ` Willy Tarreau
2018-01-22 18:30 ` Ben Greear
2018-01-22 18:44 ` Ben Greear
2018-01-22 18:46 ` Josh Hunt
2018-01-23 22:06 ` Ben Greear [this message]
2018-01-23 21:49 ` TCP many-connection regression (bisected to 4.5.0-rc2+) Ben Greear
2018-01-23 22:07 ` Eric Dumazet
2018-01-23 22:09 ` Ben Greear
2018-01-23 22:29 ` Eric Dumazet
2018-01-23 23:10 ` Ben Greear
2018-01-23 23:21 ` Eric Dumazet
2018-01-23 23:27 ` Ben Greear
2018-01-24 0:05 ` Ben Greear
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=847d24f9-9bcc-6de3-fd58-7414c22eebeb@candelatech.com \
--to=greearb@candelatech.com \
--cc=eric.dumazet@gmail.com \
--cc=joshhunt00@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).