From: Willy Tarreau <w@1wt.eu>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Major network performance regression in 3.7
Date: Sun, 6 Jan 2013 20:53:59 +0100 [thread overview]
Message-ID: <20130106195359.GN16031@1wt.eu> (raw)
In-Reply-To: <1357501171.6919.650.camel@edumazet-glaptop>
On Sun, Jan 06, 2013 at 11:39:31AM -0800, Eric Dumazet wrote:
> On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote:
>
> > OK it works like a charm here now ! I can't break it anymore, so it
> > looks like you finally got it !
> >
> > I noticed that the data rate was higher when the loopback's MTU
> > is exactly a multiple of 4096 (making the 64k choice optimal)
> > while I would have assumed that in order to efficiently splice
> > TCP segments, we'd need to have some space for IP/TCP headers
> > and n*4k for the payload.
> >
> > I also got the transfer freezes again a few times when starting
> > tcpdump on the server, but this is not 100% reproducible I'm afraid.
> > So I'll bring this back when I manage to get some analysable pattern.
> >
> > The spliced transfer through all the chain haproxy works fine again
> > at 10gig with your fix. The issue is closed for me. Feel free to add
> > my Tested-By if you want.
> >
>
> Good to know !
>
> What is the max speed you get now ?
Line rate with 1500 MTU and LRO enabled :
# time eth1(ikb ipk okb opk) eth2(ikb ipk okb opk)
1357060023 19933.3 41527.7 9355538.2 62167.7 9757888.1 808701.1 19400.3 40417.7
1357060024 26124.1 54425.5 9290064.9 48804.4 9778294.0 810210.0 18068.8 37643.3
1357060025 27015.2 56281.1 9296115.3 46868.8 9797125.9 811271.1 8790.1 18312.2
1357060026 27556.0 57408.8 9291701.4 46805.5 9805371.6 811410.0 3494.8 7280.0
1357060027 27577.0 57452.2 9293606.8 46804.4 9806122.3 811314.4 2558.7 5330.0
1357060028 27476.1 57242.2 9296885.4 46830.0 9794537.3 810527.7 2516.1 5242.2
^^^^^^^ ^^^^^^^
kbps out kbps in
eth1=facing the client
eth2=facing the server
Top reports the following usage :
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 31.7%id, 0.0%wa, 0.0%hi, 68.3%si, 0.0%st
Cpu1 : 1.0%us, 37.3%sy, 0.0%ni, 61.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
(IRQ bound to cpu 0, haproxy to cpu 1)
This is a core2duo 2.66 GHz and the myris are 1st generation.
BTW I was very happy to see that the LRO->GRO conversion patches in 3.8-rc2
don't affect byte rate anymore (just a minor CPU usage increase but this is
not critical here), now I won't complain about it being slower anymore, you
won :-)
With the GRO patches backported, still at 1500 MTU but with GRO now :
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 28.7%id, 0.0%wa, 0.0%hi, 71.3%si, 0.0%st
Cpu1 : 0.0%us, 37.6%sy, 0.0%ni, 62.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
# time eth1(ikb ipk okb opk) eth2(ikb ipk okb opk)
1357058637 18319.3 38165.5 9401736.3 65159.9 9761613.4 808963.3 19403.6 40424.4
1357058638 20009.8 41687.7 9400903.7 62706.6 9770555.8 809522.2 18696.5 38951.1
1357058639 25439.5 52999.9 9301635.3 50267.7 9773666.7 809721.1 19174.1 39946.6
1357058640 26808.2 55850.0 9298301.4 46876.6 9790470.1 810843.3 12408.7 25851.1
1357058641 27110.9 56481.1 9297009.2 46832.2 9803308.4 811339.9 5692.5 11859.9
1357058642 27411.1 57106.6 9291419.2 46796.6 9806846.5 811378.8 2804.4 5842.2
This kernel is getting really good :-)
Cheers,
Willy
next prev parent reply other threads:[~2013-01-06 19:53 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-05 21:49 Major network performance regression in 3.7 Willy Tarreau
2013-01-05 23:18 ` Eric Dumazet
2013-01-05 23:29 ` Willy Tarreau
2013-01-06 0:02 ` Eric Dumazet
2013-01-06 0:50 ` Willy Tarreau
2013-01-06 1:21 ` Eric Dumazet
2013-01-06 1:30 ` Willy Tarreau
2013-01-06 1:40 ` Eric Dumazet
2013-01-06 1:51 ` Eric Dumazet
2013-01-06 2:16 ` Eric Dumazet
2013-01-06 2:18 ` Willy Tarreau
2013-01-06 2:22 ` Eric Dumazet
2013-01-06 2:32 ` Willy Tarreau
2013-01-06 2:44 ` Eric Dumazet
2013-01-06 2:52 ` Willy Tarreau
2013-01-06 7:31 ` [PATCH net-next] net: splice: avoid high order page splitting Eric Dumazet
2013-01-07 5:07 ` David Miller
2013-01-06 7:35 ` Major network performance regression in 3.7 Eric Dumazet
2013-01-06 9:24 ` Willy Tarreau
2013-01-06 10:25 ` Willy Tarreau
2013-01-06 11:46 ` Romain Francoise
2013-01-06 11:53 ` Willy Tarreau
2013-01-06 12:01 ` Willy Tarreau
2013-01-06 14:59 ` Eric Dumazet
2013-01-06 15:51 ` Willy Tarreau
2013-01-06 16:39 ` Eric Dumazet
2013-01-06 16:44 ` Willy Tarreau
2013-01-06 17:10 ` Eric Dumazet
2013-01-06 17:35 ` Willy Tarreau
2013-01-06 18:39 ` Eric Dumazet
2013-01-06 18:43 ` Eric Dumazet
2013-01-06 18:51 ` Eric Dumazet
2013-01-06 19:00 ` Eric Dumazet
2013-01-06 19:34 ` Willy Tarreau
2013-01-06 19:39 ` Eric Dumazet
2013-01-06 19:53 ` Willy Tarreau [this message]
2013-01-07 4:21 ` [PATCH] tcp: fix MSG_SENDPAGE_NOTLAST logic Eric Dumazet
2013-01-07 4:59 ` David Miller
2013-01-06 21:49 ` Major network performance regression in 3.7 John Stoffel
2013-01-06 21:52 ` Willy Tarreau
2013-01-06 21:55 ` John Stoffel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130106195359.GN16031@1wt.eu \
--to=w@1wt.eu \
--cc=eric.dumazet@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).