From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Ronny.Hegewald@online.de, ian.campbell@citrix.com
Cc: xen-devel@lists.xensource.com
Subject: Re: huge tcp performance-regression on pvops - kernel (+Solution)
Date: Tue, 27 Oct 2009 10:54:07 -0400 [thread overview]
Message-ID: <20091027145407.GD1193@phenom.dumpdata.com> (raw)
In-Reply-To: <733454931.1283414.1256600244808.JavaMail.tomcat55@mrmseu2.kundenserver.de>
On Tue, Oct 27, 2009 at 12:37:25AM +0100, Ronny.Hegewald@online.de wrote:
> < xmlns="http://www.w3.org/1999/x" xml:lang="en" lang="en"><title></title><head><meta http-equiv="Content-type" content="text/; charset=UTF-8" /><style type="text/css"> , body {overflow-x: visible; } { width:100%; height:100%;margin:0px; padding:0px; overflow-y: auto; overflow-x: auto; }body { font-size: 100.01%; font-family : Verdana, Geneva, Arial, Helvetica, sans-serif; background-color:transparent; overflow:show; background-image:none; margin:0px; padding:5px; }p { margin:0px; padding:0px; } body { font-size: 12px; font-family : Verdana, Geneva, Arial, Helvetica, sans-serif; } p { margin: 0; padding: 0; } blockquote { padding-left: 5px; margin-left: 5px; margin-bottom: 0px; margin-top: 0px; } blockquote.quote { border-left: 1px solid #CCC; padding-left: 5px; margin-left: 5px; } .misspelled { background: transparent url(//webmailerng.1und1.de/static_resource/mailclient/widgets/basic/parts/maileditor/spellchecking_underline.gif) repeat-x scroll center bottom; } .correct {} .unknown {} .ignored {}</style></head><body id="bodyElement" style="">
> <p>Setup: pvops dom0 kernel 2.6.31.4 from Jeremys git-repository from 2009-10-18<span></span></p><p id="__paragraph__1256597098000" style=""><span style=""><br></span></p>Problem: Very slow network-performance (3-10 kbs) when tcp-packets are used (noticed when using scp, samba, nfs over tcp)<p id="__paragraph__1256597073000" style=""><br></p><p id="__paragraph__1256597073000" style="">But this occurs only in the following situations:<br></p><p id="__paragraph__1256597073000" style=""> <span></span></p><p id="__paragraph__1256597073000" style="">1.) domU to domU on same PC (domUs are paravirtualized linux-kernels)<br></p><p id="__paragraph__1256597073000" style="">2.) domU to another PC that doesn't use<span id="misspelled-have" class="misspelled" name="misspelled-have"></span> pvops-kernel (packets sent from another PC to domU works fine)<br></p><br>domU to dom0 and the opposite way works without performance-regression.<br><br><p>Reason: bigger tcp-packets get dropped from the domU the tcp-packets are sent from (netstat -s in domU shows many retransmitted tcp-segments)</p><p><br></p><p>tcpdump shows that the bigger packets leave the vif from the domU they were sent from, but never arrive the vif from the domU they are sent to.<br><span></span></p><br><span style=""></span><p>This is caused by this lines in drivers/xen/netback.c at line 1325 :<span></span></p><p id="__paragraph__1256597423000" style=""><br><span style=""></span></p><p id="__paragraph__1256597423000" style=""> if (skb->data_len < skb_shinfo(skb)->gso_size) {
> </p><div class="pre"> skb_shinfo(skb)->gso_size = 0;</div>
> <div class="pre"> skb_shinfo(skb)->gso_type = 0;</div>
> <div class="pre"> }</div><br><p id="__paragraph__1256597121000" style="">These lines were reverted from the linux-2.6.18-xen mercurial repository <br></p><p id="__paragraph__1256597121000" style=""><br></p><p id="__paragraph__1256597121000" style="">on 2009-01-13 in changeset 774:
> <a href="http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/107e10e0e07c">107e10e0e07c: netfront/back: do not mark packets of length < MSS as GSO</a></p><p id="__paragraph__1256598462000" style=""><br><span style=""></span></p><p id="__paragraph__1256598462000" style=""><span style="">I used the patch on the above mentioned pvops tree and the problem was gone.</span><span></span></p><p id="__paragraph__1256600217000" style=""><br><span style=""></span></p><p id="__paragraph__1256600217000" style=""><span style="">I never noticed such problems on the 2.6.18-xen kernel or the forward-ported xen-kernel 2.6.31.4 (from </span>Andrew Lyon)</p></body></>
>
Kudos for discovering this. Did you verify whether the domU has the corresponding
patch:
diff -r 28acedb66302 -r 107e10e0e07c drivers/xen/netfront/netfront.c
--- a/drivers/xen/netfront/netfront.c Wed Jan 07 12:21:54 2009 +0900
+++ b/drivers/xen/netfront/netfront.c Tue Jan 13 15:17:54 2009 +0000
@@ -1439,6 +1439,14 @@
np->stats.rx_packets++;
np->stats.rx_bytes += skb->len;
+#if HAVE_TSO
+ if (skb->data_len < skb_shinfo(skb)->gso_size) {
+ skb_shinfo(skb)->gso_size = 0;
+#if HAVE_GSO
+ skb_shinfo(skb)->gso_type = 0;
+#endif
+ }
+#endif
__skb_queue_tail(&rxq, skb);
np->rx.rsp_cons = ++i;
Looking at the PV_OPs kernel it doesn't look to be there, but I was
wondering if the domU you are using has it?
Ian,
Would it make sense to remove this changeset if most of the DomU's
don't have this fix?
next prev parent reply other threads:[~2009-10-27 14:54 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-10-26 23:37 huge tcp performance-regression on pvops - kernel (+Solution) Ronny.Hegewald
2009-10-27 14:53 ` Keir Fraser
2009-10-27 14:54 ` Konrad Rzeszutek Wilk [this message]
2009-10-27 20:34 ` Re: huge tcp performance-regression on pvops - kernel Ronny.Hegewald
2009-10-27 20:42 ` Jeremy Fitzhardinge
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20091027145407.GD1193@phenom.dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=Ronny.Hegewald@online.de \
--cc=ian.campbell@citrix.com \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.