* huge tcp performance-regression on pvops - kernel (+Solution)
@ 2009-10-26 23:37 Ronny.Hegewald
2009-10-27 14:53 ` Keir Fraser
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Ronny.Hegewald @ 2009-10-26 23:37 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/html, Size: 4250 bytes --]
[-- Attachment #2: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: huge tcp performance-regression on pvops - kernel (+Solution)
2009-10-26 23:37 huge tcp performance-regression on pvops - kernel (+Solution) Ronny.Hegewald
@ 2009-10-27 14:53 ` Keir Fraser
2009-10-27 14:54 ` Konrad Rzeszutek Wilk
2009-10-27 20:34 ` Re: huge tcp performance-regression on pvops - kernel Ronny.Hegewald
2 siblings, 0 replies; 5+ messages in thread
From: Keir Fraser @ 2009-10-27 14:53 UTC (permalink / raw)
To: Ronny.Hegewald@online.de, xen-devel@lists.xensource.com
Cc¹ing Jeremy. I think the patch of interest in linux-2.6.18-xen is 776,
which is the one that removes those lines from the front/back drivers in
favour of a solution integrated into the protocol stack.
-- Keir
On 26/10/2009 23:37, "Ronny.Hegewald@online.de" <Ronny.Hegewald@online.de>
wrote:
> Setup: pvops dom0 kernel 2.6.31.4 from Jeremys git-repository from 2009-10-18
>
>
> Problem: Very slow network-performance (3-10 kbs) when tcp-packets are used
> (noticed when using scp, samba, nfs over tcp)
>
>
>
> But this occurs only in the following situations:
>
>
>
> 1.) domU to domU on same PC (domUs are paravirtualized linux-kernels)
>
> 2.) domU to another PC that doesn't use pvops-kernel (packets sent from
> another PC to domU works fine)
>
> domU to dom0 and the opposite way works without performance-regression.
>
>
> Reason: bigger tcp-packets get dropped from the domU the tcp-packets are sent
> from (netstat -s in domU shows many retransmitted tcp-segments)
>
>
>
> tcpdump shows that the bigger packets leave the vif from the domU they were
> sent from, but never arrive the vif from the domU they are sent to.
>
>
> This is caused by this lines in drivers/xen/netback.c at line 1325 :
>
>
>
> if (skb->data_len < skb_shinfo(skb)->gso_size) {
>
> skb_shinfo(skb)->gso_size = 0;
> skb_shinfo(skb)->gso_type = 0;
> }
>
>
> These lines were reverted from the linux-2.6.18-xen mercurial repository
>
>
>
> on 2009-01-13 in changeset 774: 107e10e0e07c: netfront/back: do not mark
> packets of length < MSS as GSO
> <http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/107e10e0e07c>
>
>
>
> I used the patch on the above mentioned pvops tree and the problem was gone.
>
>
>
> I never noticed such problems on the 2.6.18-xen kernel or the forward-ported
> xen-kernel 2.6.31.4 (from Andrew Lyon)
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: huge tcp performance-regression on pvops - kernel (+Solution)
2009-10-26 23:37 huge tcp performance-regression on pvops - kernel (+Solution) Ronny.Hegewald
2009-10-27 14:53 ` Keir Fraser
@ 2009-10-27 14:54 ` Konrad Rzeszutek Wilk
2009-10-27 20:34 ` Re: huge tcp performance-regression on pvops - kernel Ronny.Hegewald
2 siblings, 0 replies; 5+ messages in thread
From: Konrad Rzeszutek Wilk @ 2009-10-27 14:54 UTC (permalink / raw)
To: Ronny.Hegewald, ian.campbell; +Cc: xen-devel
On Tue, Oct 27, 2009 at 12:37:25AM +0100, Ronny.Hegewald@online.de wrote:
> < xmlns="http://www.w3.org/1999/x" xml:lang="en" lang="en"><title></title><head><meta http-equiv="Content-type" content="text/; charset=UTF-8" /><style type="text/css"> , body {overflow-x: visible; } { width:100%; height:100%;margin:0px; padding:0px; overflow-y: auto; overflow-x: auto; }body { font-size: 100.01%; font-family : Verdana, Geneva, Arial, Helvetica, sans-serif; background-color:transparent; overflow:show; background-image:none; margin:0px; padding:5px; }p { margin:0px; padding:0px; } body { font-size: 12px; font-family : Verdana, Geneva, Arial, Helvetica, sans-serif; } p { margin: 0; padding: 0; } blockquote { padding-left: 5px; margin-left: 5px; margin-bottom: 0px; margin-top: 0px; } blockquote.quote { border-left: 1px solid #CCC; padding-left: 5px; margin-left: 5px; } .misspelled { background: transparent url(//webmailerng.1und1.de/static_resource/mailclient/widgets/basic/parts/maileditor/spellchecking_underline.gif) repeat-x scroll center bottom; } .correct {} .unknown {} .ignored {}</style></head><body id="bodyElement" style="">
> <p>Setup: pvops dom0 kernel 2.6.31.4 from Jeremys git-repository from 2009-10-18<span></span></p><p id="__paragraph__1256597098000" style=""><span style=""><br></span></p>Problem: Very slow network-performance (3-10 kbs) when tcp-packets are used (noticed when using scp, samba, nfs over tcp)<p id="__paragraph__1256597073000" style=""><br></p><p id="__paragraph__1256597073000" style="">But this occurs only in the following situations:<br></p><p id="__paragraph__1256597073000" style=""> <span></span></p><p id="__paragraph__1256597073000" style="">1.) domU to domU on same PC (domUs are paravirtualized linux-kernels)<br></p><p id="__paragraph__1256597073000" style="">2.) domU to another PC that doesn't use<span id="misspelled-have" class="misspelled" name="misspelled-have"></span> pvops-kernel (packets sent from another PC to domU works fine)<br></p><br>domU to dom0 and the opposite way works without performance-regression.<br><br><p>Reason: bigger tcp-packets get dropped from the domU the tcp-packets are sent from (netstat -s in domU shows many retransmitted tcp-segments)</p><p><br></p><p>tcpdump shows that the bigger packets leave the vif from the domU they were sent from, but never arrive the vif from the domU they are sent to.<br><span></span></p><br><span style=""></span><p>This is caused by this lines in drivers/xen/netback.c at line 1325 :<span></span></p><p id="__paragraph__1256597423000" style=""><br><span style=""></span></p><p id="__paragraph__1256597423000" style=""> if (skb->data_len < skb_shinfo(skb)->gso_size) {
> </p><div class="pre"> skb_shinfo(skb)->gso_size = 0;</div>
> <div class="pre"> skb_shinfo(skb)->gso_type = 0;</div>
> <div class="pre"> }</div><br><p id="__paragraph__1256597121000" style="">These lines were reverted from the linux-2.6.18-xen mercurial repository <br></p><p id="__paragraph__1256597121000" style=""><br></p><p id="__paragraph__1256597121000" style="">on 2009-01-13 in changeset 774:
> <a href="http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/107e10e0e07c">107e10e0e07c: netfront/back: do not mark packets of length < MSS as GSO</a></p><p id="__paragraph__1256598462000" style=""><br><span style=""></span></p><p id="__paragraph__1256598462000" style=""><span style="">I used the patch on the above mentioned pvops tree and the problem was gone.</span><span></span></p><p id="__paragraph__1256600217000" style=""><br><span style=""></span></p><p id="__paragraph__1256600217000" style=""><span style="">I never noticed such problems on the 2.6.18-xen kernel or the forward-ported xen-kernel 2.6.31.4 (from </span>Andrew Lyon)</p></body></>
>
Kudos for discovering this. Did you verify whether the domU has the corresponding
patch:
diff -r 28acedb66302 -r 107e10e0e07c drivers/xen/netfront/netfront.c
--- a/drivers/xen/netfront/netfront.c Wed Jan 07 12:21:54 2009 +0900
+++ b/drivers/xen/netfront/netfront.c Tue Jan 13 15:17:54 2009 +0000
@@ -1439,6 +1439,14 @@
np->stats.rx_packets++;
np->stats.rx_bytes += skb->len;
+#if HAVE_TSO
+ if (skb->data_len < skb_shinfo(skb)->gso_size) {
+ skb_shinfo(skb)->gso_size = 0;
+#if HAVE_GSO
+ skb_shinfo(skb)->gso_type = 0;
+#endif
+ }
+#endif
__skb_queue_tail(&rxq, skb);
np->rx.rsp_cons = ++i;
Looking at the PV_OPs kernel it doesn't look to be there, but I was
wondering if the domU you are using has it?
Ian,
Would it make sense to remove this changeset if most of the DomU's
don't have this fix?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: huge tcp performance-regression on pvops - kernel
2009-10-26 23:37 huge tcp performance-regression on pvops - kernel (+Solution) Ronny.Hegewald
2009-10-27 14:53 ` Keir Fraser
2009-10-27 14:54 ` Konrad Rzeszutek Wilk
@ 2009-10-27 20:34 ` Ronny.Hegewald
2009-10-27 20:42 ` Jeremy Fitzhardinge
2 siblings, 1 reply; 5+ messages in thread
From: Ronny.Hegewald @ 2009-10-27 20:34 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2003 bytes --]
Cc¹ing Jeremy. I think the patch of interest in linux-2.6.18-xen is 776,
which is the one that removes those lines from the front/back drivers in
favour of a solution integrated into the protocol stack.
-- Keir
On 26/10/2009 23:37, "Ronny.Hegewald@online.de" <Ronny.Hegewald@online.de>
wrote:
> Setup: pvops dom0 kernel 2.6.31.4 from Jeremys git-repository from 2009-10-18
>
>
> Problem: Very slow network-performance (3-10 kbs) when tcp-packets are used
> (noticed when using scp, samba, nfs over tcp)
>
>
>
> But this occurs only in the following situations:
>
>
>
> 1.) domU to domU on same PC (domUs are paravirtualized linux-kernels)
>
> 2.) domU to another PC that doesn't use pvops-kernel (packets sent from
> another PC to domU works fine)
>
> domU to dom0 and the opposite way works without performance-regression.
>
>
> Reason: bigger tcp-packets get dropped from the domU the tcp-packets are sent
> from (netstat -s in domU shows many retransmitted tcp-segments)
>
>
>
> tcpdump shows that the bigger packets leave the vif from the domU they were
> sent from, but never arrive the vif from the domU they are sent to.
>
>
> This is caused by this lines in drivers/xen/netback.c at line 1325 :
>
>
>
> if (skb->data_len < skb_shinfo(skb)->gso_size) {
>
> skb_shinfo(skb)->gso_size = 0;
> skb_shinfo(skb)->gso_type = 0;
> }
>
>
> These lines were reverted from the linux-2.6.18-xen mercurial repository
>
>
>
> on 2009-01-13 in changeset 774: 107e10e0e07c: netfront/back: do not mark
> packets of length < MSS as GSO
> <http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/107e10e0e07c>
>
>
>
> I used the patch on the above mentioned pvops tree and the problem was gone.
>
>
>
> I never noticed such problems on the 2.6.18-xen kernel or the forward-ported
> xen-kernel 2.6.31.4 (from Andrew Lyon)
>
[-- Attachment #1.2: Type: text/html, Size: 1793 bytes --]
[-- Attachment #2: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: huge tcp performance-regression on pvops - kernel
2009-10-27 20:34 ` Re: huge tcp performance-regression on pvops - kernel Ronny.Hegewald
@ 2009-10-27 20:42 ` Jeremy Fitzhardinge
0 siblings, 0 replies; 5+ messages in thread
From: Jeremy Fitzhardinge @ 2009-10-27 20:42 UTC (permalink / raw)
To: Ronny.Hegewald; +Cc: xen-devel
On 10/27/09 13:34, Ronny.Hegewald@online.de wrote:
>>
>> >> Cc¹ing Jeremy. I think the patch of interest in linux-2.6.18-xen
>> is 776,
>> >> which is the one that removes those lines from the front/back
>> drivers in
>> >> favour of a solution integrated into the protocol stack.
>>
>> >> -- Keir
>>
>>
>> Right, i patched it with 776 to solve the regression, not with 774.
>> Copied the wrong changeset in the email....
>>
>> And only the netback part, because the netfront-part of the patch
>> isnt in the pvops-kernel or upstream.
>
I'm about to push this into xen.git.
J
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2009-10-27 20:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-10-26 23:37 huge tcp performance-regression on pvops - kernel (+Solution) Ronny.Hegewald
2009-10-27 14:53 ` Keir Fraser
2009-10-27 14:54 ` Konrad Rzeszutek Wilk
2009-10-27 20:34 ` Re: huge tcp performance-regression on pvops - kernel Ronny.Hegewald
2009-10-27 20:42 ` Jeremy Fitzhardinge
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.