* tx offload issue w/stubdoms + igb
@ 2010-12-14 10:12 John Weekes
2010-12-14 21:59 ` Jeremy Fitzhardinge
0 siblings, 1 reply; 4+ messages in thread
From: John Weekes @ 2010-12-14 10:12 UTC (permalink / raw)
To: xen-devel@lists.xensource.com
Running a very recent xen-unstable and xen/stable-2.6.32.x along with
any Linux domU using HVM and a stubdom, I notice that TCP performance
when downloading from certain sites is extremely low with dom0's tx
offload enabled on the stubdom's vif. For instance, from kernel.org, I
see a paltry 30-50 K/s from inside the domU:
testvds5 ~ # wget
http://mirrors.kernel.org/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
--2010-12-14 04:29:53--
http://mirrors.kernel.org/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
Resolving mirrors.kernel.org... 149.20.20.135, 204.152.191.39
Connecting to mirrors.kernel.org|149.20.20.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2798649344 (2.6G) [application/x-iso9660-image]
Saving to: `livedvd-amd64-multilib-10.1.iso.1'
0%
[
] 680,237 31.5K/s eta 18h 39m ^C
testds5 ~ #
But, if I turn off tx offload for the stubdom's vif with a line like
this in the dom0..
ethtool -K vif59.0 tx off
.. I then get normal speeds in the domU:
testvds5 ~ # wget
http://mirrors.kernel.org/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
--2010-12-14 04:31:44--
http://mirrors.kernel.org/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
Resolving mirrors.kernel.org... 149.20.20.135, 204.152.191.39
Connecting to mirrors.kernel.org|149.20.20.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2798649344 (2.6G) [application/x-iso9660-image]
Saving to: `livedvd-amd64-multilib-10.1.iso.5'
2%
[=>
] 73,762,776 1.59M/s eta 37m 57s ^C
I tested further and found that:
* dom0 does't have the issue, normal PV domains do not have the issue,
and Windows GPLPV-based domains do not have the issue. It seems to be
specific to stubdom-based domains.
* Other machines running the exact same Xen release and kernel version,
but that use the e1000 driver instead of the igb driver, don't seem to
have the problem. I don't know if it's related (I have not yet been able
to test with MSI disabled), but those machines without the problem also
aren't using MSI-X, whereas the igb-based machine that shows the problem
is. From dmesg:
[ 21.209923] Intel(R) Gigabit Ethernet Network Driver - version 1.3.16-k2
[ 21.210026] Copyright (c) 2007-2009 Intel Corporation.
[ 21.210140] xen: registering gsi 28 triggering 0 polarity 1
[ 21.210145] xen: --> irq=28
[ 21.210151] igb 0000:01:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
[ 21.210279] igb 0000:01:00.0: setting latency timer to 64
[ 21.382336] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network
Connection
[ 21.382435] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4)
00:25:90:09:e4:00
[ 21.382605] igb 0000:01:00.0: eth0: PBA No: ffffff-0ff
[ 21.382698] igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s),
4 tx queue(s)
(Both the e1000 and igb machines have the hvm_directio flag in the "xl
info" output.)
* Different GSO/TSO settings do not appear to make a difference. Only
the tx offload setting does.
* Inside the problematic domU, the bad segment counter increments when
the issue is occurring:
testvds5 ~ # netstat -s eth0
Ip:
22162 total packets received
44 with invalid addresses
0 forwarded
0 incoming packets discarded
22113 incoming packets delivered
19582 requests sent out
Icmp:
2694 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
timeout in transit: 2447
echo replies: 247
2698 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 2
IcmpMsg:
InType0: 247
InType11: 2447
OutType3: 2
OutType69: 2696
Tcp:
4 active connections openings
3 passive connection openings
0 failed connection attempts
0 connection resets received
3 connections established
18819 segments received
16795 segments send out
0 segments retransmited
2366 bad segments received.
8 resets sent
Udp:
65 packets received
2 packets to unknown port received.
0 packet receive errors
89 packets sent
UdpLite:
TcpExt:
1 TCP sockets finished time wait in fast timer
172 delayed acks sent
Quick ack mode was activated 89 times
3 packets directly queued to recvmsg prequeue.
33304 bytes directly in process context from backlog
3 bytes directly received in process context from prequeue
7236 packet headers predicted
23 packets header predicted and directly queued to user
3117 acknowledgments not containing data payload received
89 DSACKs sent for old packets
2 DSACKs sent for out of order packets
2 connections reset due to unexpected data
IpExt:
InBcastPkts: 533
InOctets: 23420805
OutOctets: 1601733
InBcastOctets: 162268
testvds5 ~ #
* Some sites transfer quickly to the domU quickly regardless of the tx
offload setting, exhibiting the symptoms less. For instance, uiuc.edu
with tx on:
root@testvds5:~# wget
http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
--2010-12-14 03:53:50--
http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
Resolving gentoo.cites.uiuc.edu... 128.174.5.78
Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2798649344 (2.6G) [text/plain]
Saving to: `livedvd-amd64-multilib-10.1.iso'
0% [ ] 25,754,272 3.06M/s eta
17m 7s ^C
root@testvds5:~#
(netstat shows 23 bad segments received over the length of that test)
and with tx off:
root@testvds5:~# wget
http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
--2010-12-14 03:54:45--
http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
Resolving gentoo.cites.uiuc.edu... 128.174.5.78
Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2798649344 (2.6G) [text/plain]
Saving to: `livedvd-amd64-multilib-10.1.iso.1'
1% [ ] 47,677,960 3.95M/s eta
12m 0s ^C
* The issue also occurs in xen-4.0-testing, as of c/s 21392.
For reference, Xen and kernel version output:
nyc-dodec266 src # xl info
host : nyc-dodec266
release : 2.6.32.26-g862ef97
version : #4 SMP Wed Dec 8 16:38:18 EST 2010
machine : x86_64
nr_cpus : 24
nr_nodes : 2
cores_per_socket : 12
threads_per_core : 1
cpu_mhz : 2674
hw_caps :
bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 49143
free_memory : 9178
free_cpus : 0
xen_major : 4
xen_minor : 1
xen_extra : -unstable
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Wed Dec 08 10:46:31 2010 +0000 22467:89116f28083f
xen_commandline : dom0_mem=2550M dom0_max_vcpus=4
cc_compiler : gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.2, pie-0.4.5)
cc_compile_by : root
cc_compile_domain : nuclearfallout.net
cc_compile_date : Fri Dec 10 00:51:50 EST 2010
xend_config_format : 4
nyc-dodec266 src # uname -a
Linux nyc-dodec266 2.6.32.26-g862ef97 #4 SMP Wed Dec 8 16:38:18 EST 2010
x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
For now, I can use the "tx off" workaround by having a script set it for
all newly created domains. Is anyone up for nailing this down and
finding a real fix? Failing that, applying the workaround in the Xen
tools might be a good idea.
-John
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: tx offload issue w/stubdoms + igb
2010-12-14 10:12 tx offload issue w/stubdoms + igb John Weekes
@ 2010-12-14 21:59 ` Jeremy Fitzhardinge
2010-12-14 22:00 ` Jeremy Fitzhardinge
0 siblings, 1 reply; 4+ messages in thread
From: Jeremy Fitzhardinge @ 2010-12-14 21:59 UTC (permalink / raw)
To: John Weekes; +Cc: Samuel Thibault, xen-devel@lists.xensource.com, Ian Campbell
On 12/14/2010 02:12 AM, John Weekes wrote:
> I tested further and found that:
>
> * dom0 does't have the issue, normal PV domains do not have the issue,
> and Windows GPLPV-based domains do not have the issue. It seems to be
> specific to stubdom-based domains.
That's interesting. There were a number of fixes to netfront/back to
make sure all this checksum offload stuff worked properly, and I was
never convinced they were also ported to stubdom's netfront. I don't
remember the specifics now, unfortunately.
J
>
> * Other machines running the exact same Xen release and kernel
> version, but that use the e1000 driver instead of the igb driver,
> don't seem to have the problem. I don't know if it's related (I have
> not yet been able to test with MSI disabled), but those machines
> without the problem also aren't using MSI-X, whereas the igb-based
> machine that shows the problem is. From dmesg:
>
> [ 21.209923] Intel(R) Gigabit Ethernet Network Driver - version
> 1.3.16-k2
> [ 21.210026] Copyright (c) 2007-2009 Intel Corporation.
> [ 21.210140] xen: registering gsi 28 triggering 0 polarity 1
> [ 21.210145] xen: --> irq=28
> [ 21.210151] igb 0000:01:00.0: PCI INT A -> GSI 28 (level, low) ->
> IRQ 28
> [ 21.210279] igb 0000:01:00.0: setting latency timer to 64
> [ 21.382336] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network
> Connection
> [ 21.382435] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4)
> 00:25:90:09:e4:00
> [ 21.382605] igb 0000:01:00.0: eth0: PBA No: ffffff-0ff
> [ 21.382698] igb 0000:01:00.0: Using MSI-X interrupts. 4 rx
> queue(s), 4 tx queue(s)
>
> (Both the e1000 and igb machines have the hvm_directio flag in the "xl
> info" output.)
>
> * Different GSO/TSO settings do not appear to make a difference. Only
> the tx offload setting does.
>
> * Inside the problematic domU, the bad segment counter increments when
> the issue is occurring:
>
> testvds5 ~ # netstat -s eth0
> Ip:
> 22162 total packets received
> 44 with invalid addresses
> 0 forwarded
> 0 incoming packets discarded
> 22113 incoming packets delivered
> 19582 requests sent out
> Icmp:
> 2694 ICMP messages received
> 0 input ICMP message failed.
> ICMP input histogram:
> timeout in transit: 2447
> echo replies: 247
> 2698 ICMP messages sent
> 0 ICMP messages failed
> ICMP output histogram:
> destination unreachable: 2
> IcmpMsg:
> InType0: 247
> InType11: 2447
> OutType3: 2
> OutType69: 2696
> Tcp:
> 4 active connections openings
> 3 passive connection openings
> 0 failed connection attempts
> 0 connection resets received
> 3 connections established
> 18819 segments received
> 16795 segments send out
> 0 segments retransmited
> 2366 bad segments received.
> 8 resets sent
> Udp:
> 65 packets received
> 2 packets to unknown port received.
> 0 packet receive errors
> 89 packets sent
> UdpLite:
> TcpExt:
> 1 TCP sockets finished time wait in fast timer
> 172 delayed acks sent
> Quick ack mode was activated 89 times
> 3 packets directly queued to recvmsg prequeue.
> 33304 bytes directly in process context from backlog
> 3 bytes directly received in process context from prequeue
> 7236 packet headers predicted
> 23 packets header predicted and directly queued to user
> 3117 acknowledgments not containing data payload received
> 89 DSACKs sent for old packets
> 2 DSACKs sent for out of order packets
> 2 connections reset due to unexpected data
> IpExt:
> InBcastPkts: 533
> InOctets: 23420805
> OutOctets: 1601733
> InBcastOctets: 162268
> testvds5 ~ #
>
> * Some sites transfer quickly to the domU quickly regardless of the tx
> offload setting, exhibiting the symptoms less. For instance, uiuc.edu
> with tx on:
>
> root@testvds5:~# wget
> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
> --2010-12-14 03:53:50--
> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
> Resolving gentoo.cites.uiuc.edu... 128.174.5.78
> Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 2798649344 (2.6G) [text/plain]
> Saving to: `livedvd-amd64-multilib-10.1.iso'
>
> 0% [ ] 25,754,272 3.06M/s eta
> 17m 7s ^C
> root@testvds5:~#
>
> (netstat shows 23 bad segments received over the length of that test)
>
> and with tx off:
>
> root@testvds5:~# wget
> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
> --2010-12-14 03:54:45--
> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
> Resolving gentoo.cites.uiuc.edu... 128.174.5.78
> Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 2798649344 (2.6G) [text/plain]
> Saving to: `livedvd-amd64-multilib-10.1.iso.1'
>
> 1% [ ] 47,677,960 3.95M/s eta
> 12m 0s ^C
>
> * The issue also occurs in xen-4.0-testing, as of c/s 21392.
>
> For reference, Xen and kernel version output:
>
> nyc-dodec266 src # xl info
> host : nyc-dodec266
> release : 2.6.32.26-g862ef97
> version : #4 SMP Wed Dec 8 16:38:18 EST 2010
> machine : x86_64
> nr_cpus : 24
> nr_nodes : 2
> cores_per_socket : 12
> threads_per_core : 1
> cpu_mhz : 2674
> hw_caps :
> bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
> virt_caps : hvm hvm_directio
> total_memory : 49143
> free_memory : 9178
> free_cpus : 0
> xen_major : 4
> xen_minor : 1
> xen_extra : -unstable
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset : Wed Dec 08 10:46:31 2010 +0000
> 22467:89116f28083f
> xen_commandline : dom0_mem=2550M dom0_max_vcpus=4
> cc_compiler : gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.2,
> pie-0.4.5)
> cc_compile_by : root
> cc_compile_domain : nuclearfallout.net
> cc_compile_date : Fri Dec 10 00:51:50 EST 2010
> xend_config_format : 4
> nyc-dodec266 src # uname -a
> Linux nyc-dodec266 2.6.32.26-g862ef97 #4 SMP Wed Dec 8 16:38:18 EST
> 2010 x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
>
> For now, I can use the "tx off" workaround by having a script set it
> for all newly created domains. Is anyone up for nailing this down and
> finding a real fix? Failing that, applying the workaround in the Xen
> tools might be a good idea.
>
> -John
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: tx offload issue w/stubdoms + igb
2010-12-14 21:59 ` Jeremy Fitzhardinge
@ 2010-12-14 22:00 ` Jeremy Fitzhardinge
2011-03-12 20:33 ` John Weekes
0 siblings, 1 reply; 4+ messages in thread
From: Jeremy Fitzhardinge @ 2010-12-14 22:00 UTC (permalink / raw)
To: John Weekes
Cc: Samuel Thibault, xen-devel@lists.xensource.com, Ian Campbell,
Stefano Stabellini
(Added Stefano.)
On 12/14/2010 01:59 PM, Jeremy Fitzhardinge wrote:
> On 12/14/2010 02:12 AM, John Weekes wrote:
>> I tested further and found that:
>>
>> * dom0 does't have the issue, normal PV domains do not have the issue,
>> and Windows GPLPV-based domains do not have the issue. It seems to be
>> specific to stubdom-based domains.
> That's interesting. There were a number of fixes to netfront/back to
> make sure all this checksum offload stuff worked properly, and I was
> never convinced they were also ported to stubdom's netfront. I don't
> remember the specifics now, unfortunately.
>
> J
>
>> * Other machines running the exact same Xen release and kernel
>> version, but that use the e1000 driver instead of the igb driver,
>> don't seem to have the problem. I don't know if it's related (I have
>> not yet been able to test with MSI disabled), but those machines
>> without the problem also aren't using MSI-X, whereas the igb-based
>> machine that shows the problem is. From dmesg:
>>
>> [ 21.209923] Intel(R) Gigabit Ethernet Network Driver - version
>> 1.3.16-k2
>> [ 21.210026] Copyright (c) 2007-2009 Intel Corporation.
>> [ 21.210140] xen: registering gsi 28 triggering 0 polarity 1
>> [ 21.210145] xen: --> irq=28
>> [ 21.210151] igb 0000:01:00.0: PCI INT A -> GSI 28 (level, low) ->
>> IRQ 28
>> [ 21.210279] igb 0000:01:00.0: setting latency timer to 64
>> [ 21.382336] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network
>> Connection
>> [ 21.382435] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4)
>> 00:25:90:09:e4:00
>> [ 21.382605] igb 0000:01:00.0: eth0: PBA No: ffffff-0ff
>> [ 21.382698] igb 0000:01:00.0: Using MSI-X interrupts. 4 rx
>> queue(s), 4 tx queue(s)
>>
>> (Both the e1000 and igb machines have the hvm_directio flag in the "xl
>> info" output.)
>>
>> * Different GSO/TSO settings do not appear to make a difference. Only
>> the tx offload setting does.
>>
>> * Inside the problematic domU, the bad segment counter increments when
>> the issue is occurring:
>>
>> testvds5 ~ # netstat -s eth0
>> Ip:
>> 22162 total packets received
>> 44 with invalid addresses
>> 0 forwarded
>> 0 incoming packets discarded
>> 22113 incoming packets delivered
>> 19582 requests sent out
>> Icmp:
>> 2694 ICMP messages received
>> 0 input ICMP message failed.
>> ICMP input histogram:
>> timeout in transit: 2447
>> echo replies: 247
>> 2698 ICMP messages sent
>> 0 ICMP messages failed
>> ICMP output histogram:
>> destination unreachable: 2
>> IcmpMsg:
>> InType0: 247
>> InType11: 2447
>> OutType3: 2
>> OutType69: 2696
>> Tcp:
>> 4 active connections openings
>> 3 passive connection openings
>> 0 failed connection attempts
>> 0 connection resets received
>> 3 connections established
>> 18819 segments received
>> 16795 segments send out
>> 0 segments retransmited
>> 2366 bad segments received.
>> 8 resets sent
>> Udp:
>> 65 packets received
>> 2 packets to unknown port received.
>> 0 packet receive errors
>> 89 packets sent
>> UdpLite:
>> TcpExt:
>> 1 TCP sockets finished time wait in fast timer
>> 172 delayed acks sent
>> Quick ack mode was activated 89 times
>> 3 packets directly queued to recvmsg prequeue.
>> 33304 bytes directly in process context from backlog
>> 3 bytes directly received in process context from prequeue
>> 7236 packet headers predicted
>> 23 packets header predicted and directly queued to user
>> 3117 acknowledgments not containing data payload received
>> 89 DSACKs sent for old packets
>> 2 DSACKs sent for out of order packets
>> 2 connections reset due to unexpected data
>> IpExt:
>> InBcastPkts: 533
>> InOctets: 23420805
>> OutOctets: 1601733
>> InBcastOctets: 162268
>> testvds5 ~ #
>>
>> * Some sites transfer quickly to the domU quickly regardless of the tx
>> offload setting, exhibiting the symptoms less. For instance, uiuc.edu
>> with tx on:
>>
>> root@testvds5:~# wget
>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>> --2010-12-14 03:53:50--
>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>> Resolving gentoo.cites.uiuc.edu... 128.174.5.78
>> Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 2798649344 (2.6G) [text/plain]
>> Saving to: `livedvd-amd64-multilib-10.1.iso'
>>
>> 0% [ ] 25,754,272 3.06M/s eta
>> 17m 7s ^C
>> root@testvds5:~#
>>
>> (netstat shows 23 bad segments received over the length of that test)
>>
>> and with tx off:
>>
>> root@testvds5:~# wget
>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>> --2010-12-14 03:54:45--
>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>> Resolving gentoo.cites.uiuc.edu... 128.174.5.78
>> Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 2798649344 (2.6G) [text/plain]
>> Saving to: `livedvd-amd64-multilib-10.1.iso.1'
>>
>> 1% [ ] 47,677,960 3.95M/s eta
>> 12m 0s ^C
>>
>> * The issue also occurs in xen-4.0-testing, as of c/s 21392.
>>
>> For reference, Xen and kernel version output:
>>
>> nyc-dodec266 src # xl info
>> host : nyc-dodec266
>> release : 2.6.32.26-g862ef97
>> version : #4 SMP Wed Dec 8 16:38:18 EST 2010
>> machine : x86_64
>> nr_cpus : 24
>> nr_nodes : 2
>> cores_per_socket : 12
>> threads_per_core : 1
>> cpu_mhz : 2674
>> hw_caps :
>> bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
>> virt_caps : hvm hvm_directio
>> total_memory : 49143
>> free_memory : 9178
>> free_cpus : 0
>> xen_major : 4
>> xen_minor : 1
>> xen_extra : -unstable
>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>> hvm-3.0-x86_32p hvm-3.0-x86_64
>> xen_scheduler : credit
>> xen_pagesize : 4096
>> platform_params : virt_start=0xffff800000000000
>> xen_changeset : Wed Dec 08 10:46:31 2010 +0000
>> 22467:89116f28083f
>> xen_commandline : dom0_mem=2550M dom0_max_vcpus=4
>> cc_compiler : gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.2,
>> pie-0.4.5)
>> cc_compile_by : root
>> cc_compile_domain : nuclearfallout.net
>> cc_compile_date : Fri Dec 10 00:51:50 EST 2010
>> xend_config_format : 4
>> nyc-dodec266 src # uname -a
>> Linux nyc-dodec266 2.6.32.26-g862ef97 #4 SMP Wed Dec 8 16:38:18 EST
>> 2010 x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
>>
>> For now, I can use the "tx off" workaround by having a script set it
>> for all newly created domains. Is anyone up for nailing this down and
>> finding a real fix? Failing that, applying the workaround in the Xen
>> tools might be a good idea.
>>
>> -John
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: tx offload issue w/stubdoms + igb
2010-12-14 22:00 ` Jeremy Fitzhardinge
@ 2011-03-12 20:33 ` John Weekes
0 siblings, 0 replies; 4+ messages in thread
From: John Weekes @ 2011-03-12 20:33 UTC (permalink / raw)
To: Jeremy Fitzhardinge
Cc: Samuel Thibault, xen-devel@lists.xensource.com, Ian Campbell,
Stefano Stabellini
Has anyone had a chance to look at the below again? It looks like it's
still broken.
It also seems that tso needs to be turned off from inside the domU for
outgoing transfers to work properly. Certain packets just don't get
through otherwise.
PV-on-HVM addresses it, of course, but that's not always doable.
-John
On 12/14/2010 2:00 PM, Jeremy Fitzhardinge wrote:
> (Added Stefano.)
>
> On 12/14/2010 01:59 PM, Jeremy Fitzhardinge wrote:
>> On 12/14/2010 02:12 AM, John Weekes wrote:
>>> I tested further and found that:
>>>
>>> * dom0 does't have the issue, normal PV domains do not have the issue,
>>> and Windows GPLPV-based domains do not have the issue. It seems to be
>>> specific to stubdom-based domains.
>> That's interesting. There were a number of fixes to netfront/back to
>> make sure all this checksum offload stuff worked properly, and I was
>> never convinced they were also ported to stubdom's netfront. I don't
>> remember the specifics now, unfortunately.
>>
>> J
>>
>>> * Other machines running the exact same Xen release and kernel
>>> version, but that use the e1000 driver instead of the igb driver,
>>> don't seem to have the problem. I don't know if it's related (I have
>>> not yet been able to test with MSI disabled), but those machines
>>> without the problem also aren't using MSI-X, whereas the igb-based
>>> machine that shows the problem is. From dmesg:
>>>
>>> [ 21.209923] Intel(R) Gigabit Ethernet Network Driver - version
>>> 1.3.16-k2
>>> [ 21.210026] Copyright (c) 2007-2009 Intel Corporation.
>>> [ 21.210140] xen: registering gsi 28 triggering 0 polarity 1
>>> [ 21.210145] xen: --> irq=28
>>> [ 21.210151] igb 0000:01:00.0: PCI INT A -> GSI 28 (level, low) ->
>>> IRQ 28
>>> [ 21.210279] igb 0000:01:00.0: setting latency timer to 64
>>> [ 21.382336] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network
>>> Connection
>>> [ 21.382435] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4)
>>> 00:25:90:09:e4:00
>>> [ 21.382605] igb 0000:01:00.0: eth0: PBA No: ffffff-0ff
>>> [ 21.382698] igb 0000:01:00.0: Using MSI-X interrupts. 4 rx
>>> queue(s), 4 tx queue(s)
>>>
>>> (Both the e1000 and igb machines have the hvm_directio flag in the "xl
>>> info" output.)
>>>
>>> * Different GSO/TSO settings do not appear to make a difference. Only
>>> the tx offload setting does.
>>>
>>> * Inside the problematic domU, the bad segment counter increments when
>>> the issue is occurring:
>>>
>>> testvds5 ~ # netstat -s eth0
>>> Ip:
>>> 22162 total packets received
>>> 44 with invalid addresses
>>> 0 forwarded
>>> 0 incoming packets discarded
>>> 22113 incoming packets delivered
>>> 19582 requests sent out
>>> Icmp:
>>> 2694 ICMP messages received
>>> 0 input ICMP message failed.
>>> ICMP input histogram:
>>> timeout in transit: 2447
>>> echo replies: 247
>>> 2698 ICMP messages sent
>>> 0 ICMP messages failed
>>> ICMP output histogram:
>>> destination unreachable: 2
>>> IcmpMsg:
>>> InType0: 247
>>> InType11: 2447
>>> OutType3: 2
>>> OutType69: 2696
>>> Tcp:
>>> 4 active connections openings
>>> 3 passive connection openings
>>> 0 failed connection attempts
>>> 0 connection resets received
>>> 3 connections established
>>> 18819 segments received
>>> 16795 segments send out
>>> 0 segments retransmited
>>> 2366 bad segments received.
>>> 8 resets sent
>>> Udp:
>>> 65 packets received
>>> 2 packets to unknown port received.
>>> 0 packet receive errors
>>> 89 packets sent
>>> UdpLite:
>>> TcpExt:
>>> 1 TCP sockets finished time wait in fast timer
>>> 172 delayed acks sent
>>> Quick ack mode was activated 89 times
>>> 3 packets directly queued to recvmsg prequeue.
>>> 33304 bytes directly in process context from backlog
>>> 3 bytes directly received in process context from prequeue
>>> 7236 packet headers predicted
>>> 23 packets header predicted and directly queued to user
>>> 3117 acknowledgments not containing data payload received
>>> 89 DSACKs sent for old packets
>>> 2 DSACKs sent for out of order packets
>>> 2 connections reset due to unexpected data
>>> IpExt:
>>> InBcastPkts: 533
>>> InOctets: 23420805
>>> OutOctets: 1601733
>>> InBcastOctets: 162268
>>> testvds5 ~ #
>>>
>>> * Some sites transfer quickly to the domU quickly regardless of the tx
>>> offload setting, exhibiting the symptoms less. For instance, uiuc.edu
>>> with tx on:
>>>
>>> root@testvds5:~# wget
>>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>>> --2010-12-14 03:53:50--
>>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>>> Resolving gentoo.cites.uiuc.edu... 128.174.5.78
>>> Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 2798649344 (2.6G) [text/plain]
>>> Saving to: `livedvd-amd64-multilib-10.1.iso'
>>>
>>> 0% [ ] 25,754,272 3.06M/s eta
>>> 17m 7s ^C
>>> root@testvds5:~#
>>>
>>> (netstat shows 23 bad segments received over the length of that test)
>>>
>>> and with tx off:
>>>
>>> root@testvds5:~# wget
>>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>>> --2010-12-14 03:54:45--
>>> http://gentoo.cites.uiuc.edu/pub/gentoo/releases/amd64/10.1/livedvd-amd64-multilib-10.1.iso
>>> Resolving gentoo.cites.uiuc.edu... 128.174.5.78
>>> Connecting to gentoo.cites.uiuc.edu|128.174.5.78|:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 2798649344 (2.6G) [text/plain]
>>> Saving to: `livedvd-amd64-multilib-10.1.iso.1'
>>>
>>> 1% [ ] 47,677,960 3.95M/s eta
>>> 12m 0s ^C
>>>
>>> * The issue also occurs in xen-4.0-testing, as of c/s 21392.
>>>
>>> For reference, Xen and kernel version output:
>>>
>>> nyc-dodec266 src # xl info
>>> host : nyc-dodec266
>>> release : 2.6.32.26-g862ef97
>>> version : #4 SMP Wed Dec 8 16:38:18 EST 2010
>>> machine : x86_64
>>> nr_cpus : 24
>>> nr_nodes : 2
>>> cores_per_socket : 12
>>> threads_per_core : 1
>>> cpu_mhz : 2674
>>> hw_caps :
>>> bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
>>> virt_caps : hvm hvm_directio
>>> total_memory : 49143
>>> free_memory : 9178
>>> free_cpus : 0
>>> xen_major : 4
>>> xen_minor : 1
>>> xen_extra : -unstable
>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>> xen_scheduler : credit
>>> xen_pagesize : 4096
>>> platform_params : virt_start=0xffff800000000000
>>> xen_changeset : Wed Dec 08 10:46:31 2010 +0000
>>> 22467:89116f28083f
>>> xen_commandline : dom0_mem=2550M dom0_max_vcpus=4
>>> cc_compiler : gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.2,
>>> pie-0.4.5)
>>> cc_compile_by : root
>>> cc_compile_domain : nuclearfallout.net
>>> cc_compile_date : Fri Dec 10 00:51:50 EST 2010
>>> xend_config_format : 4
>>> nyc-dodec266 src # uname -a
>>> Linux nyc-dodec266 2.6.32.26-g862ef97 #4 SMP Wed Dec 8 16:38:18 EST
>>> 2010 x86_64 Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
>>>
>>> For now, I can use the "tx off" workaround by having a script set it
>>> for all newly created domains. Is anyone up for nailing this down and
>>> finding a real fix? Failing that, applying the workaround in the Xen
>>> tools might be a good idea.
>>>
>>> -John
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2011-03-12 20:33 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-12-14 10:12 tx offload issue w/stubdoms + igb John Weekes
2010-12-14 21:59 ` Jeremy Fitzhardinge
2010-12-14 22:00 ` Jeremy Fitzhardinge
2011-03-12 20:33 ` John Weekes
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).