* Re: tcp_fastretrans_alert since 2.6.32 ? [not found] <AANLkTinnaA3Qgny45BRyW3i5u+nXn1XgWwkxA_QbN8xN@mail.gmail.com> @ 2010-09-22 6:48 ` Eric Dumazet 2010-09-25 2:34 ` Sunny 0 siblings, 1 reply; 2+ messages in thread From: Eric Dumazet @ 2010-09-22 6:48 UTC (permalink / raw) To: Sunny; +Cc: linux-kernel, netdev, Ilpo Järvinen CC netdev and Ilpo Le mardi 21 septembre 2010 à 23:37 -0700, Sunny a écrit : > Hi, > > After upgrading to the latest ubuntu kernel, I started to have this > alert showing up, this is a vm running on ESXi host with vmxnet3 > adapter, mainly serving large traffic ~70Mbps towards overboard > customers, so I expect packet loss and retrans is pretty high. So I'm > posting to ask whether others been seeing the same issue? When this > happens, machine freeze several seconds for about three times in a row > then seems to start working again. > > Please cc me on the thread, thanks in advance. > > > [212670.327617] ------------[ cut here ]------------ > [212670.327689] WARNING: at > /build/buildd/linux-2.6.32/net/ipv4/tcp_input.c:2919 tcp_fastret > rans_alert+0x3f9/0xd90() > [212670.327692] Hardware name: VMware Virtual Platform > [212670.327693] Modules linked in: btrfs zlib_deflate crc32c libcrc32c > ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs > reiserfs sch_sfq cls_u32 sch_htb nf_conntrack_ipv 6 ip6table_filter > ip6_tables xt_tcpudp xt_state xt_multiport iptable_filter iptable_nat > nf_n at nf_conntrack_ipv4 sit nf_conntrack nf_defrag_ipv4 tunnel4 > ip_tables x_tables acpiphp ppde v psmouse parport_pc serio_raw fbcon > tileblit font bitblit softcursor vmxnet3 vga16fb vgasta te lp parport > floppy vmw_pvscsi > [212670.327741] Pid: 0, comm: swapper Not tainted 2.6.32-24-server #42-Ubuntu > [212670.327742] Call Trace: > [212670.327744] <IRQ> [<ffffffff81066dab>] warn_slowpath_common+0x7b/0xc0 > [212670.327754] [<ffffffff81066e04>] warn_slowpath_null+0x14/0x20 > [212670.327757] [<ffffffff814b98f9>] tcp_fastretrans_alert+0x3f9/0xd90 > [212670.327759] [<ffffffff814bad24>] tcp_ack+0x284/0x6b0 > [212670.327762] [<ffffffff814bbab0>] tcp_rcv_established+0x400/0x730 > [212670.327766] [<ffffffff814c3553>] tcp_v4_do_rcv+0xf3/0x160 > [212670.327769] [<ffffffff814c4ca5>] tcp_v4_rcv+0x5b5/0x7e0 > [212670.327772] [<ffffffff814a2c00>] ? ip_local_deliver_finish+0x0/0x2d0 > [212670.327776] [<ffffffff8149a6b4>] ? nf_hook_slow+0x74/0x100 > [212670.327779] [<ffffffff814a2c00>] ? ip_local_deliver_finish+0x0/0x2d0 > [212670.327781] [<ffffffff814a2cdd>] ip_local_deliver_finish+0xdd/0x2d0 > [212670.327783] [<ffffffff814a2f60>] ip_local_deliver+0x90/0xa0 > [212670.327785] [<ffffffff814a241d>] ip_rcv_finish+0x12d/0x440 > [212670.327787] [<ffffffff814a29a5>] ip_rcv+0x275/0x360 > [212670.327794] [<ffffffff8147319a>] netif_receive_skb+0x38a/0x5d0 > [212670.327801] [<ffffffffa005203c>] > vmxnet3_rq_rx_complete+0x23c/0x610 [vmxnet3] > [212670.327806] [<ffffffffa005245f>] vmxnet3_do_poll+0x4f/0xf0 [vmxnet3] > [212670.327809] [<ffffffffa005252d>] vmxnet3_poll+0x2d/0x60 [vmxnet3] > [212670.327816] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10 > [212670.327820] [<ffffffff81473c8f>] net_rx_action+0x10f/0x250 > [212670.327824] [<ffffffff8106e2f7>] __do_softirq+0xb7/0x1e0 > [212670.327828] [<ffffffff810c4d40>] ? handle_IRQ_event+0x60/0x170 > [212670.327833] [<ffffffff810142ec>] call_softirq+0x1c/0x30 > [212670.327836] [<ffffffff81015cb5>] do_softirq+0x65/0xa0 > [212670.327837] [<ffffffff8106e195>] irq_exit+0x85/0x90 > [212670.327845] [<ffffffff8155fd45>] do_IRQ+0x75/0xf0 > [212670.327848] [<ffffffff81013b13>] ret_from_intr+0x0/0x11 > [212670.327849] <EOI> [<ffffffff81038acb>] ? native_safe_halt+0xb/0x10 > [212670.327855] [<ffffffff8101b69d>] ? default_idle+0x3d/0x90 > [212670.327860] [<ffffffff81011e63>] ? cpu_idle+0xb3/0x110 > [212670.327867] [<ffffffff81552b53>] ? start_secondary+0xa8/0xaa > [212670.327869] ---[ end trace e26694ca94def524 ]--- > > > Here's netstat -s output > > Ip: > 362267935 total packets received > 0 forwarded > 0 incoming packets discarded > 362257901 incoming packets delivered > 326868094 requests sent out > 9 fragments dropped after timeout > 2777 reassemblies required > 1357 packets reassembled ok > 9 packet reassembles failed > 490 fragments received ok > 1034 fragments created > Icmp: > 22430 ICMP messages received > 283 input ICMP message failed. > ICMP input histogram: > destination unreachable: 17113 > timeout in transit: 613 > redirects: 9 > echo requests: 4685 > echo replies: 10 > 7383 ICMP messages sent > 0 ICMP messages failed > ICMP output histogram: > destination unreachable: 2695 > echo request: 11 > echo replies: 4677 > IcmpMsg: > InType0: 10 > InType3: 17113 > InType5: 9 > InType8: 4685 > InType11: 613 > OutType0: 4677 > OutType3: 2695 > OutType8: 11 > Tcp: > 6563076 active connections openings > 1610183 passive connection openings > 1646225 failed connection attempts > 75246 connection resets received > 1141 connections established > 361192315 segments received > 323797760 segments send out > 1997456 segments retransmited > 12990 bad segments received. > 1851272 resets sent > Udp: > 4138313 packets received > 186 packets to unknown port received. > 0 packet receive errors > 4161268 packets sent > UdpLite: > TcpExt: > 5096 invalid SYN cookies received > 167 resets received for embryonic SYN_RECV sockets > 58826 packets pruned from receive queue because of socket buffer overrun > 312 packets pruned from receive queue > 19 packets dropped from out-of-order queue because of socket buffer overrun > 365 ICMP packets dropped because they were out-of-window > 2712111 TCP sockets finished time wait in fast timer > 82068 time wait sockets recycled by time stamp > 311 packets rejects in established connections because of timestamp > 11516250 delayed acks sent > 6415 delayed acks further delayed because of locked socket > Quick ack mode was activated 208750 times > 41789 packets directly queued to recvmsg prequeue. > 1 bytes directly in process context from backlog > 2969 bytes directly received in process context from prequeue > 155660989 packet headers predicted > 55448128 acknowledgments not containing data payload received > 89389461 predicted acknowledgments > 9544 times recovered from packet loss due to fast retransmit > 585991 times recovered from packet loss by selective acknowledgements > 710 bad SACK blocks received > Detected reordering 1944 times using FACK > Detected reordering 26599 times using SACK > Detected reordering 267 times using reno fast retransmit > Detected reordering 1819 times using time stamp > 3169 congestion windows fully recovered without slow start > 26810 congestion windows partially recovered using Hoe heuristic > 14402 congestion windows recovered without slow start by DSACK > 40392 congestion windows recovered without slow start after partial ack > 115772 TCP data loss events > TCPLostRetransmit: 5320 > 1922 timeouts after reno fast retransmit > 69095 timeouts after SACK recovery > 11101 timeouts in loss state > 1151834 fast retransmits > 74479 forward retransmits > 225380 retransmits in slow start > 275631 other TCP timeouts > 2606 classic Reno fast retransmits failed > 30507 SACK retransmits failed > 4862645 packets collapsed in receive queue due to low socket buffer > 260934 DSACKs sent for old packets > 5109 DSACKs sent for out of order packets > 95650 DSACKs received > 819 DSACKs for out of order packets received > 28142 connections reset due to unexpected data > 22325 connections reset due to early user close > 85423 connections aborted due to timeout > TCPSACKDiscard: 2319 > TCPDSACKIgnoredOld: 52587 > TCPDSACKIgnoredNoUndo: 29859 > TCPSpuriousRTOs: 2605 > TCPSackShifted: 3539688 > TCPSackMerged: 3241543 > TCPSackShiftFallback: 8938458 > IpExt: > InMcastPkts: 2105 > InBcastPkts: 12 > InOctets: 1670966745 > OutOctets: -417434189 > InMcastOctets: 58940 > InBcastOctets: 1792 ^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: tcp_fastretrans_alert since 2.6.32 ? 2010-09-22 6:48 ` tcp_fastretrans_alert since 2.6.32 ? Eric Dumazet @ 2010-09-25 2:34 ` Sunny 0 siblings, 0 replies; 2+ messages in thread From: Sunny @ 2010-09-25 2:34 UTC (permalink / raw) To: Eric Dumazet; +Cc: linux-kernel, netdev, Ilpo Järvinen ping, anyone? On Tue, Sep 21, 2010 at 11:48 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > CC netdev and Ilpo > > Le mardi 21 septembre 2010 à 23:37 -0700, Sunny a écrit : >> Hi, >> >> After upgrading to the latest ubuntu kernel, I started to have this >> alert showing up, this is a vm running on ESXi host with vmxnet3 >> adapter, mainly serving large traffic ~70Mbps towards overboard >> customers, so I expect packet loss and retrans is pretty high. So I'm >> posting to ask whether others been seeing the same issue? When this >> happens, machine freeze several seconds for about three times in a row >> then seems to start working again. >> >> Please cc me on the thread, thanks in advance. >> >> >> [212670.327617] ------------[ cut here ]------------ >> [212670.327689] WARNING: at >> /build/buildd/linux-2.6.32/net/ipv4/tcp_input.c:2919 tcp_fastret >> rans_alert+0x3f9/0xd90() >> [212670.327692] Hardware name: VMware Virtual Platform >> [212670.327693] Modules linked in: btrfs zlib_deflate crc32c libcrc32c >> ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs >> reiserfs sch_sfq cls_u32 sch_htb nf_conntrack_ipv 6 ip6table_filter >> ip6_tables xt_tcpudp xt_state xt_multiport iptable_filter iptable_nat >> nf_n at nf_conntrack_ipv4 sit nf_conntrack nf_defrag_ipv4 tunnel4 >> ip_tables x_tables acpiphp ppde v psmouse parport_pc serio_raw fbcon >> tileblit font bitblit softcursor vmxnet3 vga16fb vgasta te lp parport >> floppy vmw_pvscsi >> [212670.327741] Pid: 0, comm: swapper Not tainted 2.6.32-24-server #42-Ubuntu >> [212670.327742] Call Trace: >> [212670.327744] <IRQ> [<ffffffff81066dab>] warn_slowpath_common+0x7b/0xc0 >> [212670.327754] [<ffffffff81066e04>] warn_slowpath_null+0x14/0x20 >> [212670.327757] [<ffffffff814b98f9>] tcp_fastretrans_alert+0x3f9/0xd90 >> [212670.327759] [<ffffffff814bad24>] tcp_ack+0x284/0x6b0 >> [212670.327762] [<ffffffff814bbab0>] tcp_rcv_established+0x400/0x730 >> [212670.327766] [<ffffffff814c3553>] tcp_v4_do_rcv+0xf3/0x160 >> [212670.327769] [<ffffffff814c4ca5>] tcp_v4_rcv+0x5b5/0x7e0 >> [212670.327772] [<ffffffff814a2c00>] ? ip_local_deliver_finish+0x0/0x2d0 >> [212670.327776] [<ffffffff8149a6b4>] ? nf_hook_slow+0x74/0x100 >> [212670.327779] [<ffffffff814a2c00>] ? ip_local_deliver_finish+0x0/0x2d0 >> [212670.327781] [<ffffffff814a2cdd>] ip_local_deliver_finish+0xdd/0x2d0 >> [212670.327783] [<ffffffff814a2f60>] ip_local_deliver+0x90/0xa0 >> [212670.327785] [<ffffffff814a241d>] ip_rcv_finish+0x12d/0x440 >> [212670.327787] [<ffffffff814a29a5>] ip_rcv+0x275/0x360 >> [212670.327794] [<ffffffff8147319a>] netif_receive_skb+0x38a/0x5d0 >> [212670.327801] [<ffffffffa005203c>] >> vmxnet3_rq_rx_complete+0x23c/0x610 [vmxnet3] >> [212670.327806] [<ffffffffa005245f>] vmxnet3_do_poll+0x4f/0xf0 [vmxnet3] >> [212670.327809] [<ffffffffa005252d>] vmxnet3_poll+0x2d/0x60 [vmxnet3] >> [212670.327816] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10 >> [212670.327820] [<ffffffff81473c8f>] net_rx_action+0x10f/0x250 >> [212670.327824] [<ffffffff8106e2f7>] __do_softirq+0xb7/0x1e0 >> [212670.327828] [<ffffffff810c4d40>] ? handle_IRQ_event+0x60/0x170 >> [212670.327833] [<ffffffff810142ec>] call_softirq+0x1c/0x30 >> [212670.327836] [<ffffffff81015cb5>] do_softirq+0x65/0xa0 >> [212670.327837] [<ffffffff8106e195>] irq_exit+0x85/0x90 >> [212670.327845] [<ffffffff8155fd45>] do_IRQ+0x75/0xf0 >> [212670.327848] [<ffffffff81013b13>] ret_from_intr+0x0/0x11 >> [212670.327849] <EOI> [<ffffffff81038acb>] ? native_safe_halt+0xb/0x10 >> [212670.327855] [<ffffffff8101b69d>] ? default_idle+0x3d/0x90 >> [212670.327860] [<ffffffff81011e63>] ? cpu_idle+0xb3/0x110 >> [212670.327867] [<ffffffff81552b53>] ? start_secondary+0xa8/0xaa >> [212670.327869] ---[ end trace e26694ca94def524 ]--- >> >> >> Here's netstat -s output >> >> Ip: >> 362267935 total packets received >> 0 forwarded >> 0 incoming packets discarded >> 362257901 incoming packets delivered >> 326868094 requests sent out >> 9 fragments dropped after timeout >> 2777 reassemblies required >> 1357 packets reassembled ok >> 9 packet reassembles failed >> 490 fragments received ok >> 1034 fragments created >> Icmp: >> 22430 ICMP messages received >> 283 input ICMP message failed. >> ICMP input histogram: >> destination unreachable: 17113 >> timeout in transit: 613 >> redirects: 9 >> echo requests: 4685 >> echo replies: 10 >> 7383 ICMP messages sent >> 0 ICMP messages failed >> ICMP output histogram: >> destination unreachable: 2695 >> echo request: 11 >> echo replies: 4677 >> IcmpMsg: >> InType0: 10 >> InType3: 17113 >> InType5: 9 >> InType8: 4685 >> InType11: 613 >> OutType0: 4677 >> OutType3: 2695 >> OutType8: 11 >> Tcp: >> 6563076 active connections openings >> 1610183 passive connection openings >> 1646225 failed connection attempts >> 75246 connection resets received >> 1141 connections established >> 361192315 segments received >> 323797760 segments send out >> 1997456 segments retransmited >> 12990 bad segments received. >> 1851272 resets sent >> Udp: >> 4138313 packets received >> 186 packets to unknown port received. >> 0 packet receive errors >> 4161268 packets sent >> UdpLite: >> TcpExt: >> 5096 invalid SYN cookies received >> 167 resets received for embryonic SYN_RECV sockets >> 58826 packets pruned from receive queue because of socket buffer overrun >> 312 packets pruned from receive queue >> 19 packets dropped from out-of-order queue because of socket buffer overrun >> 365 ICMP packets dropped because they were out-of-window >> 2712111 TCP sockets finished time wait in fast timer >> 82068 time wait sockets recycled by time stamp >> 311 packets rejects in established connections because of timestamp >> 11516250 delayed acks sent >> 6415 delayed acks further delayed because of locked socket >> Quick ack mode was activated 208750 times >> 41789 packets directly queued to recvmsg prequeue. >> 1 bytes directly in process context from backlog >> 2969 bytes directly received in process context from prequeue >> 155660989 packet headers predicted >> 55448128 acknowledgments not containing data payload received >> 89389461 predicted acknowledgments >> 9544 times recovered from packet loss due to fast retransmit >> 585991 times recovered from packet loss by selective acknowledgements >> 710 bad SACK blocks received >> Detected reordering 1944 times using FACK >> Detected reordering 26599 times using SACK >> Detected reordering 267 times using reno fast retransmit >> Detected reordering 1819 times using time stamp >> 3169 congestion windows fully recovered without slow start >> 26810 congestion windows partially recovered using Hoe heuristic >> 14402 congestion windows recovered without slow start by DSACK >> 40392 congestion windows recovered without slow start after partial ack >> 115772 TCP data loss events >> TCPLostRetransmit: 5320 >> 1922 timeouts after reno fast retransmit >> 69095 timeouts after SACK recovery >> 11101 timeouts in loss state >> 1151834 fast retransmits >> 74479 forward retransmits >> 225380 retransmits in slow start >> 275631 other TCP timeouts >> 2606 classic Reno fast retransmits failed >> 30507 SACK retransmits failed >> 4862645 packets collapsed in receive queue due to low socket buffer >> 260934 DSACKs sent for old packets >> 5109 DSACKs sent for out of order packets >> 95650 DSACKs received >> 819 DSACKs for out of order packets received >> 28142 connections reset due to unexpected data >> 22325 connections reset due to early user close >> 85423 connections aborted due to timeout >> TCPSACKDiscard: 2319 >> TCPDSACKIgnoredOld: 52587 >> TCPDSACKIgnoredNoUndo: 29859 >> TCPSpuriousRTOs: 2605 >> TCPSackShifted: 3539688 >> TCPSackMerged: 3241543 >> TCPSackShiftFallback: 8938458 >> IpExt: >> InMcastPkts: 2105 >> InBcastPkts: 12 >> InOctets: 1670966745 >> OutOctets: -417434189 >> InMcastOctets: 58940 >> InBcastOctets: 1792 > > > ^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2010-09-25 2:34 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <AANLkTinnaA3Qgny45BRyW3i5u+nXn1XgWwkxA_QbN8xN@mail.gmail.com>
2010-09-22 6:48 ` tcp_fastretrans_alert since 2.6.32 ? Eric Dumazet
2010-09-25 2:34 ` Sunny
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).