* WARNING: at net/ipv4/tcp_input.c @ 2008-04-28 12:02 Andy Furniss 2008-04-28 15:57 ` Ilpo Järvinen 0 siblings, 1 reply; 15+ messages in thread From: Andy Furniss @ 2008-04-28 12:02 UTC (permalink / raw) To: netdev Since upgrading to 2.6.25 from 2.6.21.1 I've noticed a few of these. timestamps are off everything else tcp is default. This is my gateway/torrent/TV server there is nearly always torrent traffic. I've always had issues with the tulip nic (4 port zynx that doesn't like my desktop being off - it's connected with a crossover cable. The ports to a switch and modem are always on and OK) I haven't seen these warnings on 2.6.21.1. ------------[ cut here ]------------ WARNING: at net/ipv4/tcp_input.c:2539 tcp_ack+0x1dc2/0x1df0() Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip Pid: 0, comm: swapper Not tainted 2.6.25 #1 [<c011b23f>] warn_on_slowpath+0x5f/0x90 [<c03191d6>] dev_queue_xmit+0x96/0x280 [<c033a910>] ip_finish_output+0x0/0x2b0 [<c033aa4c>] ip_finish_output+0x13c/0x2b0 [<d4884b58>] __nf_ct_refresh_acct+0xe8/0x160 [nf_conntrack] [<d488814a>] tcp_packet+0x48a/0xb70 [nf_conntrack] [<c036a52d>] bictcp_cong_avoid+0x1d/0x1f0 [<c0347fb2>] tcp_ack+0x1dc2/0x1df0 [<c0349cbe>] tcp_rcv_established+0x3de/0x730 [<c0350af3>] tcp_v4_do_rcv+0xb3/0x1d0 [<c032fda0>] nf_iterate+0x60/0x90 [<c0335cd0>] ip_local_deliver_finish+0x0/0x1a0 [<c0351239>] tcp_v4_rcv+0x629/0x6f0 [<c0335d64>] ip_local_deliver_finish+0x94/0x1a0 [<c0335cd0>] ip_local_deliver_finish+0x0/0x1a0 [<c0335853>] ip_rcv_finish+0x103/0x320 [<c0335750>] ip_rcv_finish+0x0/0x320 [<c0335c0a>] ip_rcv+0x19a/0x260 [<c0335750>] ip_rcv_finish+0x0/0x320 [<c0319660>] netif_receive_skb+0x2a0/0x450 [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] [<c0134c96>] tick_program_event+0x36/0x70 [<c0316483>] net_rx_action+0x93/0x150 [<c011f992>] __do_softirq+0x42/0xa0 [<c011fa16>] do_softirq+0x26/0x30 [<c010645a>] do_IRQ+0x4a/0x80 [<c01344eb>] tick_notify+0x1ab/0x270 [<c010474f>] common_interrupt+0x23/0x28 [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] [<c02d8fbb>] cpuidle_idle_call+0x6b/0xa0 [<c02d8f50>] cpuidle_idle_call+0x0/0xa0 [<c0102c66>] cpu_idle+0x26/0x70 ======================= ---[ end trace bbc6d0b1edc8c032 ]--- ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-04-28 12:02 WARNING: at net/ipv4/tcp_input.c Andy Furniss @ 2008-04-28 15:57 ` Ilpo Järvinen 2008-04-28 19:07 ` Andy Furniss 0 siblings, 1 reply; 15+ messages in thread From: Ilpo Järvinen @ 2008-04-28 15:57 UTC (permalink / raw) To: Andy Furniss; +Cc: Netdev On Mon, 28 Apr 2008, Andy Furniss wrote: > Since upgrading to 2.6.25 from 2.6.21.1 I've noticed a few of these. ...Thanks for the report, you could actually reproduce it in non-infinite time?? ...Finally somebody who might actually have a change to catch this shows up... :-) > timestamps are off everything else tcp is default. > > This is my gateway/torrent/TV server there is nearly always torrent traffic. I've tried to catch these with torrent but never got anything, it seems to be quite sensitive to "network weather", which varies too much for me to catch it. Maybe I'll try w/o timestamps if that helps me to reproduce it (more likely). > I've always had issues with the tulip nic (4 port zynx that doesn't like my > desktop being off - it's connected with a crossover cable. The ports to a > switch and modem are always on and OK) I haven't seen these warnings on > 2.6.21.1. > > ------------[ cut here ]------------ > WARNING: at net/ipv4/tcp_input.c:2539 tcp_ack+0x1dc2/0x1df0() We weren't checking this invariant in 2.6.21.1, actually, I made the invariant more rigid and added tracking for it to see what is breaking things. We still reset the state like the older did but I'd also like to find out the cause. The debug patch below will add considerable amount of processing per ACK to verify where this invariant gets broken for the first time, in case that's fine with your server, please consider adding that and waiting until it spits something out... :-) That WARNING itself is usually too late but that's inexpensive enough to be always on to work as early warning system (and TCP anyway corrects the inaccurate state if it triggersbecause it's trivial to do there, in general these warning only pose a theoretical threat to network as a form of packet bursts or cause minor, nearly impossible to notice delays if one needs to recover by RTO instead of more timely means). -- i. --- [PATCH] TCP: debug S+L (for 2.6.25/.26-rcs), v1.4.2 Debugs sacked & lost skb inconstencies in TCP write queue & verifies fackets_out related variables. --- include/net/tcp.h | 10 +++- net/ipv4/tcp_input.c | 34 ++++++++++++- net/ipv4/tcp_ipv4.c | 134 +++++++++++++++++++++++++++++++++++++++++++++++++ net/ipv4/tcp_output.c | 20 ++++++-- 4 files changed, 191 insertions(+), 7 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 4fd3eb2..b7b73fb 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -272,6 +272,9 @@ DECLARE_SNMP_STAT(struct tcp_mib, tcp_statistics); #define TCP_ADD_STATS_BH(field, val) SNMP_ADD_STATS_BH(tcp_statistics, field, val) #define TCP_ADD_STATS_USER(field, val) SNMP_ADD_STATS_USER(tcp_statistics, field, val) +extern void tcp_print_queue(struct sock *sk); +extern int tcp_verify_wq(struct sock *sk); + extern void tcp_v4_err(struct sk_buff *skb, u32); extern void tcp_shutdown (struct sock *sk, int how); @@ -770,7 +773,12 @@ static inline __u32 tcp_current_ssthresh(const struct sock *sk) } /* Use define here intentionally to get WARN_ON location shown at the caller */ -#define tcp_verify_left_out(tp) WARN_ON(tcp_left_out(tp) > tp->packets_out) +#define tcp_verify_left_out(tp) \ + do {\ + int res; \ + res = tcp_verify_wq((struct sock *)tp); \ + WARN_ON(res || tcp_left_out(tp) > tp->packets_out); \ + } while(0) extern void tcp_enter_cwr(struct sock *sk, const int set_ssthresh); extern __u32 tcp_init_cwnd(struct tcp_sock *tp, struct dst_entry *dst); diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index bbb7d88..7dd7939 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -1428,8 +1428,10 @@ tcp_sacktag_write_queue(struct sock *sk, struct sk_buff *ack_skb, int first_sack_index; if (!tp->sacked_out) { - if (WARN_ON(tp->fackets_out)) + if (WARN_ON(tp->fackets_out)) { + tcp_verify_left_out(tp); tp->fackets_out = 0; + } tcp_highest_sack_reset(sk); } @@ -2157,7 +2159,14 @@ static void tcp_mark_head_lost(struct sock *sk, int packets) int err; unsigned int mss; + tcp_verify_left_out(tp); + BUG_TRAP(packets <= tp->packets_out); + if (packets > tp->packets_out) { + pr_err("TCP debug %u/%u-%u/%u %d\n", + tp->fackets_out, tp->sacked_out, tp->reordering, + tp->packets_out, tp->rx_opt.sack_ok); + } if (tp->lost_skb_hint) { skb = tp->lost_skb_hint; cnt = tp->lost_cnt_hint; @@ -2534,10 +2543,15 @@ static void tcp_fastretrans_alert(struct sock *sk, int pkts_acked, int flag) (tcp_fackets_out(tp) > tp->reordering)); int fast_rexmit = 0; + tcp_verify_left_out(tp); + if (WARN_ON(!tp->packets_out && tp->sacked_out)) tp->sacked_out = 0; - if (WARN_ON(!tp->sacked_out && tp->fackets_out)) + if (WARN_ON(!tp->sacked_out && tp->fackets_out)) { + printk(KERN_ERR "TCP DEBUG %u %u %d %d\n", tp->fackets_out, + tp->packets_out, pkts_acked, tp->rx_opt.sack_ok); tp->fackets_out = 0; + } /* Now state machine starts. * A. ECE, hence prohibit cwnd undoing, the reduction is required. */ @@ -2680,6 +2694,10 @@ static void tcp_fastretrans_alert(struct sock *sk, int pkts_acked, int flag) if (do_lost || (tcp_is_fack(tp) && tcp_head_timedout(sk))) tcp_update_scoreboard(sk, fast_rexmit); tcp_cwnd_down(sk, flag); + + WARN_ON(tcp_write_queue_head(sk) == NULL); + WARN_ON(!tp->packets_out); + tcp_xmit_retransmit_queue(sk); } @@ -2832,6 +2850,9 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets) tcp_mtup_probe_success(sk, skb); } + WARN_ON((sacked & (TCPCB_SACKED_ACKED|TCPCB_LOST)) == + (TCPCB_SACKED_ACKED|TCPCB_LOST)); + if (sacked & TCPCB_RETRANS) { if (sacked & TCPCB_SACKED_RETRANS) tp->retrans_out -= acked_pcount; @@ -2922,6 +2943,8 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets) } } + tcp_verify_left_out(tp); + #if FASTRETRANS_DEBUG > 0 BUG_TRAP((int)tp->sacked_out >= 0); BUG_TRAP((int)tp->lost_out >= 0); @@ -3210,6 +3233,8 @@ static int tcp_ack(struct sock *sk, struct sk_buff *skb, int flag) prior_fackets = tp->fackets_out; prior_in_flight = tcp_packets_in_flight(tp); + tcp_verify_left_out(tp); + if (!(flag & FLAG_SLOWPATH) && after(ack, prior_snd_una)) { /* Window is constant, pure forward advance. * No more checks are required. @@ -3269,9 +3294,14 @@ static int tcp_ack(struct sock *sk, struct sk_buff *skb, int flag) tcp_cong_avoid(sk, ack, prior_in_flight); } + if (WARN_ON((icsk->icsk_ca_state == TCP_CA_Open) && tcp_left_out(tp))) + tcp_print_queue(sk); + if ((flag & FLAG_FORWARD_PROGRESS) || !(flag & FLAG_NOT_DUP)) dst_confirm(sk->sk_dst_cache); + tcp_verify_left_out(tp); + return 1; no_queue: diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 00156bf..52447b0 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -108,6 +108,140 @@ struct inet_hashinfo __cacheline_aligned tcp_hashinfo = { .lhash_wait = __WAIT_QUEUE_HEAD_INITIALIZER(tcp_hashinfo.lhash_wait), }; +void tcp_print_queue(struct sock *sk) +{ + struct tcp_sock *tp = tcp_sk(sk); + struct sk_buff *skb; + char s[50+1]; + char h[50+1]; + int idx = 0; + int i; + + i = 0; + tcp_for_write_queue(skb, sk) { + if (skb == tcp_send_head(sk)) + printk(KERN_ERR "head %u %p\n", i, skb); + else + printk(KERN_ERR "skb %u %p\n", i, skb); + i++; + } + + tcp_for_write_queue(skb, sk) { + if (skb == tcp_send_head(sk)) + break; + + for (i = 0; i < tcp_skb_pcount(skb); i++) { + if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) { + s[idx] = 'S'; + if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) + s[idx] = 'B'; + + } else if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) { + s[idx] = 'L'; + } else { + s[idx] = ' '; + } + if (s[idx] != ' ' && skb->len < tp->mss_cache) + s[idx] += 'a' - 'A'; + + if (i == 0) { + if (skb == tcp_highest_sack(sk)) + h[idx] = 'h'; + else + h[idx] = '+'; + } else { + h[idx] = '-'; + } + + if (++idx >= 50) { + s[idx] = 0; + h[idx] = 0; + printk(KERN_ERR "TCP wq(s) %s\n", s); + printk(KERN_ERR "TCP wq(h) %s\n", h); + idx = 0; + } + } + } + if (idx) { + s[idx] = '<'; + s[idx+1] = 0; + h[idx] = '<'; + h[idx+1] = 0; + printk(KERN_ERR "TCP wq(s) %s\n", s); + printk(KERN_ERR "TCP wq(h) %s\n", h); + } + printk(KERN_ERR "l%u s%u f%u p%u seq: su%u hs%u sn%u\n", + tp->lost_out, tp->sacked_out, tp->fackets_out, tp->packets_out, + tp->snd_una, tcp_highest_sack_seq(tp), tp->snd_nxt); +} + +int tcp_verify_wq(struct sock *sk) +{ + struct tcp_sock *tp = tcp_sk(sk); + u32 lost = 0; + u32 sacked = 0; + u32 packets = 0; + u32 fackets = 0; + int hs_valid = 0; + int inconsistent = 0; + struct sk_buff *skb; + + tcp_for_write_queue(skb, sk) { + if (skb == tcp_send_head(sk)) + break; + + if ((fackets == packets) && (skb == tp->highest_sack)) + hs_valid = 1; + + packets += tcp_skb_pcount(skb); + + if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) { + sacked += tcp_skb_pcount(skb); + if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) + printk(KERN_ERR "Sacked bitmap S+L: %u %u-%u/%u\n", + TCP_SKB_CB(skb)->sacked, + TCP_SKB_CB(skb)->end_seq - tp->snd_una, + TCP_SKB_CB(skb)->seq - tp->snd_una, + tp->snd_una); + fackets = packets; + hs_valid = 0; + } + if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) + lost += tcp_skb_pcount(skb); + } + + if ((fackets == packets) && (tp->highest_sack == tcp_send_head(sk))) + hs_valid = 1; + + if ((lost != tp->lost_out) || + (tcp_is_sack(tp) && (sacked != tp->sacked_out)) || + ((sacked || (tcp_is_sack(tp) && tp->sacked_out)) && !hs_valid) || + (packets != tp->packets_out) || + (fackets != tp->fackets_out) || + (tcp_is_reno(tp) && tp->packets_out && + (tcp_left_out(tp) == tp->packets_out) && + !(TCP_SKB_CB(tcp_write_queue_head(sk))->sacked & TCPCB_LOST)) || + tcp_left_out(tp) > tp->packets_out) { + printk(KERN_ERR "P: %u L: %u vs %u S: %u vs %u F: %u vs %u w: %u-%u (%u)\n", + tp->packets_out, + lost, tp->lost_out, + sacked, tp->sacked_out, + fackets, tp->fackets_out, + tp->snd_una, tp->snd_nxt, + tp->rx_opt.sack_ok); + tcp_print_queue(sk); + inconsistent = 1; + } + + WARN_ON(lost != tp->lost_out); + WARN_ON(tcp_is_sack(tp) && (sacked != tp->sacked_out)); + WARN_ON(packets != tp->packets_out); + WARN_ON(fackets != tp->fackets_out); + WARN_ON(tcp_left_out(tp) > tp->packets_out); + + return inconsistent; +} + static inline __u32 tcp_v4_init_sequence(struct sk_buff *skb) { return secure_tcp_sequence_number(ip_hdr(skb)->daddr, diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index d29ef79..84113be 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -779,10 +779,9 @@ int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len, tp->lost_out -= diff; /* Adjust Reno SACK estimate. */ - if (tcp_is_reno(tp) && diff > 0) { + if (tcp_is_reno(tp) && diff > 0) tcp_dec_pcount_approx_int(&tp->sacked_out, diff); - tcp_verify_left_out(tp); - } + tcp_adjust_fackets_out(sk, skb, diff); } @@ -790,6 +789,8 @@ int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len, skb_header_release(buff); tcp_insert_write_queue_after(skb, buff, sk); + tcp_verify_left_out(tp); + return 0; } @@ -1471,6 +1472,7 @@ static int tcp_write_xmit(struct sock *sk, unsigned int mss_now, int nonagle) } else if (result > 0) { sent_pkts = 1; } + tcp_verify_left_out(tp); while ((skb = tcp_send_head(sk))) { unsigned int limit; @@ -1772,6 +1774,7 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *skb, tcp_clear_retrans_hints_partial(tp); sk_wmem_free_skb(sk, next_skb); + tcp_verify_left_out(tp); } /* Do a simple retransmit without using the backoff mechanisms in @@ -1895,10 +1898,13 @@ int tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb) (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_FIN) && tp->snd_una == (TCP_SKB_CB(skb)->end_seq - 1)) { if (!pskb_trim(skb, 0)) { + WARN_ON(TCP_SKB_CB(skb)->sacked & TCPCB_LOST); /* Reuse, even though it does some unnecessary work */ tcp_init_nondata_skb(skb, TCP_SKB_CB(skb)->end_seq - 1, TCP_SKB_CB(skb)->flags); skb->ip_summed = CHECKSUM_NONE; + + tcp_verify_left_out(tp); } } @@ -1981,8 +1987,10 @@ void tcp_xmit_retransmit_queue(struct sock *sk) * packet to be MSS sized and all the * packet counting works out. */ - if (tcp_packets_in_flight(tp) >= tp->snd_cwnd) + if (tcp_packets_in_flight(tp) >= tp->snd_cwnd) { + tcp_verify_left_out(tp); return; + } if (sacked & TCPCB_LOST) { if (!(sacked & (TCPCB_SACKED_ACKED|TCPCB_SACKED_RETRANS))) { @@ -2008,6 +2016,8 @@ void tcp_xmit_retransmit_queue(struct sock *sk) } } + tcp_verify_left_out(tp); + /* OK, demanded retransmission is finished. */ /* Forward retransmissions are possible only during Recovery. */ @@ -2065,6 +2075,8 @@ void tcp_xmit_retransmit_queue(struct sock *sk) NET_INC_STATS_BH(LINUX_MIB_TCPFORWARDRETRANS); } + + tcp_verify_left_out(tp); } /* Send a fin. The caller locks the socket for us. This cannot be -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-04-28 15:57 ` Ilpo Järvinen @ 2008-04-28 19:07 ` Andy Furniss 2008-04-28 19:21 ` Ilpo Järvinen 0 siblings, 1 reply; 15+ messages in thread From: Andy Furniss @ 2008-04-28 19:07 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev Ilpo Järvinen wrote: > On Mon, 28 Apr 2008, Andy Furniss wrote: > >> Since upgrading to 2.6.25 from 2.6.21.1 I've noticed a few of these. > > ...Thanks for the report, you could actually reproduce it in non-infinite > time?? ...Finally somebody who might actually have a change to catch > this shows up... :-) Hopefully - grepping shows I've had a few, I was running -rc7 for a while and see a different message from that. I changed to .25 on the 21st. andy@noki:~$ grep WARNING /var/log/kern.log Mar 29 20:35:03 noki kernel: WARNING: at net/ipv4/tcp_input.c:2510 tcp_fastretrans_alert+0x685/0x6b0() Mar 30 03:02:04 noki kernel: WARNING: at net/ipv4/tcp_input.c:2510 tcp_fastretrans_alert+0x685/0x6b0() Apr 15 05:11:02 noki kernel: WARNING: at net/ipv4/tcp_input.c:2510 tcp_fastretrans_alert+0x685/0x6b0() Apr 19 17:11:36 noki kernel: WARNING: at net/ipv4/tcp_input.c:2510 tcp_fastretrans_alert+0x685/0x6b0() Apr 26 19:42:33 noki kernel: WARNING: at net/ipv4/tcp_input.c:2539 tcp_ack+0x1dc2/0x1df0() Apr 28 08:35:58 noki kernel: WARNING: at net/ipv4/tcp_input.c:2539 tcp_ack+0x1dc2/0x1df0() Apr 28 17:38:00 noki kernel: WARNING: at net/ipv4/tcp_input.c:2539 tcp_ack+0x1dc2/0x1df0() > I've tried to catch these with torrent but never got anything, it seems to > be quite sensitive to "network weather", which varies too much for me to > catch it. Maybe I'll try w/o timestamps if that helps me to reproduce it > (more likely). Could be my qos shaking things up. bt uses piggy backed acks and I must drop a fair few. I probably reorder a bit as well by treating small tcp as higher prio than large. Egress tcp is also mss clamped to 1150. > The debug patch below will add considerable amount of processing per ACK > to verify where this invariant gets broken for the first time, in case > that's fine with your server, please consider adding that and waiting > until it spits something out... :-) No problem it's 90% idle most of the time and my wan isn't that fast. I'll rebuild tonight. Andy. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-04-28 19:07 ` Andy Furniss @ 2008-04-28 19:21 ` Ilpo Järvinen 2008-05-01 9:01 ` Andy Furniss 0 siblings, 1 reply; 15+ messages in thread From: Ilpo Järvinen @ 2008-04-28 19:21 UTC (permalink / raw) To: Andy Furniss; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 1812 bytes --] On Mon, 28 Apr 2008, Andy Furniss wrote: > Ilpo Järvinen wrote: > > On Mon, 28 Apr 2008, Andy Furniss wrote: > > > > > Since upgrading to 2.6.25 from 2.6.21.1 I've noticed a few of these. > > > > ...Thanks for the report, you could actually reproduce it in non-infinite > > time?? ...Finally somebody who might actually have a change to catch this > > shows up... :-) > > Hopefully - grepping shows I've had a few, More than one is a good start already. Most people who have reported had just one or have a loaded server which isn't that suitable for expensive tracking that is necessary. > I was running -rc7 for a while and > see a different message from that. I changed to .25 on the 21st. Anything pre -rc9 is not worth to mention. There have been other issues that were fixed in it. > > I've tried to catch these with torrent but never got anything, it seems to > > be quite sensitive to "network weather", which varies too much for me to > > catch it. Maybe I'll try w/o timestamps if that helps me to reproduce it > > (more likely). > > Could be my qos shaking things up. bt uses piggy backed acks and I must drop a > fair few. I probably reorder a bit as well by treating small tcp as higher > prio than large. Egress tcp is also mss clamped to 1150. Thanks for the info, though I've tried to use netem to generate some reordering & drops in my setup but just couldn't make it to happen with them either. > > The debug patch below will add considerable amount of processing per ACK to > > verify where this invariant gets broken for the first time, in case that's > > fine with your server, please consider adding that and waiting until it > > spits something out... :-) > > No problem it's 90% idle most of the time and my wan isn't that fast. > > I'll rebuild tonight. Thanks. -- i. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-04-28 19:21 ` Ilpo Järvinen @ 2008-05-01 9:01 ` Andy Furniss 2008-05-01 9:37 ` Ilpo Järvinen 0 siblings, 1 reply; 15+ messages in thread From: Andy Furniss @ 2008-05-01 9:01 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev Caught one - May 1 03:35:45 noki kernel: 4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] tcp_transmit_skb+0x47d/0x730 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_xmit_retransmit_queue+346/976] tcp_xmit_retransmit_queue+0x15a/0x3d0 May 1 03:35:45 noki kernel: [tcp_ack+4069/7232] tcp_ack+0xfe5/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+990/1840] tcp_rcv_established+0x3de/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: P: 4 L: 1 vs 1 S: 3 vs 3 F: 4 vs 6 w: 2200787703-2200792303 (7) May 1 03:35:45 noki kernel: skb 0 cd7c8900 May 1 03:35:45 noki kernel: skb 1 d1b90760 May 1 03:35:45 noki kernel: skb 2 d1b69e60 May 1 03:35:45 noki kernel: skb 3 d1997080 May 1 03:35:45 noki kernel: head 4 d2dc91c0 May 1 03:35:45 noki kernel: skb 5 d2fb07a0 May 1 03:35:45 noki kernel: skb 6 d3b8cb60 May 1 03:35:45 noki kernel: skb 7 d1baca60 May 1 03:35:45 noki kernel: skb 8 d19ae380 May 1 03:35:45 noki kernel: skb 9 d1997ce0 May 1 03:35:45 noki kernel: skb 10 d2e084a0 May 1 03:35:45 noki kernel: skb 11 d19978c0 May 1 03:35:45 noki kernel: TCP wq(s) LSSS< May 1 03:35:45 noki kernel: TCP wq(h) ++++< May 1 03:35:45 noki kernel: l1 s3 f6 p4 seq: su2200787703 hs2200792303 sn2200792303 May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_ipv4.c:239 tcp_verify_wq+0x35f/0x3b0() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [printk+27/32] printk+0x1b/0x20 May 1 03:35:45 noki kernel: [tcp_print_queue+535/592] tcp_print_queue+0x217/0x250 May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] tcp_transmit_skb+0x47d/0x730 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_xmit_retransmit_queue+467/976] tcp_xmit_retransmit_queue+0x1d3/0x3d0 May 1 03:35:45 noki kernel: [tcp_ack+4069/7232] tcp_ack+0xfe5/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+990/1840] tcp_rcv_established+0x3de/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_output.c:2079 tcp_ack+0xfe5/0x1c40() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] tcp_transmit_skb+0x47d/0x730 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_xmit_retransmit_queue+467/976] tcp_xmit_retransmit_queue+0x1d3/0x3d0 May 1 03:35:45 noki kernel: [tcp_ack+4069/7232] tcp_ack+0xfe5/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+990/1840] tcp_rcv_established+0x3de/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: P: 4 L: 1 vs 1 S: 3 vs 3 F: 4 vs 6 w: 2200787703-2200792303 (7) May 1 03:35:45 noki kernel: skb 0 cd7c8900 May 1 03:35:45 noki kernel: skb 1 d1b90760 May 1 03:35:45 noki kernel: skb 2 d1b69e60 May 1 03:35:45 noki kernel: skb 3 d1997080 May 1 03:35:45 noki kernel: head 4 d2dc91c0 May 1 03:35:45 noki kernel: skb 5 d2fb07a0 May 1 03:35:45 noki kernel: skb 6 d3b8cb60 May 1 03:35:45 noki kernel: skb 7 d1baca60 May 1 03:35:45 noki kernel: skb 8 d19ae380 May 1 03:35:45 noki kernel: skb 9 d1997ce0 May 1 03:35:45 noki kernel: skb 10 d2e084a0 May 1 03:35:45 noki kernel: skb 11 d19978c0 May 1 03:35:45 noki kernel: TCP wq(s) LSSS< May 1 03:35:45 noki kernel: TCP wq(h) ++++< May 1 03:35:45 noki kernel: l1 s3 f6 p4 seq: su2200787703 hs2200792303 sn2200792303 May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_ipv4.c:239 tcp_verify_wq+0x35f/0x3b0() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [printk+27/32] printk+0x1b/0x20 May 1 03:35:45 noki kernel: [tcp_print_queue+535/592] tcp_print_queue+0x217/0x250 May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] tcp_transmit_skb+0x47d/0x730 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+3266/7232] tcp_ack+0xcc2/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+990/1840] tcp_rcv_established+0x3de/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_input.c:3303 tcp_ack+0xcef/0x1c40() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] tcp_transmit_skb+0x47d/0x730 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+3311/7232] tcp_ack+0xcef/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+990/1840] tcp_rcv_established+0x3de/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: P: 4 L: 1 vs 1 S: 3 vs 3 F: 4 vs 6 w: 2200787703-2200792303 (7) May 1 03:35:45 noki kernel: skb 0 cd7c8900 May 1 03:35:45 noki kernel: skb 1 d1b90760 May 1 03:35:45 noki kernel: skb 2 d1b69e60 May 1 03:35:45 noki kernel: skb 3 d1997080 May 1 03:35:45 noki kernel: head 4 d2dc91c0 May 1 03:35:45 noki kernel: skb 5 d2fb07a0 May 1 03:35:45 noki kernel: skb 6 d3b8cb60 May 1 03:35:45 noki kernel: skb 7 d1baca60 May 1 03:35:45 noki kernel: skb 8 d19ae380 May 1 03:35:45 noki kernel: skb 9 d1997ce0 May 1 03:35:45 noki kernel: skb 10 d2e084a0 May 1 03:35:45 noki kernel: skb 11 d19978c0 May 1 03:35:45 noki kernel: TCP wq(s) LSSS< May 1 03:35:45 noki kernel: TCP wq(h) ++++< May 1 03:35:45 noki kernel: l1 s3 f6 p4 seq: su2200787703 hs2200792303 sn2200792303 May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_ipv4.c:239 tcp_verify_wq+0x35f/0x3b0() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [printk+27/32] printk+0x1b/0x20 May 1 03:35:45 noki kernel: [tcp_print_queue+535/592] tcp_print_queue+0x217/0x250 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [__tcp_push_pending_frames+80/2048] __tcp_push_pending_frames+0x50/0x800 May 1 03:35:45 noki kernel: [tcp_rcv_established+646/1840] tcp_rcv_established+0x286/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_output.c:1475 __tcp_push_pending_frames+0x67/0x800() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [__tcp_push_pending_frames+103/2048] __tcp_push_pending_frames+0x67/0x800 May 1 03:35:45 noki kernel: [tcp_rcv_established+646/1840] tcp_rcv_established+0x286/0x730 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: 4> [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+179/7232] tcp_ack+0xb3/0x1c40 May 1 03:35:45 noki kernel: [<d48876c5>] tcp_error+0x95/0x1d0 [nf_conntrack] May 1 03:35:45 noki kernel: [tcp_init_tso_segs+50/80] tcp_init_tso_segs+0x32/0x50 May 1 03:35:45 noki kernel: [ip_route_input+67/3344] ip_route_input+0x43/0xd10 May 1 03:35:45 noki kernel: [<d481d566>] ipt_do_table+0x226/0x4a0 [ip_tables] May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_input.c:3236 tcp_ack+0xca/0x1c40() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [dev_queue_xmit+150/640] dev_queue_xmit+0x96/0x280 May 1 03:35:45 noki kernel: [ip_finish_output+0/688] ip_finish_output+0x0/0x2b0 May 1 03:35:45 noki kernel: [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+202/7232] tcp_ack+0xca/0x1c40 May 1 03:35:45 noki kernel: [<d48876c5>] tcp_error+0x95/0x1d0 [nf_conntrack] May 1 03:35:45 noki kernel: [tcp_init_tso_segs+50/80] tcp_init_tso_segs+0x32/0x50 May 1 03:35:45 noki kernel: [ip_route_input+67/3344] ip_route_input+0x43/0xd10 May 1 03:35:45 noki kernel: [<d481d566>] ipt_do_table+0x226/0x4a0 [ip_tables] May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: P: 0 L: 0 vs 0 S: 0 vs 0 F: 0 vs 2 w: 2200792303-2200792303 (7) May 1 03:35:45 noki kernel: head 0 d2dc91c0 May 1 03:35:45 noki kernel: skb 1 d2fb07a0 May 1 03:35:45 noki kernel: skb 2 d3b8cb60 May 1 03:35:45 noki kernel: skb 3 d1baca60 May 1 03:35:45 noki kernel: skb 4 d19ae380 May 1 03:35:45 noki kernel: skb 5 d1997ce0 May 1 03:35:45 noki kernel: skb 6 d2e084a0 May 1 03:35:45 noki kernel: skb 7 d19978c0 May 1 03:35:45 noki kernel: l0 s0 f2 p0 seq: su2200792303 hs2200792303 sn2200792303 May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_ipv4.c:239 tcp_verify_wq+0x35f/0x3b0() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [printk+27/32] printk+0x1b/0x20 May 1 03:35:45 noki kernel: [tcp_print_queue+535/592] tcp_print_queue+0x217/0x250 May 1 03:35:45 noki kernel: [dev_queue_xmit+150/640] dev_queue_xmit+0x96/0x280 May 1 03:35:45 noki kernel: [ip_finish_output+0/688] ip_finish_output+0x0/0x2b0 May 1 03:35:45 noki kernel: [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [bictcp_acked+0/48] bictcp_acked+0x0/0x30 May 1 03:35:45 noki kernel: [tcp_ack+1987/7232] tcp_ack+0x7c3/0x1c40 May 1 03:35:45 noki kernel: [<d481d566>] ipt_do_table+0x226/0x4a0 [ip_tables] May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_input.c:2946 tcp_ack+0x7ec/0x1c40() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [dev_queue_xmit+150/640] dev_queue_xmit+0x96/0x280 May 1 03:35:45 noki kernel: [ip_finish_output+0/688] ip_finish_output+0x0/0x2b0 May 1 03:35:45 noki kernel: [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [bictcp_acked+0/48] bictcp_acked+0x0/0x30 May 1 03:35:45 noki kernel: [tcp_ack+2028/7232] tcp_ack+0x7ec/0x1c40 May 1 03:35:45 noki kernel: [<d481d566>] ipt_do_table+0x226/0x4a0 [ip_tables] May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: P: 0 L: 0 vs 0 S: 0 vs 0 F: 0 vs 2 w: 2200792303-2200792303 (7) May 1 03:35:45 noki kernel: head 0 d2dc91c0 May 1 03:35:45 noki kernel: skb 1 d2fb07a0 May 1 03:35:45 noki kernel: skb 2 d3b8cb60 May 1 03:35:45 noki kernel: skb 3 d1baca60 May 1 03:35:45 noki kernel: skb 4 d19ae380 May 1 03:35:45 noki kernel: skb 5 d1997ce0 May 1 03:35:45 noki kernel: skb 6 d2e084a0 May 1 03:35:45 noki kernel: skb 7 d19978c0 May 1 03:35:45 noki kernel: l0 s0 f2 p0 seq: su2200792303 hs2200792303 sn2200792303 May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_ipv4.c:239 tcp_verify_wq+0x35f/0x3b0() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [printk+27/32] printk+0x1b/0x20 May 1 03:35:45 noki kernel: [tcp_print_queue+535/592] tcp_print_queue+0x217/0x250 May 1 03:35:45 noki kernel: [dev_queue_xmit+150/640] dev_queue_xmit+0x96/0x280 May 1 03:35:45 noki kernel: [ip_finish_output+0/688] ip_finish_output+0x0/0x2b0 May 1 03:35:45 noki kernel: [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+2348/7232] tcp_ack+0x92c/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_input.c:2546 tcp_ack+0x955/0x1c40() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [dev_queue_xmit+150/640] dev_queue_xmit+0x96/0x280 May 1 03:35:45 noki kernel: [ip_finish_output+0/688] ip_finish_output+0x0/0x2b0 May 1 03:35:45 noki kernel: [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+2389/7232] tcp_ack+0x955/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: ------------[ cut here ]------------ May 1 03:35:45 noki kernel: WARNING: at net/ipv4/tcp_input.c:2550 tcp_ack+0x1bba/0x1c40() May 1 03:35:45 noki kernel: Modules linked in: saa7134_dvb tda826x mt352 tda10086 dvb_pll tuner tea5767 tda8290 tuner_xc2028 tda9887 tuner_simple mt20xx tea5761 saa7134 videodev v4l1_compat compat_ioctl32 v4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [dev_queue_xmit+150/640] dev_queue_xmit+0x96/0x280 May 1 03:35:45 noki kernel: [ip_finish_output+0/688] ip_finish_output+0x0/0x2b0 May 1 03:35:45 noki kernel: [ip_finish_output+316/688] ip_finish_output+0x13c/0x2b0 May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] tcp_verify_wq+0x35f/0x3b0 May 1 03:35:45 noki kernel: [tcp_ack+7098/7232] tcp_ack+0x1bba/0x1c40 May 1 03:35:45 noki kernel: [tcp_rcv_established+899/1840] tcp_rcv_established+0x383/0x730 May 1 03:35:45 noki kernel: [atomic_notifier_call_chain+23/32] atomic_notifier_call_chain+0x17/0x20 May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 May 1 03:35:45 noki kernel: [nf_iterate+96/144] nf_iterate+0x60/0x90 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [tcp_v4_rcv+1577/1776] tcp_v4_rcv+0x629/0x6f0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+148/416] ip_local_deliver_finish+0x94/0x1a0 May 1 03:35:45 noki kernel: [ip_local_deliver_finish+0/416] ip_local_deliver_finish+0x0/0x1a0 May 1 03:35:45 noki kernel: [ip_rcv_finish+259/800] ip_rcv_finish+0x103/0x320 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [ip_rcv+410/608] ip_rcv+0x19a/0x260 May 1 03:35:45 noki kernel: [ip_rcv_finish+0/800] ip_rcv_finish+0x0/0x320 May 1 03:35:45 noki kernel: [netif_receive_skb+672/1104] netif_receive_skb+0x2a0/0x450 May 1 03:35:45 noki kernel: [<d48254f8>] tulip_poll+0x3c8/0x610 [tulip] May 1 03:35:45 noki kernel: [<d4825ca8>] tulip_interrupt+0x568/0x840 [tulip] May 1 03:35:45 noki kernel: [tick_program_event+54/112] tick_program_event+0x36/0x70 May 1 03:35:45 noki kernel: [net_rx_action+147/336] net_rx_action+0x93/0x150 May 1 03:35:45 noki kernel: [__do_softirq+66/160] __do_softirq+0x42/0xa0 May 1 03:35:45 noki kernel: [do_softirq+38/48] do_softirq+0x26/0x30 May 1 03:35:45 noki kernel: [do_IRQ+74/128] do_IRQ+0x4a/0x80 May 1 03:35:45 noki kernel: [tick_notify+427/624] tick_notify+0x1ab/0x270 May 1 03:35:45 noki kernel: [common_interrupt+35/40] common_interrupt+0x23/0x28 May 1 03:35:45 noki kernel: [<d483cd05>] acpi_idle_enter_simple+0x153/0x1c6 [processor] May 1 03:35:45 noki kernel: [cpuidle_idle_call+107/160] cpuidle_idle_call+0x6b/0xa0 May 1 03:35:45 noki kernel: [cpuidle_idle_call+0/160] cpuidle_idle_call+0x0/0xa0 May 1 03:35:45 noki kernel: [cpu_idle+38/112] cpu_idle+0x26/0x70 May 1 03:35:45 noki kernel: ======================= May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- May 1 03:35:45 noki kernel: TCP DEBUG 2 0 4 7 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-01 9:01 ` Andy Furniss @ 2008-05-01 9:37 ` Ilpo Järvinen 2008-05-01 9:54 ` Andy Furniss 0 siblings, 1 reply; 15+ messages in thread From: Ilpo Järvinen @ 2008-05-01 9:37 UTC (permalink / raw) To: Andy Furniss; +Cc: Netdev On Thu, 1 May 2008, Andy Furniss wrote: > Caught one - > > > May 1 03:35:45 noki kernel: 4l2_common ir_kbd_i2c ir_common tveeprom Things above this got lost somewhere? > videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x > firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 > sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss > xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables > nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan > thermal processor button tulip > May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 > May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] > warn_on_slowpath+0x5f/0x90 > May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] > tcp_transmit_skb+0x47d/0x730 > May 1 03:35:45 noki kernel: [tcp_verify_wq+863/944] > tcp_verify_wq+0x35f/0x3b0 > May 1 03:35:45 noki kernel: [tcp_xmit_retransmit_queue+346/976] > tcp_xmit_retransmit_queue+0x15a/0x3d0 > May 1 03:35:45 noki kernel: [tcp_ack+4069/7232] tcp_ack+0xfe5/0x1c40 > May 1 03:35:45 noki kernel: [tcp_rcv_established+990/1840] > tcp_rcv_established+0x3de/0x730 > May 1 03:35:45 noki kernel: [tcp_v4_do_rcv+179/464] tcp_v4_do_rcv+0xb3/0x1d0 [...snip...] > May 1 03:35:45 noki kernel: ---[ end trace ca0e42fa80915e32 ]--- > May 1 03:35:45 noki kernel: P: 4 L: 1 vs 1 S: 3 vs 3 F: 4 vs 6 w: > 2200787703-2200792303 (7) Thanks, a genuine one, fackets_out (= 6) > packets_out (= 4). :-) ...I'll try to figure it out anyway. -- i. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-01 9:37 ` Ilpo Järvinen @ 2008-05-01 9:54 ` Andy Furniss 2008-05-02 10:51 ` Ilpo Järvinen 0 siblings, 1 reply; 15+ messages in thread From: Andy Furniss @ 2008-05-01 9:54 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev Ilpo Järvinen wrote: > On Thu, 1 May 2008, Andy Furniss wrote: > >> Caught one - >> >> >> May 1 03:35:45 noki kernel: 4l2_common ir_kbd_i2c ir_common tveeprom > > Things above this got lost somewhere? No - well not by the paste, I selected either side when pasting and edited them out of the mail. I did notice that and double checked - it is the first May 1 in my kernel log. Apr 30 21:23:13 noki kernel: br0: topology change detected, propagating Apr 30 21:23:13 noki kernel: br0: port 2(eth1) entering forwarding state May 1 03:35:45 noki kernel: 4l2_common ir_kbd_i2c ir_common tveeprom videobuf_dvb dvb_core videobuf_dma_sg videobuf_core nxt200x isl6421 tda1004x firmware_class tda827x cls_flow sch_ingress xt_mark xt_length xt_MARK cls_u32 sch_sfq cls_fw sch_htb ipt_REJECT xt_state iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle nf_nat_ftp nf_conntrack_ftp iptable_nat ip_tables nf_nat x_tables nf_conntrack_ipv4 nf_conntrack bridge llc container fan thermal processor button tulip May 1 03:35:45 noki kernel: Pid: 0, comm: swapper Not tainted 2.6.25 #2 May 1 03:35:45 noki kernel: [warn_on_slowpath+95/144] warn_on_slowpath+0x5f/0x90 May 1 03:35:45 noki kernel: [tcp_transmit_skb+1149/1840] tcp_transmit_skb+0x47d/0x730 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-01 9:54 ` Andy Furniss @ 2008-05-02 10:51 ` Ilpo Järvinen 2008-05-03 12:03 ` Andy Furniss 0 siblings, 1 reply; 15+ messages in thread From: Ilpo Järvinen @ 2008-05-02 10:51 UTC (permalink / raw) To: Andy Furniss; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 12098 bytes --] On Thu, 1 May 2008, Andy Furniss wrote: > Ilpo Järvinen wrote: > > On Thu, 1 May 2008, Andy Furniss wrote: > > > > > Caught one - > > > > > > > > > May 1 03:35:45 noki kernel: 4l2_common ir_kbd_i2c ir_common tveeprom > > > > Things above this got lost somewhere? > > No - well not by the paste, I selected either side when pasting and edited > them out of the mail. I did notice that and double checked - it is the first > May 1 in my kernel log. I suppose it flooded the log so that head of the log got lost already before userspace could consume it... (I think it's ringbuffer or something though I'm not sure), it might be wise to enlarge it (I think it's controlled through CONFIG_LOG_BUF_SHIFT). Since I've not yet found any other path that could lead to it except the yesterday's false-positive, getting the write queue state more accurately captured by the debug patch on the very first occassion might help. Here's less spammy version of the debug patch (I added one printout as well to get more usefulness to transitional state reported). -- i. [PATCH] TCP: debug S+L (for 2.6.25/.26-rcs), v1.4.4 Debugs sacked & lost skb inconstencies in TCP write queue & verifies fackets_out related variables. --- include/net/tcp.h | 10 ++++- net/ipv4/tcp_input.c | 34 ++++++++++++- net/ipv4/tcp_ipv4.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++++ net/ipv4/tcp_output.c | 20 ++++++-- 4 files changed, 186 insertions(+), 7 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 4fd3eb2..b7b73fb 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -272,6 +272,9 @@ DECLARE_SNMP_STAT(struct tcp_mib, tcp_statistics); #define TCP_ADD_STATS_BH(field, val) SNMP_ADD_STATS_BH(tcp_statistics, field, val) #define TCP_ADD_STATS_USER(field, val) SNMP_ADD_STATS_USER(tcp_statistics, field, val) +extern void tcp_print_queue(struct sock *sk); +extern int tcp_verify_wq(struct sock *sk); + extern void tcp_v4_err(struct sk_buff *skb, u32); extern void tcp_shutdown (struct sock *sk, int how); @@ -770,7 +773,12 @@ static inline __u32 tcp_current_ssthresh(const struct sock *sk) } /* Use define here intentionally to get WARN_ON location shown at the caller */ -#define tcp_verify_left_out(tp) WARN_ON(tcp_left_out(tp) > tp->packets_out) +#define tcp_verify_left_out(tp) \ + do {\ + int res; \ + res = tcp_verify_wq((struct sock *)tp); \ + WARN_ON(res || tcp_left_out(tp) > tp->packets_out); \ + } while(0) extern void tcp_enter_cwr(struct sock *sk, const int set_ssthresh); extern __u32 tcp_init_cwnd(struct tcp_sock *tp, struct dst_entry *dst); diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index bbb7d88..7dd7939 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -1428,8 +1428,10 @@ tcp_sacktag_write_queue(struct sock *sk, struct sk_buff *ack_skb, int first_sack_index; if (!tp->sacked_out) { - if (WARN_ON(tp->fackets_out)) + if (WARN_ON(tp->fackets_out)) { + tcp_verify_left_out(tp); tp->fackets_out = 0; + } tcp_highest_sack_reset(sk); } @@ -2157,7 +2159,14 @@ static void tcp_mark_head_lost(struct sock *sk, int packets) int err; unsigned int mss; + tcp_verify_left_out(tp); + BUG_TRAP(packets <= tp->packets_out); + if (packets > tp->packets_out) { + pr_err("TCP debug %u/%u-%u/%u %d\n", + tp->fackets_out, tp->sacked_out, tp->reordering, + tp->packets_out, tp->rx_opt.sack_ok); + } if (tp->lost_skb_hint) { skb = tp->lost_skb_hint; cnt = tp->lost_cnt_hint; @@ -2534,10 +2543,15 @@ static void tcp_fastretrans_alert(struct sock *sk, int pkts_acked, int flag) (tcp_fackets_out(tp) > tp->reordering)); int fast_rexmit = 0; + tcp_verify_left_out(tp); + if (WARN_ON(!tp->packets_out && tp->sacked_out)) tp->sacked_out = 0; - if (WARN_ON(!tp->sacked_out && tp->fackets_out)) + if (WARN_ON(!tp->sacked_out && tp->fackets_out)) { + printk(KERN_ERR "TCP DEBUG %u %u %d %d\n", tp->fackets_out, + tp->packets_out, pkts_acked, tp->rx_opt.sack_ok); tp->fackets_out = 0; + } /* Now state machine starts. * A. ECE, hence prohibit cwnd undoing, the reduction is required. */ @@ -2680,6 +2694,10 @@ static void tcp_fastretrans_alert(struct sock *sk, int pkts_acked, int flag) if (do_lost || (tcp_is_fack(tp) && tcp_head_timedout(sk))) tcp_update_scoreboard(sk, fast_rexmit); tcp_cwnd_down(sk, flag); + + WARN_ON(tcp_write_queue_head(sk) == NULL); + WARN_ON(!tp->packets_out); + tcp_xmit_retransmit_queue(sk); } @@ -2832,6 +2850,9 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets) tcp_mtup_probe_success(sk, skb); } + WARN_ON((sacked & (TCPCB_SACKED_ACKED|TCPCB_LOST)) == + (TCPCB_SACKED_ACKED|TCPCB_LOST)); + if (sacked & TCPCB_RETRANS) { if (sacked & TCPCB_SACKED_RETRANS) tp->retrans_out -= acked_pcount; @@ -2922,6 +2943,8 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets) } } + tcp_verify_left_out(tp); + #if FASTRETRANS_DEBUG > 0 BUG_TRAP((int)tp->sacked_out >= 0); BUG_TRAP((int)tp->lost_out >= 0); @@ -3210,6 +3233,8 @@ static int tcp_ack(struct sock *sk, struct sk_buff *skb, int flag) prior_fackets = tp->fackets_out; prior_in_flight = tcp_packets_in_flight(tp); + tcp_verify_left_out(tp); + if (!(flag & FLAG_SLOWPATH) && after(ack, prior_snd_una)) { /* Window is constant, pure forward advance. * No more checks are required. @@ -3269,9 +3294,14 @@ static int tcp_ack(struct sock *sk, struct sk_buff *skb, int flag) tcp_cong_avoid(sk, ack, prior_in_flight); } + if (WARN_ON((icsk->icsk_ca_state == TCP_CA_Open) && tcp_left_out(tp))) + tcp_print_queue(sk); + if ((flag & FLAG_FORWARD_PROGRESS) || !(flag & FLAG_NOT_DUP)) dst_confirm(sk->sk_dst_cache); + tcp_verify_left_out(tp); + return 1; no_queue: diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 00156bf..0aaaeb7 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -108,6 +108,135 @@ struct inet_hashinfo __cacheline_aligned tcp_hashinfo = { .lhash_wait = __WAIT_QUEUE_HEAD_INITIALIZER(tcp_hashinfo.lhash_wait), }; +void tcp_print_queue(struct sock *sk) +{ + struct tcp_sock *tp = tcp_sk(sk); + struct sk_buff *skb; + char s[50+1]; + char h[50+1]; + int idx = 0; + int i; + +/* i = 0; + tcp_for_write_queue(skb, sk) { + if (skb == tcp_send_head(sk)) + printk(KERN_ERR "head %u %p\n", i, skb); + else + printk(KERN_ERR "skb %u %p\n", i, skb); + i++; + } */ + + tcp_for_write_queue(skb, sk) { + if (skb == tcp_send_head(sk)) + break; + + for (i = 0; i < tcp_skb_pcount(skb); i++) { + if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) { + s[idx] = 'S'; + if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) + s[idx] = 'B'; + + } else if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) { + s[idx] = 'L'; + } else { + s[idx] = ' '; + } + if (s[idx] != ' ' && skb->len < tp->mss_cache) + s[idx] += 'a' - 'A'; + + if (i == 0) { + if (skb == tcp_highest_sack(sk)) + h[idx] = 'h'; + else + h[idx] = '+'; + } else { + h[idx] = '-'; + } + + if (++idx >= 50) { + s[idx] = 0; + h[idx] = 0; + printk(KERN_ERR "TCP wq(s) %s\n", s); + printk(KERN_ERR "TCP wq(h) %s\n", h); + idx = 0; + } + } + } + if (idx) { + s[idx] = '<'; + s[idx+1] = 0; + h[idx] = '<'; + h[idx+1] = 0; + printk(KERN_ERR "TCP wq(s) %s\n", s); + printk(KERN_ERR "TCP wq(h) %s\n", h); + } + printk(KERN_ERR "l%u s%u f%u p%u seq: wq%u, su%u hs%u sn%u\n", + tp->lost_out, tp->sacked_out, tp->fackets_out, tp->packets_out, + TCP_SKB_CB(tcp_write_queue_head(sk))->seq, + tp->snd_una, tcp_highest_sack_seq(tp), tp->snd_nxt); +} + +int tcp_verify_wq(struct sock *sk) +{ + struct tcp_sock *tp = tcp_sk(sk); + u32 lost = 0; + u32 sacked = 0; + u32 packets = 0; + u32 fackets = 0; + int hs_valid = 0; + int inconsistent = 0; + struct sk_buff *skb; + + tcp_for_write_queue(skb, sk) { + if (skb == tcp_send_head(sk)) + break; + + if ((fackets == packets) && (skb == tp->highest_sack)) + hs_valid = 1; + + packets += tcp_skb_pcount(skb); + + if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) { + sacked += tcp_skb_pcount(skb); + if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) + printk(KERN_ERR "Sacked bitmap S+L: %u %u-%u/%u\n", + TCP_SKB_CB(skb)->sacked, + TCP_SKB_CB(skb)->end_seq - tp->snd_una, + TCP_SKB_CB(skb)->seq - tp->snd_una, + tp->snd_una); + fackets = packets; + hs_valid = 0; + } + if (TCP_SKB_CB(skb)->sacked & TCPCB_LOST) + lost += tcp_skb_pcount(skb); + } + + if ((fackets == packets) && (tp->highest_sack == tcp_send_head(sk))) + hs_valid = 1; + + if ((lost != tp->lost_out) || + (tcp_is_sack(tp) && (sacked != tp->sacked_out)) || + ((sacked || (tcp_is_sack(tp) && tp->sacked_out)) && !hs_valid) || + (packets != tp->packets_out) || + (fackets != tp->fackets_out) || + (tcp_is_reno(tp) && tp->packets_out && + (tcp_left_out(tp) == tp->packets_out) && + !(TCP_SKB_CB(tcp_write_queue_head(sk))->sacked & TCPCB_LOST)) || + (tcp_left_out(tp) > tp->packets_out)) { + printk(KERN_ERR "P: %u L: %u vs %u S: %u vs %u F: %u vs %u w: %u-%u (%u)\n", + tp->packets_out, + lost, tp->lost_out, + sacked, tp->sacked_out, + fackets, tp->fackets_out, + tp->snd_una, tp->snd_nxt, + tp->rx_opt.sack_ok); + tcp_print_queue(sk); + inconsistent = 1; + } + + return inconsistent; +} + static inline __u32 tcp_v4_init_sequence(struct sk_buff *skb) { return secure_tcp_sequence_number(ip_hdr(skb)->daddr, diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index d29ef79..84113be 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -779,10 +779,9 @@ int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len, tp->lost_out -= diff; /* Adjust Reno SACK estimate. */ - if (tcp_is_reno(tp) && diff > 0) { + if (tcp_is_reno(tp) && diff > 0) tcp_dec_pcount_approx_int(&tp->sacked_out, diff); - tcp_verify_left_out(tp); - } + tcp_adjust_fackets_out(sk, skb, diff); } @@ -790,6 +789,8 @@ int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len, skb_header_release(buff); tcp_insert_write_queue_after(skb, buff, sk); + tcp_verify_left_out(tp); + return 0; } @@ -1471,6 +1472,7 @@ static int tcp_write_xmit(struct sock *sk, unsigned int mss_now, int nonagle) } else if (result > 0) { sent_pkts = 1; } + tcp_verify_left_out(tp); while ((skb = tcp_send_head(sk))) { unsigned int limit; @@ -1772,6 +1774,7 @@ static void tcp_retrans_try_collapse(struct sock *sk, struct sk_buff *skb, tcp_clear_retrans_hints_partial(tp); sk_wmem_free_skb(sk, next_skb); + tcp_verify_left_out(tp); } /* Do a simple retransmit without using the backoff mechanisms in @@ -1895,10 +1898,13 @@ int tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb) (TCP_SKB_CB(skb)->flags & TCPCB_FLAG_FIN) && tp->snd_una == (TCP_SKB_CB(skb)->end_seq - 1)) { if (!pskb_trim(skb, 0)) { + WARN_ON(TCP_SKB_CB(skb)->sacked & TCPCB_LOST); /* Reuse, even though it does some unnecessary work */ tcp_init_nondata_skb(skb, TCP_SKB_CB(skb)->end_seq - 1, TCP_SKB_CB(skb)->flags); skb->ip_summed = CHECKSUM_NONE; + + tcp_verify_left_out(tp); } } @@ -1981,8 +1987,10 @@ void tcp_xmit_retransmit_queue(struct sock *sk) * packet to be MSS sized and all the * packet counting works out. */ - if (tcp_packets_in_flight(tp) >= tp->snd_cwnd) + if (tcp_packets_in_flight(tp) >= tp->snd_cwnd) { + tcp_verify_left_out(tp); return; + } if (sacked & TCPCB_LOST) { if (!(sacked & (TCPCB_SACKED_ACKED|TCPCB_SACKED_RETRANS))) { @@ -2008,6 +2016,8 @@ void tcp_xmit_retransmit_queue(struct sock *sk) } } + tcp_verify_left_out(tp); + /* OK, demanded retransmission is finished. */ /* Forward retransmissions are possible only during Recovery. */ @@ -2065,6 +2075,8 @@ void tcp_xmit_retransmit_queue(struct sock *sk) NET_INC_STATS_BH(LINUX_MIB_TCPFORWARDRETRANS); } + + tcp_verify_left_out(tp); } /* Send a fin. The caller locks the socket for us. This cannot be -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-02 10:51 ` Ilpo Järvinen @ 2008-05-03 12:03 ` Andy Furniss 2008-05-12 10:50 ` Ilpo Järvinen 0 siblings, 1 reply; 15+ messages in thread From: Andy Furniss @ 2008-05-03 12:03 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev Ilpo Järvinen wrote: > I suppose it flooded the log so that head of the log got lost already > before userspace could consume it... (I think it's ringbuffer or something > though I'm not sure), it might be wise to enlarge it (I think it's > controlled through CONFIG_LOG_BUF_SHIFT). > > Since I've not yet found any other path that could lead to it except the > yesterday's false-positive, getting the write queue state more accurately > captured by the debug patch on the very first occassion might help. Here's > less spammy version of the debug patch (I added one printout as well to > get more usefulness to transitional state reported). > OK, increased CONFIG_LOG_BUF_SHIFT from 14 to 18 and reversed old / applied new patch. Andy. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-03 12:03 ` Andy Furniss @ 2008-05-12 10:50 ` Ilpo Järvinen 2008-05-13 10:20 ` Andy Furniss 2008-05-29 1:35 ` Andy Furniss 0 siblings, 2 replies; 15+ messages in thread From: Ilpo Järvinen @ 2008-05-12 10:50 UTC (permalink / raw) To: Andy Furniss; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 559 bytes --] On Sat, 3 May 2008, Andy Furniss wrote: > Ilpo Järvinen wrote: > > > Since I've not yet found any other path that could lead to it except the > > yesterday's false-positive, getting the write queue state more accurately > > captured by the debug patch on the very first occassion might help. Here's > > less spammy version of the debug patch (I added one printout as well to get > > more usefulness to transitional state reported). > > > > OK, increased CONFIG_LOG_BUF_SHIFT from 14 to 18 and reversed old / applied > new patch. Any luck so far? -- i. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-12 10:50 ` Ilpo Järvinen @ 2008-05-13 10:20 ` Andy Furniss 2008-05-29 1:35 ` Andy Furniss 1 sibling, 0 replies; 15+ messages in thread From: Andy Furniss @ 2008-05-13 10:20 UTC (permalink / raw) To: �; +Cc: Netdev � wrote: > On Sat, 3 May 2008, Andy Furniss wrote: > >> Ilpo J�rvinen wrote: >> >>> Since I've not yet found any other path that could lead to it except the >>> yesterday's false-positive, getting the write queue state more accurately >>> captured by the debug patch on the very first occassion might help. Here's >>> less spammy version of the debug patch (I added one printout as well to get >>> more usefulness to transitional state reported). >>> >> OK, increased CONFIG_LOG_BUF_SHIFT from 14 to 18 and reversed old / applied >> new patch. > > Any luck so far? > No, nothing yet :-( ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-12 10:50 ` Ilpo Järvinen 2008-05-13 10:20 ` Andy Furniss @ 2008-05-29 1:35 ` Andy Furniss 2008-05-29 8:10 ` Ilpo Järvinen 2008-06-04 10:42 ` [PATCH] tcp: fix skb vs fack_count out-of-sync condition Ilpo Järvinen 1 sibling, 2 replies; 15+ messages in thread From: Andy Furniss @ 2008-05-29 1:35 UTC (permalink / raw) To: �; +Cc: Netdev [-- Attachment #1: Type: text/plain, Size: 679 bytes --] � wrote: > On Sat, 3 May 2008, Andy Furniss wrote: > >> Ilpo J�rvinen wrote: >> >>> Since I've not yet found any other path that could lead to it except the >>> yesterday's false-positive, getting the write queue state more accurately >>> captured by the debug patch on the very first occassion might help. Here's >>> less spammy version of the debug patch (I added one printout as well to get >>> more usefulness to transitional state reported). >>> >> OK, increased CONFIG_LOG_BUF_SHIFT from 14 to 18 and reversed old / applied >> new patch. > > Any luck so far? > Finally - looks like I got two together. It doesn't start with WARNING - but dmesg isn't full. Andy. [-- Attachment #2: tcp-debug.bz2 --] [-- Type: application/x-bzip2, Size: 6907 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: WARNING: at net/ipv4/tcp_input.c 2008-05-29 1:35 ` Andy Furniss @ 2008-05-29 8:10 ` Ilpo Järvinen 2008-06-04 10:42 ` [PATCH] tcp: fix skb vs fack_count out-of-sync condition Ilpo Järvinen 1 sibling, 0 replies; 15+ messages in thread From: Ilpo Järvinen @ 2008-05-29 8:10 UTC (permalink / raw) To: Andy Furniss; +Cc: Netdev On Thu, 29 May 2008, Andy Furniss wrote: > ? wrote: > > On Sat, 3 May 2008, Andy Furniss wrote: > > > > > Ilpo J?rvinen wrote: > > > > > > > Since I've not yet found any other path that could lead to it except the > > > > yesterday's false-positive, getting the write queue state more > > > > accurately > > > > captured by the debug patch on the very first occassion might help. > > > > Here's > > > > less spammy version of the debug patch (I added one printout as well to > > > > get > > > > more usefulness to transitional state reported). > > > > > > > OK, increased CONFIG_LOG_BUF_SHIFT from 14 to 18 and reversed old / > > > applied > > > new patch. > > > > Any luck so far? > > > > Finally - looks like I got two together. > > It doesn't start with WARNING - but dmesg isn't full. This looks ok trace this time :-), I changed the ordering of the lines so that warning was supposed to appear only after the wq print. Now it's just the hard par remaining, ie., I must figure out what was the state of TCP before the event the event... :-). If you still get additional events (you could if a particular torrent peer is there having the same network behavior), please post them as well if they reveal something that helps exclude cases (if I don't right away get the idea :-)). -- i. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH] tcp: fix skb vs fack_count out-of-sync condition 2008-05-29 1:35 ` Andy Furniss 2008-05-29 8:10 ` Ilpo Järvinen @ 2008-06-04 10:42 ` Ilpo Järvinen 2008-06-04 19:08 ` David Miller 1 sibling, 1 reply; 15+ messages in thread From: Ilpo Järvinen @ 2008-06-04 10:42 UTC (permalink / raw) To: Andy Furniss, David Miller Cc: Netdev, int 986, Brian Vowell, Robin Johnson, Rafael J. Wysocki, Andrew Morton [-- Attachment #1: Type: TEXT/PLAIN, Size: 3777 bytes --] On Thu, 29 May 2008, Andy Furniss wrote: > ? wrote: > > On Sat, 3 May 2008, Andy Furniss wrote: > > > > > Ilpo J?rvinen wrote: > > > > > > > Since I've not yet found any other path that could lead to it except the > > > > yesterday's false-positive, getting the write queue state more > > > > accurately > > > > captured by the debug patch on the very first occassion might help. > > > > Here's > > > > less spammy version of the debug patch (I added one printout as well to > > > > get > > > > more usefulness to transitional state reported). > > > > > > > OK, increased CONFIG_LOG_BUF_SHIFT from 14 to 18 and reversed old / > > > applied > > > new patch. > > > > Any luck so far? > > > > Finally - looks like I got two together. > > It doesn't start with WARNING - but dmesg isn't full. Ok, here's finally the fix to those WARN_ON(!tp->sacked_out && tp->fackets_out) WARNINGs, both stable-2.6.25 and linus' tree affected. -- i. -- [PATCH] tcp: fix skb vs fack_count out-of-sync condition This bug is able to corrupt fackets_out in very rare cases. In order for this to cause corruption: 1) DSACK in the middle of previous SACK block must be generated. 2) In order to take that particular branch, part or all of the DSACKed segment must already be SACKed so that we have that in cache in the first place. 3) The new info must be top enough so that fackets_out will be updated on this iteration. ...then fack_count is updated while skb wasn't, then we walk again that particular segment thus updating fack_count twice for a single skb and finally that value is assigned to fackets_out by tcp_sacktag_one. It is safe to call tcp_sacktag_one just once for a segment (at DSACK), no need to call again for plain SACK. Potential problem of the miscount are limited to premature entry to recovery and to inflated reordering metric (which could even cancel each other out in the most the luckiest scenarios :-)). Both are quite insignificant in worst case too and there exists also code to reset them (fackets_out once sacked_out becomes zero and reordering metric on RTO). This has been reported by a number of people, because it occurred quite rarely, it has been very evasive. Andy Furniss was able to get it to occur couple of times so that a bit more info was collected about the problem using a debug patch, though it still required lot of checking around. Thanks also to others who have tried to help here. This is listed as Bugzilla #10346. The bug was introduced by me in commit 68f8353b48 ([TCP]: Rewrite SACK block processing & sack_recv_cache use), I probably thought back then that there's need to scan that entry twice or didn't dare to make it go through it just once there. Going through twice would have required restoring fack_count after the walk but as noted above, I chose to drop the additional walk step altogether here. Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> Cc: int 986 <int986@gmail.com> Cc: Andy Furniss <lists@andyfurniss.entadsl.com> Cc: Brian Vowell <brian.vowell@gmail.com> --- net/ipv4/tcp_input.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 5d5b08b..a00c7f8 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -1392,9 +1392,9 @@ static struct sk_buff *tcp_maybe_skipping_dsack(struct sk_buff *skb, if (before(next_dup->start_seq, skip_to_seq)) { skb = tcp_sacktag_skip(skb, sk, next_dup->start_seq, fack_count); - tcp_sacktag_walk(skb, sk, NULL, - next_dup->start_seq, next_dup->end_seq, - 1, fack_count, reord, flag); + skb = tcp_sacktag_walk(skb, sk, NULL, + next_dup->start_seq, next_dup->end_seq, + 1, fack_count, reord, flag); } return skb; -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH] tcp: fix skb vs fack_count out-of-sync condition 2008-06-04 10:42 ` [PATCH] tcp: fix skb vs fack_count out-of-sync condition Ilpo Järvinen @ 2008-06-04 19:08 ` David Miller 0 siblings, 0 replies; 15+ messages in thread From: David Miller @ 2008-06-04 19:08 UTC (permalink / raw) To: ilpo.jarvinen; +Cc: lists, netdev, int986, brian.vowell, robbat2, rjw, akpm From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> Date: Wed, 4 Jun 2008 13:42:43 +0300 (EEST) > [PATCH] tcp: fix skb vs fack_count out-of-sync condition Applied and queued up for -stable, thanks! ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2008-06-04 19:08 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-04-28 12:02 WARNING: at net/ipv4/tcp_input.c Andy Furniss 2008-04-28 15:57 ` Ilpo Järvinen 2008-04-28 19:07 ` Andy Furniss 2008-04-28 19:21 ` Ilpo Järvinen 2008-05-01 9:01 ` Andy Furniss 2008-05-01 9:37 ` Ilpo Järvinen 2008-05-01 9:54 ` Andy Furniss 2008-05-02 10:51 ` Ilpo Järvinen 2008-05-03 12:03 ` Andy Furniss 2008-05-12 10:50 ` Ilpo Järvinen 2008-05-13 10:20 ` Andy Furniss 2008-05-29 1:35 ` Andy Furniss 2008-05-29 8:10 ` Ilpo Järvinen 2008-06-04 10:42 ` [PATCH] tcp: fix skb vs fack_count out-of-sync condition Ilpo Järvinen 2008-06-04 19:08 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).