* Question regarding kernel panic in net/ipv4/tcp_output.c
@ 2012-09-04 11:55 smka2012
2012-09-04 12:23 ` Eric Dumazet
0 siblings, 1 reply; 3+ messages in thread
From: smka2012 @ 2012-09-04 11:55 UTC (permalink / raw)
To: netdev
Hi,
I recently had a severe issue with multiple servers that stopped working at the same time. By using the stacktrace, I could locate the following line in the tcp_retransmit_skb function of the tcp_output.c kernel source that led to a kernel panic on all servers. I am using Debian 6.0 with the kernel version 2.6.32-45.
(gdb) list *(tcp_retransmit_skb+0x66)
0x1a93 is in tcp_retransmit_skb (net/ipv4/tcp_output.c:1905).
1900 if (atomic_read(&sk->sk_wmem_alloc) >
1901 min(sk->sk_wmem_queued + (sk->sk_wmem_queued >> 2), sk->sk_sndbuf))
1902 return -EAGAIN;
1903
1904 if (before(TCP_SKB_CB(skb)->seq, tp->snd_una)) {
--->
1905 if (before(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
1906 BUG();
<---
1907 if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
1908 return -ENOMEM;
1909 }
Now I am interested in getting to know what can lead to a situation that the condition in line 1905 evaluates true and why the kernel goes into the BUG() function in that case and does not only return an error. All servers reached this line of code. They all were connected to a switch that broke the same time. However, I cannot say if the switch broke before the servers and eventually affected the servers or if the switch was also itself affected by some external event.
Thank you very much for your help.
Kind Regards,
Sascha
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Question regarding kernel panic in net/ipv4/tcp_output.c
2012-09-04 11:55 Question regarding kernel panic in net/ipv4/tcp_output.c smka2012
@ 2012-09-04 12:23 ` Eric Dumazet
2012-09-05 9:01 ` Aw: " "Sascha Mühlbach"
0 siblings, 1 reply; 3+ messages in thread
From: Eric Dumazet @ 2012-09-04 12:23 UTC (permalink / raw)
To: smka2012; +Cc: netdev
On Tue, 2012-09-04 at 13:55 +0200, smka2012@email.de wrote:
> Hi,
>
> I recently had a severe issue with multiple servers that stopped
> working at the same time. By using the stacktrace, I could locate the
> following line in the tcp_retransmit_skb function of the tcp_output.c
> kernel source that led to a kernel panic on all servers. I am using
> Debian 6.0 with the kernel version 2.6.32-45.
>
> (gdb) list *(tcp_retransmit_skb+0x66)
> 0x1a93 is in tcp_retransmit_skb (net/ipv4/tcp_output.c:1905).
> 1900 if (atomic_read(&sk->sk_wmem_alloc) >
> 1901 min(sk->sk_wmem_queued + (sk->sk_wmem_queued >> 2),
> sk->sk_sndbuf))
> 1902 return -EAGAIN;
> 1903
> 1904 if (before(TCP_SKB_CB(skb)->seq, tp->snd_una)) {
>
> --->
> 1905 if (before(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
> 1906 BUG();
> <---
>
> 1907 if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
> 1908 return -ENOMEM;
> 1909 }
>
> Now I am interested in getting to know what can lead to a situation
> that the condition in line 1905 evaluates true and why the kernel goes
> into the BUG() function in that case and does not only return an
> error. All servers reached this line of code. They all were connected
> to a switch that broke the same time. However, I cannot say if the
> switch broke before the servers and eventually affected the servers or
> if the switch was also itself affected by some external event.
>
> Thank you very much for your help.
>
> Kind Regards,
> Sascha
> --
You can see this BUG() as an early notification of a hard to
debug/diagnose bug.
We shouldnt take this path at all.
If we do, we have an earlier bug that we should fix anyway, because
machine is going to crash.
It would be nice you sent the stack trace you had.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Aw: Re: Question regarding kernel panic in net/ipv4/tcp_output.c
2012-09-04 12:23 ` Eric Dumazet
@ 2012-09-05 9:01 ` "Sascha Mühlbach"
0 siblings, 0 replies; 3+ messages in thread
From: "Sascha Mühlbach" @ 2012-09-05 9:01 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
Hi,
this is the stack trace I could save by making a screenshot of the remote console:
[<ffffffff8101193b>] ? invalid_op+0x1b/0x20
[<ffffffff81289e6a>] ? tcp_retransmit_skb+0x66/0x5aa
[<ffffffff8128a546>] ? tcp_xmit_retransmit_queue+0x198/0x223
[<ffffffff81286113>] ? tcp_ack+0x1744/0x1952
[<ffffffff81286871>] ? tcp_validate_incoming+0x1ba/0x2be
[<ffffffff81286f5e>] ? tcp_rcv_established+0x5e9/0x6d9
[<ffffffff8128e00f>] ? tcp_v4_do_rcv+0x1bb/0x376
[<ffffffff8128e639>] ? tcp_v4_rcv+0x46f/0x6f8
[<ffffffff81273afa>] ? ip_local_deliver_finish+0x0/0x1e9
[<ffffffff81273afa>] ? ip_local_deliver_finish+0x0/0x1e9
[<ffffffff81273c40>] ? ip_local_deliver_finish+0x146/0x1e9
[<ffffffff8127378f>] ? ip_rcv_finish+0x373/0x38d
[<ffffffffa004d95c>] ? bnx2_poll_work+0x954/0xa7e [bnx2]
[<ffffffffa004d95c>] ? bnx2_poll_work+0x954/0xa7e [bnx2]
[<ffffffff8105aeb6>] ? __mod_timer+0x141/0x153
[<ffffffff810964a1>] ? handle_edge_irq+0xdd/0x101
[<ffffffffa004daae>] ? bnx2_poll_msix+0x28/0xa6 [bnx2]
[<ffffffff8125039f>] ? net_rx_action+0xae/0x1c9
[<ffffffff81053d6f>] ? __do_softirq+0xdd/0x1a6
[<ffffffff81011cac>] ? call_softirq+0x1c/0x30
[<ffffffff8101322b>] ? do_softirq+0x3f/0x7c
[<ffffffff81053bdf>] ? irq_exit+0x36/0x76
[<ffffffff81012922>] ? do_IRQ+0xa0/0xb6
[<ffffffff810114d3>] ? ret_from_intr+0x0/0x11
<E0I> [<ffffffffa0135509>] ? acpi_idle_enter_bm+0x27d/0x2af [processor]
[<ffffffffa0135509>] ? acpi_idle_enter_bm+0x27d/0x2af [processor]
[<ffffffffa0135502>] ? acpi_idle_enter_bm+0x276/0x2af [processor]
[<ffffffff8123a2c6>] ? cpuidle_idle_call+0x94/0xee
[<ffffffff8100fe97>] ? cpu_idle+0xa2/0xda
[<ffffffff8151c140>] ? early_idt_handler+0x0/0x71
[<ffffffff8151ccdd>] ? start_kernel+0x3dc/0x3e8
[<ffffffff8151c3b7>] ? x86_64_start_kernel+0xf9/0x106
Kind Regards,
Sascha
>Gesendet: Dienstag, 04. September 2012 um 14:23 Uhr
>Von: "Eric Dumazet" <eric.dumazet@gmail.com>
>An: smka2012@email.de
>Cc: netdev@vger.kernel.org
>Betreff: Re: Question regarding kernel panic in net/ipv4/tcp_output.c
>On Tue, 2012-09-04 at 13:55 +0200, smka2012@email.de wrote:
>> Hi,
>>
>> I recently had a severe issue with multiple servers that stopped
>> working at the same time. By using the stacktrace, I could locate the
>> following line in the tcp_retransmit_skb function of the tcp_output.c
>> kernel source that led to a kernel panic on all servers. I am using
>> Debian 6.0 with the kernel version 2.6.32-45.
>>
>> (gdb) list *(tcp_retransmit_skb+0x66)
>> 0x1a93 is in tcp_retransmit_skb (net/ipv4/tcp_output.c:1905).
>> 1900 if (atomic_read(&sk->sk_wmem_alloc) >
>> 1901 min(sk->sk_wmem_queued + (sk->sk_wmem_queued >> 2),
>> sk->sk_sndbuf))
>> 1902 return -EAGAIN;
>> 1903
>> 1904 if (before(TCP_SKB_CB(skb)->seq, tp->snd_una)) {
>>
>> --->
>> 1905 if (before(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
>> 1906 BUG();
>> <---
>>
>> 1907 if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
>> 1908 return -ENOMEM;
>> 1909 }
>>
>> Now I am interested in getting to know what can lead to a situation
>> that the condition in line 1905 evaluates true and why the kernel goes
>> into the BUG() function in that case and does not only return an
>> error. All servers reached this line of code. They all were connected
>> to a switch that broke the same time. However, I cannot say if the
>> switch broke before the servers and eventually affected the servers or
>> if the switch was also itself affected by some external event.
>>
>> Thank you very much for your help.
>>
>> Kind Regards,
>> Sascha
>> --
>
>You can see this BUG() as an early notification of a hard to
>debug/diagnose bug.
>
>We shouldnt take this path at all.
>
>If we do, we have an earlier bug that we should fix anyway, because
>machine is going to crash.
>
>It would be nice you sent the stack trace you had.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-09-05 9:01 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-04 11:55 Question regarding kernel panic in net/ipv4/tcp_output.c smka2012
2012-09-04 12:23 ` Eric Dumazet
2012-09-05 9:01 ` Aw: " "Sascha Mühlbach"
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox