* [PATCH][RFC] race in generic address resolution
@ 2008-02-04 14:27 Blaschka
2008-02-06 4:51 ` David Miller
0 siblings, 1 reply; 6+ messages in thread
From: Blaschka @ 2008-02-04 14:27 UTC (permalink / raw)
To: netdev, davem
I'm running a SMP maschine (2 CPUs) configured as a router. During heavy
traffic kernel dies with following message:
<2>kernel BUG at /home/autobuild/BUILD/linux-2.6.23-20080125/net/core/skbuff.c:648!
<4>illegal operation: 0001 [#1] PREEMPT SMP
<4>Modules linked in: dm_multipath sunrpc dm_mod qeth_l3 vmur vmcp qeth_l2 qeth ccwgroup
<4>CPU: 1 Not tainted 2.6.23-26.x.20080125-s390xdefault #1
<4>Process swapper (pid: 0, task: 000000001ff80bb8, ksp: 000000001ff8dd98)
<4>Krnl PSW : 0704100180000000 000000000034877e (pskb_expand_head+0x3a/0x210)
<4> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:0 CC:1 PM:0 EA:3
<4>Krnl GPRS: 0000000000000000 0000000000000100 000000001b2b9200 0000000000000000
<4> 00000000000022be 0000000000000020 000000001acc0301 00000000000022e0
<4> 00000000000022e0 0000000000000001 0000000000000000 000000001b2b9200
<4> 000000001b2b9200 0000000000444e08 000000001ff5fa38 000000001ff5f9e8
<4>Krnl Code: 0000000000348770: d503d01020cc clc 16(4,%r13),204(%r2)
<4> 0000000000348776: a7840004 brc 8,34877e
<4> 000000000034877a: a7f40001 brc 15,34877c
<4> >000000000034877e: 18b1 lr %r11,%r1
<4> 0000000000348780: 1aba ar %r11,%r10
<4> 0000000000348782: 1ab4 ar %r11,%r4
<4> 0000000000348784: a7ba00ff ahi %r11,255
<4> 0000000000348788: a5b7ff00 nill %r11,65280
<4>Call Trace:
<4>([<070000001fb96000>] 0x70000001fb96000)
<4> [<0000000000348c08>] __pskb_pull_tail+0x2b4/0x38c
<4> [<0000000000352e62>] dev_queue_xmit+0x1a6/0x310
<4> [<0000000000357b98>] neigh_update+0x314/0x524
<4> [<00000000003a11d6>] arp_process+0x2be/0x6f8
<4> [<00000000003a1708>] arp_rcv+0xf8/0x184
<4> [<000000000034f840>] netif_receive_skb+0x244/0x338
<4> [<0000000000352296>] process_backlog+0xc2/0x1a8
<4> [<0000000000352416>] net_rx_action+0x9a/0x154
<4> [<0000000000136ba4>] __do_softirq+0x98/0x12c
<4> [<00000000001106b0>] do_softirq+0xac/0xb0
<4> [<0000000000136d94>] irq_exit+0x8c/0x90
<4> [<00000000002e62dc>] do_IRQ+0x108/0x18c
<4> [<0000000000113f10>] io_return+0x0/0x10
<4> [<000000000010a6f0>] cpu_idle+0x21c/0x23c
<4>([<000000000010a6a4>] cpu_idle+0x1d0/0x23c)
<4> [<00000000001168e6>] start_secondary+0x9e/0xac
<4> [<0000000000000000>] 0x0
<4> [<0000000000000000>] 0x0
<4>
<4> <0>Kernel panic - not syncing: Fatal exception in interrupt
<4>
Following patch fixes the problem but I do not know if it is a good sollution.
From: Frank Blaschka <frank.blaschka@de.ibm.com>
neigh_update sends skb from neigh->arp_queue while
neigh_timer_handler has increased skbs refcount and calls
solicit with the skb. Do not send neighbour skbs
marked for solicit (skb_shared).
Signed-off-by: Frank Blaschka <frank.blaschka@de.ibm.com>
---
net/core/neighbour.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
Index: git_linus/net/core/neighbour.c
===================================================================
--- git_linus.orig/net/core/neighbour.c 2008-01-31 12:59:57.000000000 +0100
+++ git_linus/net/core/neighbour.c 2008-01-31 13:00:25.000000000 +0100
@@ -1060,7 +1060,8 @@
/* On shaper/eql skb->dst->neighbour != neigh :( */
if (skb->dst && skb->dst->neighbour)
n1 = skb->dst->neighbour;
- n1->output(skb);
+ if (!skb_shared(skb))
+ n1->output(skb);
write_lock_bh(&neigh->lock);
}
skb_queue_purge(&neigh->arp_queue);
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH][RFC] race in generic address resolution
2008-02-04 14:27 [PATCH][RFC] race in generic address resolution Blaschka
@ 2008-02-06 4:51 ` David Miller
2008-02-11 9:01 ` Frank Blaschka
0 siblings, 1 reply; 6+ messages in thread
From: David Miller @ 2008-02-06 4:51 UTC (permalink / raw)
To: frank.blaschka; +Cc: netdev
From: Blaschka <frank.blaschka@de.ibm.com>
Date: Mon, 4 Feb 2008 15:27:17 +0100
> I'm running a SMP maschine (2 CPUs) configured as a router. During heavy
> traffic kernel dies with following message:
>
> <2>kernel BUG at /home/autobuild/BUILD/linux-2.6.23-20080125/net/core/skbuff.c:648!
...
> Following patch fixes the problem but I do not know if it is a good sollution.
>
> From: Frank Blaschka <frank.blaschka@de.ibm.com>
>
> neigh_update sends skb from neigh->arp_queue while
> neigh_timer_handler has increased skbs refcount and calls
> solicit with the skb. Do not send neighbour skbs
> marked for solicit (skb_shared).
>
> Signed-off-by: Frank Blaschka <frank.blaschka@de.ibm.com>
Thanks for finding this bug.
I'm fine with your approach as a temporary fix, but there is a slight
problem with your patch. If the skb is shared we have to free it if
we don't pass it on to ->output(), otherwise this creates a leak.
In the longer term, this is an unfortunate limitation. The
->solicit() code just wants to look at a few header fields to
determine how to construct the solicitation request.
What's funny is that we added these skb_get() calls for
the solications exactly to deal with this race condition.
I considered various ways to fix this. The simplest is probably just
to skb_copy() in the ->solicit() case. Solicitation is a rare event
so it's not big deal to copy the packet until the neighbour is
resolved.
The other option is holding the write lock on neigh->lock during the
->solicit() call. I looked at all of the ndisc_ops implementations
and this seems workable. The only case that needs special care is the
IPV4 ARP implementation of arp_solicit(). It wants to take
neigh->lock as a reader to protect the header entry in neigh->ha
during the emission of the soliciation. We can simply remove the read
lock calls to take care of that since holding the lock as a writer at
the caller providers a superset of the protection afforded by the
existing read locking.
The rest of the ->solicit() implementations don't care whether
the neigh is locked or not.
Can you see if this version of the patch fixes your problem?
Thanks!
diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index a16cf1e..7bb6a9a 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -834,18 +834,12 @@ static void neigh_timer_handler(unsigned long arg)
}
if (neigh->nud_state & (NUD_INCOMPLETE | NUD_PROBE)) {
struct sk_buff *skb = skb_peek(&neigh->arp_queue);
- /* keep skb alive even if arp_queue overflows */
- if (skb)
- skb_get(skb);
- write_unlock(&neigh->lock);
+
neigh->ops->solicit(neigh, skb);
atomic_inc(&neigh->probes);
- if (skb)
- kfree_skb(skb);
- } else {
-out:
- write_unlock(&neigh->lock);
}
+out:
+ write_unlock(&neigh->lock);
if (notify)
neigh_update_notify(neigh);
diff --git a/net/ipv4/arp.c b/net/ipv4/arp.c
index 8e17f65..c663fa5 100644
--- a/net/ipv4/arp.c
+++ b/net/ipv4/arp.c
@@ -368,7 +368,6 @@ static void arp_solicit(struct neighbour *neigh, struct sk_buff *skb)
if (!(neigh->nud_state&NUD_VALID))
printk(KERN_DEBUG "trying to ucast probe in NUD_INVALID\n");
dst_ha = neigh->ha;
- read_lock_bh(&neigh->lock);
} else if ((probes -= neigh->parms->app_probes) < 0) {
#ifdef CONFIG_ARPD
neigh_app_ns(neigh);
@@ -378,8 +377,6 @@ static void arp_solicit(struct neighbour *neigh, struct sk_buff *skb)
arp_send(ARPOP_REQUEST, ETH_P_ARP, target, dev, saddr,
dst_ha, dev->dev_addr, NULL);
- if (dst_ha)
- read_unlock_bh(&neigh->lock);
}
static int arp_ignore(struct in_device *in_dev, __be32 sip, __be32 tip)
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH][RFC] race in generic address resolution
2008-02-06 4:51 ` David Miller
@ 2008-02-11 9:01 ` Frank Blaschka
2008-02-12 5:47 ` David Miller
0 siblings, 1 reply; 6+ messages in thread
From: Frank Blaschka @ 2008-02-11 9:01 UTC (permalink / raw)
To: David Miller, netdev
David Miller schrieb:
> From: Blaschka <frank.blaschka@de.ibm.com>
> Date: Mon, 4 Feb 2008 15:27:17 +0100
>
>> I'm running a SMP maschine (2 CPUs) configured as a router. During heavy
>> traffic kernel dies with following message:
>>
>> <2>kernel BUG at /home/autobuild/BUILD/linux-2.6.23-20080125/net/core/skbuff.c:648!
> ...
>> Following patch fixes the problem but I do not know if it is a good sollution.
>>
>> From: Frank Blaschka <frank.blaschka@de.ibm.com>
>>
>> neigh_update sends skb from neigh->arp_queue while
>> neigh_timer_handler has increased skbs refcount and calls
>> solicit with the skb. Do not send neighbour skbs
>> marked for solicit (skb_shared).
>>
>> Signed-off-by: Frank Blaschka <frank.blaschka@de.ibm.com>
>
> Thanks for finding this bug.
>
> I'm fine with your approach as a temporary fix, but there is a slight
> problem with your patch. If the skb is shared we have to free it if
> we don't pass it on to ->output(), otherwise this creates a leak.
>
> In the longer term, this is an unfortunate limitation. The
> ->solicit() code just wants to look at a few header fields to
> determine how to construct the solicitation request.
>
> What's funny is that we added these skb_get() calls for
> the solications exactly to deal with this race condition.
>
> I considered various ways to fix this. The simplest is probably just
> to skb_copy() in the ->solicit() case. Solicitation is a rare event
> so it's not big deal to copy the packet until the neighbour is
> resolved.
>
> The other option is holding the write lock on neigh->lock during the
> ->solicit() call. I looked at all of the ndisc_ops implementations
> and this seems workable. The only case that needs special care is the
> IPV4 ARP implementation of arp_solicit(). It wants to take
> neigh->lock as a reader to protect the header entry in neigh->ha
> during the emission of the soliciation. We can simply remove the read
> lock calls to take care of that since holding the lock as a writer at
> the caller providers a superset of the protection afforded by the
> existing read locking.
>
> The rest of the ->solicit() implementations don't care whether
> the neigh is locked or not.
>
> Can you see if this version of the patch fixes your problem?
>
> Thanks!
>
> diff --git a/net/core/neighbour.c b/net/core/neighbour.c
> index a16cf1e..7bb6a9a 100644
> --- a/net/core/neighbour.c
> +++ b/net/core/neighbour.c
> @@ -834,18 +834,12 @@ static void neigh_timer_handler(unsigned long arg)
> }
> if (neigh->nud_state & (NUD_INCOMPLETE | NUD_PROBE)) {
> struct sk_buff *skb = skb_peek(&neigh->arp_queue);
> - /* keep skb alive even if arp_queue overflows */
> - if (skb)
> - skb_get(skb);
> - write_unlock(&neigh->lock);
> +
> neigh->ops->solicit(neigh, skb);
> atomic_inc(&neigh->probes);
> - if (skb)
> - kfree_skb(skb);
> - } else {
> -out:
> - write_unlock(&neigh->lock);
> }
> +out:
> + write_unlock(&neigh->lock);
>
> if (notify)
> neigh_update_notify(neigh);
> diff --git a/net/ipv4/arp.c b/net/ipv4/arp.c
> index 8e17f65..c663fa5 100644
> --- a/net/ipv4/arp.c
> +++ b/net/ipv4/arp.c
> @@ -368,7 +368,6 @@ static void arp_solicit(struct neighbour *neigh, struct sk_buff *skb)
> if (!(neigh->nud_state&NUD_VALID))
> printk(KERN_DEBUG "trying to ucast probe in NUD_INVALID\n");
> dst_ha = neigh->ha;
> - read_lock_bh(&neigh->lock);
> } else if ((probes -= neigh->parms->app_probes) < 0) {
> #ifdef CONFIG_ARPD
> neigh_app_ns(neigh);
> @@ -378,8 +377,6 @@ static void arp_solicit(struct neighbour *neigh, struct sk_buff *skb)
>
> arp_send(ARPOP_REQUEST, ETH_P_ARP, target, dev, saddr,
> dst_ha, dev->dev_addr, NULL);
> - if (dst_ha)
> - read_unlock_bh(&neigh->lock);
> }
>
> static int arp_ignore(struct in_device *in_dev, __be32 sip, __be32 tip)
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Hi Dave,
we run your patch during the weekend on single CPU and SMP machines. We do not
see any problems. Thanks for providing the fix.
Best regards,
Frank
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH][RFC] race in generic address resolution
2008-02-11 9:01 ` Frank Blaschka
@ 2008-02-12 5:47 ` David Miller
2008-02-14 16:56 ` Benjamin Thery
0 siblings, 1 reply; 6+ messages in thread
From: David Miller @ 2008-02-12 5:47 UTC (permalink / raw)
To: blaschka; +Cc: netdev
From: Frank Blaschka <blaschka@linux.vnet.ibm.com>
Date: Mon, 11 Feb 2008 10:01:20 +0100
> we run your patch during the weekend on single CPU and SMP
> machines. We do not see any problems. Thanks for providing the fix.
Thanks for testing Frank, I can now push this fix upstream.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH][RFC] race in generic address resolution
2008-02-12 5:47 ` David Miller
@ 2008-02-14 16:56 ` Benjamin Thery
2008-02-14 16:59 ` Benjamin Thery
0 siblings, 1 reply; 6+ messages in thread
From: Benjamin Thery @ 2008-02-14 16:56 UTC (permalink / raw)
To: David Miller; +Cc: blaschka, netdev
Hi,
It seems this patch hangs my machine very quickly when there are some
ICMPv6 traffic.
I'm using net-2.6, pulled today (14th Feb).
I had some unexpected hangs on my SMP test machines and I bisected the
problem to 69cc64d8d92bf852f933e90c888dfff083bd4fc9
"[NDISC]: Fix race in generic address resolution".
Looks like a deadlock:
"BUG: soft lockup - CPU#1 stuck for 61s! [swapper:0]"
Here are some traces printed on the console:
Pid: 0, comm: swapper Not tainted (2.6.25-rc1-netns-00113-g69cc64d-dirty #34)
EIP: 0060:[<c02eb5f6>] EFLAGS: 00000287 CPU: 0
EIP is at __write_lock_failed+0xa/0x20
EAX: c7b3fab4 EBX: c7b3fab4 ECX: 00000000 EDX: c0377986
ESI: c7b3fa90 EDI: c7b6f290 EBP: c03cbd24 ESP: c03cbd24
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7f9b404 CR3: 07ac8000 CR4: 00000690
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: 00000000 DR7: 00000000
[<c020e43f>] _raw_write_lock+0x57/0x6c
[<c02eba95>] _write_lock_bh+0x25/0x2d
[<c026b107>] ? neigh_resolve_output+0x93/0x238
[<c026b107>] neigh_resolve_output+0x93/0x238
[<c02a5635>] ip6_output2+0x241/0x289
[<c02a61cd>] ip6_output+0xa92/0xaad
[<c025ff11>] ? __alloc_skb+0x4f/0xfb
[<c02b2596>] ? __ndisc_send+0x1fb/0x3f5
[<c02b26a0>] __ndisc_send+0x305/0x3f5
[<c02b2fb5>] ndisc_send_ns+0x63/0x6e
[<c02b3f3e>] ndisc_solicit+0x183/0x18d
[<c0121071>] ? __mod_timer+0x96/0xa1
[<c026b81e>] neigh_timer_handler+0x214/0x252
[<c0120c90>] run_timer_softirq+0xfe/0x159
[<c026b60a>] ? neigh_timer_handler+0x0/0x252
[<c011dbfa>] __do_softirq+0x6f/0xe9
[<c011dcae>] do_softirq+0x3a/0x52
[<c011dfc3>] irq_exit+0x44/0x46
[<c0105273>] do_IRQ+0x5a/0x73
[<c0103666>] common_interrupt+0x2e/0x34
[<c0101954>] ? default_idle+0x4a/0x77
[<c010190a>] ? default_idle+0x0/0x77
[<c0101855>] cpu_idle+0x89/0x9d
[<c02e6135>] rest_init+0x49/0x4b
=======================
BUG: soft lockup - CPU#1 stuck for 61s! [swapper:0]
Pid: 0, comm: swapper Not tainted (2.6.25-rc1-netns-00113-g69cc64d-dirty #34)
EIP: 0060:[<c02eb5f6>] EFLAGS: 00000287 CPU: 1
EIP is at __write_lock_failed+0xa/0x20
EAX: c7b3fab4 EBX: c7b3fab4 ECX: 00000000 EDX: 00000000
ESI: c03bb9c0 EDI: c7b3fab4 EBP: c7841eb0 ESP: c7841eb0
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 08560008 CR3: 07b04000 CR4: 00000690
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: 00000000 DR7: 00000000
[<c020e43f>] _raw_write_lock+0x57/0x6c
[<c02eba68>] _write_lock+0x20/0x28
[<c026982c>] ? neigh_periodic_timer+0x99/0x142
[<c026982c>] neigh_periodic_timer+0x99/0x142
[<c0120c90>] run_timer_softirq+0xfe/0x159
[<c0269793>] ? neigh_periodic_timer+0x0/0x142
[<c011dbfa>] __do_softirq+0x6f/0xe9
[<c011dcae>] do_softirq+0x3a/0x52
[<c011dfc3>] irq_exit+0x44/0x46
[<c010d680>] smp_apic_timer_interrupt+0x71/0x81
[<c0103747>] apic_timer_interrupt+0x33/0x38
[<c0101954>] ? default_idle+0x4a/0x77
[<c010190a>] ? default_idle+0x0/0x77
[<c0101855>] cpu_idle+0x89/0x9d
=======================
BUG: soft lockup - CPU#0 stuck for 61s! [swapper:0]
Pid: 0, comm: swapper Not tainted (2.6.25-rc1-netns-00113-g69cc64d-dirty #34)
EIP: 0060:[<c02eb5f6>] EFLAGS: 00000287 CPU: 0
EIP is at __write_lock_failed+0xa/0x20
EAX: c7b3fab4 EBX: c7b3fab4 ECX: 00000000 EDX: c0377986
ESI: c7b3fa90 EDI: c7b6f290 EBP: c03cbd24 ESP: c03cbd24
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7f9b404 CR3: 07ac8000 CR4: 00000690
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: 00000000 DR7: 00000000
[<c020e43f>] _raw_write_lock+0x57/0x6c
[<c02eba95>] _write_lock_bh+0x25/0x2d
[<c026b107>] ? neigh_resolve_output+0x93/0x238
[<c026b107>] neigh_resolve_output+0x93/0x238
[<c02a5635>] ip6_output2+0x241/0x289
[<c02a61cd>] ip6_output+0xa92/0xaad
[<c025ff11>] ? __alloc_skb+0x4f/0xfb
[<c02b2596>] ? __ndisc_send+0x1fb/0x3f5
[<c02b26a0>] __ndisc_send+0x305/0x3f5
[<c02b2fb5>] ndisc_send_ns+0x63/0x6e
[<c02b3f3e>] ndisc_solicit+0x183/0x18d
[<c0121071>] ? __mod_timer+0x96/0xa1
[<c026b81e>] neigh_timer_handler+0x214/0x252
[<c0120c90>] run_timer_softirq+0xfe/0x159
[<c026b60a>] ? neigh_timer_handler+0x0/0x252
[<c011dbfa>] __do_softirq+0x6f/0xe9
[<c011dcae>] do_softirq+0x3a/0x52
[<c011dfc3>] irq_exit+0x44/0x46
[<c0105273>] do_IRQ+0x5a/0x73
[<c0103666>] common_interrupt+0x2e/0x34
[<c0101954>] ? default_idle+0x4a/0x77
[<c010190a>] ? default_idle+0x0/0x77
[<c0101855>] cpu_idle+0x89/0x9d
[<c02e6135>] rest_init+0x49/0x4b
=======================
BUG: soft lockup - CPU#1 stuck for 61s! [swapper:0]
...
Benjamin
On Tue, Feb 12, 2008 at 6:47 AM, David Miller <davem@davemloft.net> wrote:
> From: Frank Blaschka <blaschka@linux.vnet.ibm.com>
> Date: Mon, 11 Feb 2008 10:01:20 +0100
>
>
> > we run your patch during the weekend on single CPU and SMP
> > machines. We do not see any problems. Thanks for providing the fix.
>
> Thanks for testing Frank, I can now push this fix upstream.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH][RFC] race in generic address resolution
2008-02-14 16:56 ` Benjamin Thery
@ 2008-02-14 16:59 ` Benjamin Thery
0 siblings, 0 replies; 6+ messages in thread
From: Benjamin Thery @ 2008-02-14 16:59 UTC (permalink / raw)
To: David Miller; +Cc: blaschka, netdev
I ran some additional tests and these traces may also be usefull.
They appears before the soft-lockup are detected.
fermi:~# ping6 -c 500 -f 2007::1
PING 2007::1(2007::1) 56 data bytes
.
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.25-rc1-00113-g69cc64d-dirty #34
-------------------------------------------------------
ping6/1058 is trying to acquire lock:
(&tbl->lock){-+-+}, at: [<c02691ac>] neigh_lookup+0x43/0xa2
but task is already holding lock:
(&n->lock){-+..}, at: [<c026b620>] neigh_timer_handler+0x16/0x252
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&n->lock){-+..}:
[<c01330b8>] __lock_acquire+0x947/0xafc
[<c026982c>] neigh_periodic_timer+0x99/0x142
[<c01332d0>] lock_acquire+0x63/0x80
[<c026982c>] neigh_periodic_timer+0x99/0x142
[<c02eba61>] _write_lock+0x19/0x28
[<c026982c>] neigh_periodic_timer+0x99/0x142
[<c026982c>] neigh_periodic_timer+0x99/0x142
[<c0120c90>] run_timer_softirq+0xfe/0x159
[<c0269793>] neigh_periodic_timer+0x0/0x142
[<c011dbfa>] __do_softirq+0x6f/0xe9
[<c011dcae>] do_softirq+0x3a/0x52
[<c011dfc3>] irq_exit+0x44/0x46
[<c010d680>] smp_apic_timer_interrupt+0x71/0x81
[<c0103747>] apic_timer_interrupt+0x33/0x38
[<c014e0ce>] mmap_region+0xe1/0x376
[<c014e680>] arch_get_unmapped_area_topdown+0x0/0x12e
[<c014e625>] do_mmap_pgoff+0x1e2/0x23d
[<c0181895>] elf_map+0xd8/0x104
[<c0182072>] load_elf_binary+0x5b4/0x11cd
[<c015ed73>] search_binary_handler+0x74/0x164
[<c0181abe>] load_elf_binary+0x0/0x11cd
[<c015ed7a>] search_binary_handler+0x7b/0x164
[<c015ff2e>] do_execve+0x121/0x16a
[<c01012e3>] sys_execve+0x29/0x52
[<c0102c56>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
-> #0 (&tbl->lock){-+-+}:
[<c0132fdf>] __lock_acquire+0x86e/0xafc
[<c01332d0>] lock_acquire+0x63/0x80
[<c02691ac>] neigh_lookup+0x43/0xa2
[<c02ebae9>] _read_lock_bh+0x1e/0x2d
[<c02691ac>] neigh_lookup+0x43/0xa2
[<c02691ac>] neigh_lookup+0x43/0xa2
[<c02af858>] ndisc_dst_alloc+0xb5/0x155
[<c02b240d>] __ndisc_send+0x72/0x3f5
[<c02a573b>] ip6_output+0x0/0xaad
[<c0133225>] __lock_acquire+0xab4/0xafc
[<c02b2fb5>] ndisc_send_ns+0x63/0x6e
[<c02eb92c>] _read_unlock_bh+0x25/0x28
[<c02b3f3e>] ndisc_solicit+0x183/0x18d
[<c0121071>] __mod_timer+0x96/0xa1
[<c026b81e>] neigh_timer_handler+0x214/0x252
[<c0120c90>] run_timer_softirq+0xfe/0x159
[<c026b60a>] neigh_timer_handler+0x0/0x252
[<c011dbfa>] __do_softirq+0x6f/0xe9
[<c011dcae>] do_softirq+0x3a/0x52
[<c011dfc3>] irq_exit+0x44/0x46
[<c0105273>] do_IRQ+0x5a/0x73
[<c0103666>] common_interrupt+0x2e/0x34
[<c02ebd3a>] _spin_unlock_irqrestore+0x38/0x3c
[<c02188fb>] tty_ldisc_deref+0x5c/0x63
[<c021a5bd>] tty_write+0x1a8/0x1b9
[<c021c5e1>] write_chan+0x0/0x2a9
[<c021a633>] redirected_tty_write+0x65/0x72
[<c021a5ce>] redirected_tty_write+0x0/0x72
[<c015be18>] vfs_write+0x8c/0x108
[<c015c3a2>] sys_write+0x3b/0x60
[<c0102c56>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
other info that might help us debug this:
1 lock held by ping6/1058:
#0: (&n->lock){-+..}, at: [<c026b620>] neigh_timer_handler+0x16/0x252
stack backtrace:
Pid: 1058, comm: ping6 Not tainted 2.6.25-rc1-netns-00113-g69cc64d-dirty #34
[<c013176b>] print_circular_bug_tail+0x5b/0x66
[<c0132fdf>] __lock_acquire+0x86e/0xafc
[<c01332d0>] lock_acquire+0x63/0x80
[<c02691ac>] ? neigh_lookup+0x43/0xa2
[<c02ebae9>] _read_lock_bh+0x1e/0x2d
[<c02691ac>] ? neigh_lookup+0x43/0xa2
[<c02691ac>] neigh_lookup+0x43/0xa2
[<c02af858>] ndisc_dst_alloc+0xb5/0x155
[<c02b240d>] __ndisc_send+0x72/0x3f5
[<c02a573b>] ? ip6_output+0x0/0xaad
[<c0133225>] ? __lock_acquire+0xab4/0xafc
[<c02b2fb5>] ndisc_send_ns+0x63/0x6e
[<c02eb92c>] ? _read_unlock_bh+0x25/0x28
[<c02b3f3e>] ndisc_solicit+0x183/0x18d
[<c0121071>] ? __mod_timer+0x96/0xa1
[<c026b81e>] neigh_timer_handler+0x214/0x252
[<c0120c90>] run_timer_softirq+0xfe/0x159
[<c026b60a>] ? neigh_timer_handler+0x0/0x252
[<c011dbfa>] __do_softirq+0x6f/0xe9
[<c011dcae>] do_softirq+0x3a/0x52
[<c011dfc3>] irq_exit+0x44/0x46
[<c0105273>] do_IRQ+0x5a/0x73
[<c0103666>] common_interrupt+0x2e/0x34
[<c02ebd3a>] ? _spin_unlock_irqrestore+0x38/0x3c
[<c02188fb>] tty_ldisc_deref+0x5c/0x63
[<c021a5bd>] tty_write+0x1a8/0x1b9
[<c021c5e1>] ? write_chan+0x0/0x2a9
[<c021a633>] redirected_tty_write+0x65/0x72
[<c021a5ce>] ? redirected_tty_write+0x0/0x72
[<c015be18>] vfs_write+0x8c/0x108
[<c015c3a2>] sys_write+0x3b/0x60
[<c0102c56>] syscall_call+0x7/0xb
=======================
On Thu, Feb 14, 2008 at 5:56 PM, Benjamin Thery <ben.thery@gmail.com> wrote:
> Hi,
>
> It seems this patch hangs my machine very quickly when there are some
> ICMPv6 traffic.
>
> I'm using net-2.6, pulled today (14th Feb).
>
> I had some unexpected hangs on my SMP test machines and I bisected the
> problem to 69cc64d8d92bf852f933e90c888dfff083bd4fc9
> "[NDISC]: Fix race in generic address resolution".
>
> Looks like a deadlock:
> "BUG: soft lockup - CPU#1 stuck for 61s! [swapper:0]"
>
> Here are some traces printed on the console:
>
> Pid: 0, comm: swapper Not tainted (2.6.25-rc1-netns-00113-g69cc64d-dirty #34)
> EIP: 0060:[<c02eb5f6>] EFLAGS: 00000287 CPU: 0
> EIP is at __write_lock_failed+0xa/0x20
> EAX: c7b3fab4 EBX: c7b3fab4 ECX: 00000000 EDX: c0377986
> ESI: c7b3fa90 EDI: c7b6f290 EBP: c03cbd24 ESP: c03cbd24
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: b7f9b404 CR3: 07ac8000 CR4: 00000690
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: 00000000 DR7: 00000000
> [<c020e43f>] _raw_write_lock+0x57/0x6c
> [<c02eba95>] _write_lock_bh+0x25/0x2d
> [<c026b107>] ? neigh_resolve_output+0x93/0x238
> [<c026b107>] neigh_resolve_output+0x93/0x238
> [<c02a5635>] ip6_output2+0x241/0x289
> [<c02a61cd>] ip6_output+0xa92/0xaad
> [<c025ff11>] ? __alloc_skb+0x4f/0xfb
> [<c02b2596>] ? __ndisc_send+0x1fb/0x3f5
> [<c02b26a0>] __ndisc_send+0x305/0x3f5
> [<c02b2fb5>] ndisc_send_ns+0x63/0x6e
> [<c02b3f3e>] ndisc_solicit+0x183/0x18d
> [<c0121071>] ? __mod_timer+0x96/0xa1
> [<c026b81e>] neigh_timer_handler+0x214/0x252
> [<c0120c90>] run_timer_softirq+0xfe/0x159
> [<c026b60a>] ? neigh_timer_handler+0x0/0x252
> [<c011dbfa>] __do_softirq+0x6f/0xe9
> [<c011dcae>] do_softirq+0x3a/0x52
> [<c011dfc3>] irq_exit+0x44/0x46
> [<c0105273>] do_IRQ+0x5a/0x73
> [<c0103666>] common_interrupt+0x2e/0x34
> [<c0101954>] ? default_idle+0x4a/0x77
> [<c010190a>] ? default_idle+0x0/0x77
> [<c0101855>] cpu_idle+0x89/0x9d
> [<c02e6135>] rest_init+0x49/0x4b
> =======================
> BUG: soft lockup - CPU#1 stuck for 61s! [swapper:0]
>
> Pid: 0, comm: swapper Not tainted (2.6.25-rc1-netns-00113-g69cc64d-dirty #34)
> EIP: 0060:[<c02eb5f6>] EFLAGS: 00000287 CPU: 1
> EIP is at __write_lock_failed+0xa/0x20
> EAX: c7b3fab4 EBX: c7b3fab4 ECX: 00000000 EDX: 00000000
> ESI: c03bb9c0 EDI: c7b3fab4 EBP: c7841eb0 ESP: c7841eb0
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: 08560008 CR3: 07b04000 CR4: 00000690
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: 00000000 DR7: 00000000
> [<c020e43f>] _raw_write_lock+0x57/0x6c
> [<c02eba68>] _write_lock+0x20/0x28
> [<c026982c>] ? neigh_periodic_timer+0x99/0x142
> [<c026982c>] neigh_periodic_timer+0x99/0x142
> [<c0120c90>] run_timer_softirq+0xfe/0x159
> [<c0269793>] ? neigh_periodic_timer+0x0/0x142
> [<c011dbfa>] __do_softirq+0x6f/0xe9
> [<c011dcae>] do_softirq+0x3a/0x52
> [<c011dfc3>] irq_exit+0x44/0x46
> [<c010d680>] smp_apic_timer_interrupt+0x71/0x81
> [<c0103747>] apic_timer_interrupt+0x33/0x38
> [<c0101954>] ? default_idle+0x4a/0x77
> [<c010190a>] ? default_idle+0x0/0x77
> [<c0101855>] cpu_idle+0x89/0x9d
> =======================
> BUG: soft lockup - CPU#0 stuck for 61s! [swapper:0]
>
> Pid: 0, comm: swapper Not tainted (2.6.25-rc1-netns-00113-g69cc64d-dirty #34)
> EIP: 0060:[<c02eb5f6>] EFLAGS: 00000287 CPU: 0
> EIP is at __write_lock_failed+0xa/0x20
> EAX: c7b3fab4 EBX: c7b3fab4 ECX: 00000000 EDX: c0377986
> ESI: c7b3fa90 EDI: c7b6f290 EBP: c03cbd24 ESP: c03cbd24
> DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> CR0: 8005003b CR2: b7f9b404 CR3: 07ac8000 CR4: 00000690
> DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> DR6: 00000000 DR7: 00000000
> [<c020e43f>] _raw_write_lock+0x57/0x6c
> [<c02eba95>] _write_lock_bh+0x25/0x2d
> [<c026b107>] ? neigh_resolve_output+0x93/0x238
> [<c026b107>] neigh_resolve_output+0x93/0x238
> [<c02a5635>] ip6_output2+0x241/0x289
> [<c02a61cd>] ip6_output+0xa92/0xaad
> [<c025ff11>] ? __alloc_skb+0x4f/0xfb
> [<c02b2596>] ? __ndisc_send+0x1fb/0x3f5
> [<c02b26a0>] __ndisc_send+0x305/0x3f5
> [<c02b2fb5>] ndisc_send_ns+0x63/0x6e
> [<c02b3f3e>] ndisc_solicit+0x183/0x18d
> [<c0121071>] ? __mod_timer+0x96/0xa1
> [<c026b81e>] neigh_timer_handler+0x214/0x252
> [<c0120c90>] run_timer_softirq+0xfe/0x159
> [<c026b60a>] ? neigh_timer_handler+0x0/0x252
> [<c011dbfa>] __do_softirq+0x6f/0xe9
> [<c011dcae>] do_softirq+0x3a/0x52
> [<c011dfc3>] irq_exit+0x44/0x46
> [<c0105273>] do_IRQ+0x5a/0x73
> [<c0103666>] common_interrupt+0x2e/0x34
> [<c0101954>] ? default_idle+0x4a/0x77
> [<c010190a>] ? default_idle+0x0/0x77
> [<c0101855>] cpu_idle+0x89/0x9d
> [<c02e6135>] rest_init+0x49/0x4b
> =======================
> BUG: soft lockup - CPU#1 stuck for 61s! [swapper:0]
> ...
>
>
> Benjamin
>
>
>
> On Tue, Feb 12, 2008 at 6:47 AM, David Miller <davem@davemloft.net> wrote:
> > From: Frank Blaschka <blaschka@linux.vnet.ibm.com>
> > Date: Mon, 11 Feb 2008 10:01:20 +0100
> >
> >
> > > we run your patch during the weekend on single CPU and SMP
> > > machines. We do not see any problems. Thanks for providing the fix.
> >
> > Thanks for testing Frank, I can now push this fix upstream.
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2008-02-14 16:59 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-02-04 14:27 [PATCH][RFC] race in generic address resolution Blaschka
2008-02-06 4:51 ` David Miller
2008-02-11 9:01 ` Frank Blaschka
2008-02-12 5:47 ` David Miller
2008-02-14 16:56 ` Benjamin Thery
2008-02-14 16:59 ` Benjamin Thery
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).