From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: Hung task when calling clone() due to netfilter/slab Date: Sat, 14 Jan 2012 18:10:43 +0100 Message-ID: <1326561043.5287.24.camel@edumazet-laptop> References: <1326558605.19951.7.camel@lappy> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Cc: Dave Jones , davem , Pekka Enberg , Christoph Lameter , Matt Mackall , kaber@trash.net, pablo@netfilter.org, linux-kernel , linux-mm , netfilter-devel@vger.kernel.org, netdev To: Sasha Levin Return-path: In-Reply-To: <1326558605.19951.7.camel@lappy> Sender: owner-linux-mm@kvack.org List-Id: netdev.vger.kernel.org Le samedi 14 janvier 2012 =C3=A0 18:30 +0200, Sasha Levin a =C3=A9crit : > Hi All, >=20 > I've stumbled on the following oops when testing the trinity fuzzer usi= ng KVM tool, running on the latest -next kernel: >=20 > [ 3960.845373] INFO: task trinity:31661 blocked for more than 120 secon= ds. > [ 3960.846757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disab= les this message. > [ 3960.850245] trinity D ffff880008e89610 4952 31661 2197 0x= 00000004 > [ 3960.855032] ffff880008e39830 0000000000000086 ffff880008e39fd8 ffff= 880008e89610 > [ 3960.859289] 00000000001d3980 ffff880008e39fd8 ffff880008e38000 0000= 0000001d3980 > [ 3960.862723] 00000000001d3980 00000000001d3980 ffff880008e39fd8 0000= 0000001d3980 > [ 3960.866083] Call Trace: > [ 3960.867332] [] schedule+0x3a/0x50 > [ 3960.869477] [] schedule_timeout+0x245/0x2c0 > [ 3960.871996] [] ? mark_held_locks+0x6e/0x130 > [ 3960.875599] [] ? lock_release_holdtime+0xb2/0x160 > [ 3960.879156] [] ? _raw_spin_unlock_irq+0x2b/0x70 > [ 3960.881682] [] ? get_parent_ip+0x11/0x50 > [ 3960.883907] [] wait_for_common+0x120/0x170 > [ 3960.886190] [] ? try_to_wake_up+0x320/0x320 > [ 3960.888570] [] wait_for_completion+0x18/0x20 > [ 3960.890968] [] call_usermodehelper_exec+0x1b5/0x1= d0 > [ 3960.893583] [] ? wait_for_common+0x44/0x170 > [ 3960.895907] [] kobject_uevent_env+0x4e8/0x580 > [ 3960.898310] [] kobject_uevent+0xb/0x10 > [ 3960.900577] [] sysfs_slab_add+0xb1/0x210 > [ 3960.902830] [] kmem_cache_create+0xcb/0x2f0 > [ 3960.905704] [] nf_conntrack_init+0x9d/0x380 > [ 3960.908797] [] nf_conntrack_net_init+0xf/0x1a0 > [ 3960.911065] [] ops_init+0x42/0x180 > [ 3960.913192] [] setup_net+0x6b/0x100 > [ 3960.915308] [] copy_net_ns+0x86/0x110 > [ 3960.917534] [] create_new_namespaces+0xd9/0x190 > [ 3960.920048] [] copy_namespaces+0x84/0xc0 > [ 3960.922326] [] copy_process+0xa21/0x1480 > [ 3960.924328] [] ? up_read+0x1e/0x40 > [ 3960.925327] [] do_fork+0x73/0x340 > [ 3960.926273] [] ? trace_hardirqs_on+0xd/0x10 > [ 3960.927372] [] ? mutex_unlock+0x9/0x10 > [ 3960.928422] [] ? ext4_sync_file+0xa3/0x340 > [ 3960.929487] [] ? retint_swapgs+0x13/0x1b > [ 3960.930520] [] sys_clone+0x23/0x30 > [ 3960.931475] [] stub_clone+0x13/0x20 > [ 3960.932490] [] ? system_call_fastpath+0x16/0x1b > [ 3960.933655] 2 locks held by trinity/31661: > [ 3960.934413] #0: (net_mutex){+.+.+.}, at: [] copy= _net_ns+0x7e/0x110 > [ 3960.936057] #1: (slub_lock){+.+.+.}, at: [] kmem= _cache_create+0x41/0x2f0 > [ 3960.937810] Kernel panic - not syncing: hung_task: blocked tasks >=20 > I had two trinity processes running, and both were stuck at clone() sys= calls: >=20 > clone(clone_flags=3D0xcc2820ff, newsp=3D0xf3f270[page_0xff], parent_tid= =3D0xf3f270[page_0xff], child_tid=3D0x400000, regs=3D0xf41290[page_allocs= ]) > clone(clone_flags=3D0x45706000, newsp=3D0xf3f270[page_0xff], parent_tid= =3D0x7f9bf26f0000, child_tid=3D0xf3f270[page_0xff], regs=3D0xf41290[page_= allocs]) >=20 > This is the second time I got this oops, and both times it originated w= ith clone calling up to the netfilter connection tracker code, so I don't= think that it's coincidental that thats the origin. >=20 Apparently SLUB calls sysfs_slab_add() from kmem_cache_create() while still holding slub_lock. So if the task launched needs to "cat /proc/slabinfo" or anything needing slub_lock, its a deadlock. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org