From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sasha Levin Subject: ipv6: tunnel: hang when destroying ipv6 tunnel Date: Sat, 31 Mar 2012 19:51:19 +0200 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Cc: netdev@vger.kernel.org, "linux-kernel@vger.kernel.org List" , Dave Jones To: davem@davemloft.net, kuznet@ms2.inr.ac.ru, jmorris@namei.org, yoshfuji@linux-ipv6.org, Patrick McHardy Return-path: Sender: linux-kernel-owner@vger.kernel.org List-Id: netdev.vger.kernel.org Hi all, It appears that a hang may occur when destroying an ipv6 tunnel, which I've reproduced several times in a KVM vm. The pattern in the stack dump below is consistent with unregistering a kobject when holding multiple locks. Unregistering a kobject usually leads to an exit back to userspace with call_usermodehelper_exec(). The userspace code may access sysfs files which in turn will require locking within the kernel, leading to a deadlock since those locks are already held by kernel. [ 1561.564172] INFO: task kworker/u:2:3140 blocked for more than 120 seconds. [ 1561.566945] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1561.570062] kworker/u:2 D ffff88006ee63000 4504 3140 2 0x00000000 [ 1561.572968] ffff88006ed9f7e0 0000000000000082 ffff88006ed9f790 ffffffff8107d346 [ 1561.575680] ffff88006ed9ffd8 00000000001d4580 ffff88006ed9e010 00000000001d4580 [ 1561.578601] 00000000001d4580 00000000001d4580 ffff88006ed9ffd8 00000000001d4580 [ 1561.581697] Call Trace: [ 1561.582650] [] ? kvm_clock_read+0x46/0x80 [ 1561.584543] [] schedule+0x24/0x70 [ 1561.586231] [] schedule_timeout+0x245/0x2c0 [ 1561.588508] [] ? mark_held_locks+0x7a/0x120 [ 1561.590858] [] ? __lock_release+0x8d/0x1d0 [ 1561.593162] [] ? _raw_spin_unlock_irq+0x2b/0x70 [ 1561.595394] [] ? get_parent_ip+0x11/0x50 [ 1561.597403] [] wait_for_common+0x119/0x190 [ 1561.599707] [] ? try_to_wake_up+0x2c0/0x2c0 [ 1561.601758] [] wait_for_completion+0x18/0x20 [ 1561.603843] [] call_usermodehelper_exec+0x228/0x240 [ 1561.606059] [] ? wait_for_common+0x44/0x190 [ 1561.608352] [] kobject_uevent_env+0x615/0x650 [ 1561.610908] [] ? get_parent_ip+0x11/0x50 [ 1561.613146] [] kobject_uevent+0xb/0x10 [ 1561.615312] [] kobject_cleanup+0xca/0x1b0 [ 1561.617509] [] kobject_release+0xd/0x10 [ 1561.619334] [] kobject_put+0x2c/0x60 [ 1561.621117] [] net_rx_queue_update_kobjects+0xa0/0xf0 [ 1561.623421] [] netdev_unregister_kobject+0x37/0x70 [ 1561.625979] [] rollback_registered_many+0x186/0x260 [ 1561.628526] [] unregister_netdevice_many+0x14/0x60 [ 1561.631064] [] ip6_tnl_destroy_tunnels+0xee/0x160 [ 1561.633549] [] ip6_tnl_exit_net+0xd3/0x1c0 [ 1561.635843] [] ? ip6_tnl_ioctl+0x550/0x550 [ 1561.637972] [] ? proc_net_remove+0x16/0x20 [ 1561.639881] [] ops_exit_list+0x39/0x60 [ 1561.641666] [] cleanup_net+0xfb/0x1a0 [ 1561.643528] [] process_one_work+0x1cd/0x460 [ 1561.645828] [] ? process_one_work+0x16c/0x460 [ 1561.648180] [] ? net_drop_ns+0x40/0x40 [ 1561.650285] [] worker_thread+0x176/0x3b0 [ 1561.652460] [] ? manage_workers+0x120/0x120 [ 1561.654734] [] kthread+0xbe/0xd0 [ 1561.656656] [] kernel_thread_helper+0x4/0x10 [ 1561.658881] [] ? finish_task_switch+0x80/0x110 [ 1561.660828] [] ? retint_restore_args+0x13/0x13 [ 1561.662795] [] ? __init_kthread_worker+0x70/0x70 [ 1561.664932] [] ? gs_change+0x13/0x13 [ 1561.667001] 4 locks held by kworker/u:2/3140: [ 1561.667599] #0: (netns){.+.+.+}, at: [] process_one_work+0x16c/0x460 [ 1561.668758] #1: (net_cleanup_work){+.+.+.}, at: [] process_one_work+0x16c/0x460 [ 1561.670002] #2: (net_mutex){+.+.+.}, at: [] cleanup_net+0x80/0x1a0 [ 1561.671700] #3: (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x12/0x20