* Re: possible deadlock in flush_work (2) @ 2017-11-05 8:53 ` Dmitry Vyukov 0 siblings, 0 replies; 6+ messages in thread From: Dmitry Vyukov @ 2017-11-05 8:53 UTC (permalink / raw) To: syzbot Cc: Lai Jiangshan, LKML, syzkaller-bugs, Tejun Heo, Bruce Fields, Jeff Layton, trond.myklebust, anna.schumaker, David Miller, linux-nfs, netdev On Sun, Nov 5, 2017 at 11:41 AM, syzbot <bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.com> wrote: > Hello, > > syzkaller hit the following crash on > 0f611fb6dcc0d6d91b4e1fec911321f434a3b858 > git://git.cmpxchg.org/linux-mmots.git/master > compiler: gcc (GCC) 7.1.1 20170620 > .config is attached > Raw console output is attached. > > xs_tcp_setup_socket: connect returned unhandled error -113 > xs_tcp_setup_socket: connect returned unhandled error -113 > xs_tcp_setup_socket: connect returned unhandled error -113 > > ====================================================== > WARNING: possible circular locking dependency detected > 4.14.0-rc5-mm1+ #20 Not tainted > ------------------------------------------------------ > kworker/0:3/3400 is trying to acquire lock: > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] start_flush_work > kernel/workqueue.c:2850 [inline] > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] flush_work+0x55a/0x8a0 > kernel/workqueue.c:2882 > > but task is already holding lock: > ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 > > which lock already depends on the new lock. > > > the existing dependency chain (in reverse order) is: > > -> #1 ((&task->u.tk_work)){+.+.}: > process_one_work+0xba2/0x1bc0 kernel/workqueue.c:2088 > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > kthread+0x38b/0x470 kernel/kthread.c:242 > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > -> #0 ("xprtiod"){+.+.}: > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 > start_flush_work kernel/workqueue.c:2851 [inline] > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] > kref_put include/linux/kref.h:70 [inline] > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 > rpc_release_task net/sunrpc/sched.c:1059 [inline] > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > kthread+0x38b/0x470 kernel/kthread.c:242 > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > other info that might help us debug this: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock((&task->u.tk_work)); > lock("xprtiod"); > lock((&task->u.tk_work)); > lock("xprtiod"); > > *** DEADLOCK *** > > 2 locks held by kworker/0:3/3400: > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] __write_once_size > include/linux/compiler.h:305 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic64_set > arch/x86/include/asm/atomic64_64.h:33 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic_long_set > include/asm-generic/atomic-long.h:56 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] set_work_data > kernel/workqueue.c:618 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] > set_work_pool_and_clear_pending kernel/workqueue.c:645 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] > process_one_work+0xadf/0x1bc0 kernel/workqueue.c:2083 > #1: ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 > > stack backtrace: > CPU: 0 PID: 3400 Comm: kworker/0:3 Not tainted 4.14.0-rc5-mm1+ #20 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > Workqueue: rpciod rpc_async_schedule > Call Trace: > __dump_stack lib/dump_stack.c:16 [inline] > dump_stack+0x194/0x257 lib/dump_stack.c:52 > print_circular_bug.isra.41+0x342/0x36a kernel/locking/lockdep.c:1258 > check_prev_add kernel/locking/lockdep.c:1901 [inline] > check_prevs_add kernel/locking/lockdep.c:2018 [inline] > validate_chain kernel/locking/lockdep.c:2460 [inline] > __lock_acquire+0x2f55/0x3d50 kernel/locking/lockdep.c:3487 > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 > start_flush_work kernel/workqueue.c:2851 [inline] > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] > kref_put include/linux/kref.h:70 [inline] > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 > rpc_release_task net/sunrpc/sched.c:1059 [inline] > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > kthread+0x38b/0x470 kernel/kthread.c:242 > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 +sunrpc maintainers > --- > This bug is generated by a dumb bot. It may contain errors. > See https://goo.gl/tpsmEJ for details. > Direct all questions to syzkaller@googlegroups.com. > Please credit me with: Reported-by: syzbot <syzkaller@googlegroups.com> > > syzbot will keep track of this bug report. > Once a fix for this bug is committed, please reply to this email with: > #syz fix: exact-commit-title > To mark this as a duplicate of another syzbot report, please reply with: > #syz dup: exact-subject-of-another-report > If it's a one-off invalid bug report, please reply with: > #syz invalid > Note: if the crash happens again, it will cause creation of a new bug > report. > Note: all commands must start from beginning of the line. > > -- > You received this message because you are subscribed to the Google Groups > "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to syzkaller-bugs+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/syzkaller-bugs/001a113ee9baf95598055d384ecb%40google.com. > For more options, visit https://groups.google.com/d/optout. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in flush_work (2) @ 2017-11-05 8:53 ` Dmitry Vyukov 0 siblings, 0 replies; 6+ messages in thread From: Dmitry Vyukov @ 2017-11-05 8:53 UTC (permalink / raw) To: syzbot Cc: Lai Jiangshan, LKML, syzkaller-bugs-/JYPxA39Uh5TLH3MbocFFw, Tejun Heo, Bruce Fields, Jeff Layton, trond.myklebust-7I+n7zu2hftEKMMhf/gKZA, anna.schumaker-HgOvQuBEEgTQT0dZR+AlfA, David Miller, linux-nfs-u79uwXL29TY76Z2rM5mHXA, netdev On Sun, Nov 5, 2017 at 11:41 AM, syzbot <bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4-Pl5Pbv+GP7P466ipTTIvnc23WoclnBCfAL8bYrjMMd8@public.gmane.org> wrote: > Hello, > > syzkaller hit the following crash on > 0f611fb6dcc0d6d91b4e1fec911321f434a3b858 > git://git.cmpxchg.org/linux-mmots.git/master > compiler: gcc (GCC) 7.1.1 20170620 > .config is attached > Raw console output is attached. > > xs_tcp_setup_socket: connect returned unhandled error -113 > xs_tcp_setup_socket: connect returned unhandled error -113 > xs_tcp_setup_socket: connect returned unhandled error -113 > > ====================================================== > WARNING: possible circular locking dependency detected > 4.14.0-rc5-mm1+ #20 Not tainted > ------------------------------------------------------ > kworker/0:3/3400 is trying to acquire lock: > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] start_flush_work > kernel/workqueue.c:2850 [inline] > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] flush_work+0x55a/0x8a0 > kernel/workqueue.c:2882 > > but task is already holding lock: > ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 > > which lock already depends on the new lock. > > > the existing dependency chain (in reverse order) is: > > -> #1 ((&task->u.tk_work)){+.+.}: > process_one_work+0xba2/0x1bc0 kernel/workqueue.c:2088 > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > kthread+0x38b/0x470 kernel/kthread.c:242 > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > -> #0 ("xprtiod"){+.+.}: > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 > start_flush_work kernel/workqueue.c:2851 [inline] > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] > kref_put include/linux/kref.h:70 [inline] > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 > rpc_release_task net/sunrpc/sched.c:1059 [inline] > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > kthread+0x38b/0x470 kernel/kthread.c:242 > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > other info that might help us debug this: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock((&task->u.tk_work)); > lock("xprtiod"); > lock((&task->u.tk_work)); > lock("xprtiod"); > > *** DEADLOCK *** > > 2 locks held by kworker/0:3/3400: > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] __write_once_size > include/linux/compiler.h:305 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic64_set > arch/x86/include/asm/atomic64_64.h:33 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic_long_set > include/asm-generic/atomic-long.h:56 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] set_work_data > kernel/workqueue.c:618 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] > set_work_pool_and_clear_pending kernel/workqueue.c:645 [inline] > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] > process_one_work+0xadf/0x1bc0 kernel/workqueue.c:2083 > #1: ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 > > stack backtrace: > CPU: 0 PID: 3400 Comm: kworker/0:3 Not tainted 4.14.0-rc5-mm1+ #20 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > Workqueue: rpciod rpc_async_schedule > Call Trace: > __dump_stack lib/dump_stack.c:16 [inline] > dump_stack+0x194/0x257 lib/dump_stack.c:52 > print_circular_bug.isra.41+0x342/0x36a kernel/locking/lockdep.c:1258 > check_prev_add kernel/locking/lockdep.c:1901 [inline] > check_prevs_add kernel/locking/lockdep.c:2018 [inline] > validate_chain kernel/locking/lockdep.c:2460 [inline] > __lock_acquire+0x2f55/0x3d50 kernel/locking/lockdep.c:3487 > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 > start_flush_work kernel/workqueue.c:2851 [inline] > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] > kref_put include/linux/kref.h:70 [inline] > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 > rpc_release_task net/sunrpc/sched.c:1059 [inline] > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > kthread+0x38b/0x470 kernel/kthread.c:242 > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 +sunrpc maintainers > --- > This bug is generated by a dumb bot. It may contain errors. > See https://goo.gl/tpsmEJ for details. > Direct all questions to syzkaller-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > Please credit me with: Reported-by: syzbot <syzkaller-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> > > syzbot will keep track of this bug report. > Once a fix for this bug is committed, please reply to this email with: > #syz fix: exact-commit-title > To mark this as a duplicate of another syzbot report, please reply with: > #syz dup: exact-subject-of-another-report > If it's a one-off invalid bug report, please reply with: > #syz invalid > Note: if the crash happens again, it will cause creation of a new bug > report. > Note: all commands must start from beginning of the line. > > -- > You received this message because you are subscribed to the Google Groups > "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to syzkaller-bugs+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To view this discussion on the web visit > https://groups.google.com/d/msgid/syzkaller-bugs/001a113ee9baf95598055d384ecb%40google.com. > For more options, visit https://groups.google.com/d/optout. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in flush_work (2) 2017-11-05 8:53 ` Dmitry Vyukov @ 2017-11-05 16:00 ` Trond Myklebust -1 siblings, 0 replies; 6+ messages in thread From: Trond Myklebust @ 2017-11-05 16:00 UTC (permalink / raw) To: dvyukov@google.com, bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.com Cc: linux-kernel@vger.kernel.org, bfields@fieldses.org, linux-nfs@vger.kernel.org, jlayton@poochiereds.net, jiangshanlai@gmail.com, anna.schumaker@netapp.com, netdev@vger.kernel.org, syzkaller-bugs@googlegroups.com, tj@kernel.org, davem@davemloft.net DQoNCk9uIFN1biwgMjAxNy0xMS0wNSBhdCAxMTo1MyArMDMwMCwgRG1pdHJ5IFZ5dWtvdiB3cm90 ZToNCj4gT24gU3VuLCBOb3YgNSwgMjAxNyBhdCAxMTo0MSBBTSwgc3l6Ym90DQo+IDxib3QrZDhm ZTk1Mjk4ZWY4MzBjZDdkMDVlMzNlZWZhNGE1YTZmNmYzMzRkNEBzeXprYWxsZXIuYXBwc3BvdG1h aWwuYw0KPiBvbT4NCj4gd3JvdGU6DQo+ID4gSGVsbG8sDQo+ID4gDQo+ID4gc3l6a2FsbGVyIGhp dCB0aGUgZm9sbG93aW5nIGNyYXNoIG9uDQo+ID4gMGY2MTFmYjZkY2MwZDZkOTFiNGUxZmVjOTEx MzIxZjQzNGEzYjg1OA0KPiA+IGdpdDovL2dpdC5jbXB4Y2hnLm9yZy9saW51eC1tbW90cy5naXQv bWFzdGVyDQo+ID4gY29tcGlsZXI6IGdjYyAoR0NDKSA3LjEuMSAyMDE3MDYyMA0KPiA+IC5jb25m aWcgaXMgYXR0YWNoZWQNCj4gPiBSYXcgY29uc29sZSBvdXRwdXQgaXMgYXR0YWNoZWQuDQo+ID4g DQo+ID4geHNfdGNwX3NldHVwX3NvY2tldDogY29ubmVjdCByZXR1cm5lZCB1bmhhbmRsZWQgZXJy b3IgLTExMw0KPiA+IHhzX3RjcF9zZXR1cF9zb2NrZXQ6IGNvbm5lY3QgcmV0dXJuZWQgdW5oYW5k bGVkIGVycm9yIC0xMTMNCj4gPiB4c190Y3Bfc2V0dXBfc29ja2V0OiBjb25uZWN0IHJldHVybmVk IHVuaGFuZGxlZCBlcnJvciAtMTEzDQo+ID4gDQo+ID4gPT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+ID4gV0FSTklORzogcG9zc2libGUgY2ly Y3VsYXIgbG9ja2luZyBkZXBlbmRlbmN5IGRldGVjdGVkDQo+ID4gNC4xNC4wLXJjNS1tbTErICMy MCBOb3QgdGFpbnRlZA0KPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLQ0KPiA+IGt3b3JrZXIvMDozLzM0MDAgaXMgdHJ5aW5nIHRvIGFjcXVp cmUgbG9jazoNCj4gPiAgKCJ4cHJ0aW9kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0NmFkZGE+ XSBzdGFydF9mbHVzaF93b3JrDQo+ID4ga2VybmVsL3dvcmtxdWV1ZS5jOjI4NTAgW2lubGluZV0N Cj4gPiAgKCJ4cHJ0aW9kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0NmFkZGE+XSBmbHVzaF93 b3JrKzB4NTVhLzB4OGEwDQo+ID4ga2VybmVsL3dvcmtxdWV1ZS5jOjI4ODINCj4gPiANCj4gPiBi dXQgdGFzayBpcyBhbHJlYWR5IGhvbGRpbmcgbG9jazoNCj4gPiAgKCgmdGFzay0+dS50a193b3Jr KSl7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0NzFlYjI+XQ0KPiA+IHByb2Nlc3Nfb25lX3dvcmsr MHhiMzIvMHgxYmMwIGtlcm5lbC93b3JrcXVldWUuYzoyMDg3DQo+ID4gDQo+ID4gd2hpY2ggbG9j ayBhbHJlYWR5IGRlcGVuZHMgb24gdGhlIG5ldyBsb2NrLg0KPiA+IA0KPiA+IA0KPiA+IHRoZSBl eGlzdGluZyBkZXBlbmRlbmN5IGNoYWluIChpbiByZXZlcnNlIG9yZGVyKSBpczoNCj4gPiANCj4g PiAtPiAjMSAoKCZ0YXNrLT51LnRrX3dvcmspKXsrLisufToNCj4gPiAgICAgICAgcHJvY2Vzc19v bmVfd29yaysweGJhMi8weDFiYzAga2VybmVsL3dvcmtxdWV1ZS5jOjIwODgNCj4gPiAgICAgICAg d29ya2VyX3RocmVhZCsweDIyMy8weDE5OTAga2VybmVsL3dvcmtxdWV1ZS5jOjIyNDYNCj4gPiAg ICAgICAga3RocmVhZCsweDM4Yi8weDQ3MCBrZXJuZWwva3RocmVhZC5jOjI0Mg0KPiA+ICAgICAg ICByZXRfZnJvbV9mb3JrKzB4MmEvMHg0MCBhcmNoL3g4Ni9lbnRyeS9lbnRyeV82NC5TOjQzMQ0K PiA+IA0KPiA+IC0+ICMwICgieHBydGlvZCIpeysuKy59Og0KPiA+ICAgICAgICBsb2NrX2FjcXVp cmUrMHgxZDUvMHg1ODAga2VybmVsL2xvY2tpbmcvbG9ja2RlcC5jOjM5OTENCj4gPiAgICAgICAg c3RhcnRfZmx1c2hfd29yayBrZXJuZWwvd29ya3F1ZXVlLmM6Mjg1MSBbaW5saW5lXQ0KPiA+ICAg ICAgICBmbHVzaF93b3JrKzB4NTdmLzB4OGEwIGtlcm5lbC93b3JrcXVldWUuYzoyODgyDQo+ID4g ICAgICAgIF9fY2FuY2VsX3dvcmtfdGltZXIrMHgzMGEvMHg3ZTAga2VybmVsL3dvcmtxdWV1ZS5j OjI5NTQNCj4gPiAgICAgICAgY2FuY2VsX3dvcmtfc3luYysweDE3LzB4MjAga2VybmVsL3dvcmtx dWV1ZS5jOjI5OTANCj4gPiAgICAgICAgeHBydF9kZXN0cm95KzB4YTEvMHgxMzAgbmV0L3N1bnJw Yy94cHJ0LmM6MTQ2Nw0KPiA+ICAgICAgICB4cHJ0X2Rlc3Ryb3lfa3JlZiBuZXQvc3VucnBjL3hw cnQuYzoxNDc3IFtpbmxpbmVdDQo+ID4gICAgICAgIGtyZWZfcHV0IGluY2x1ZGUvbGludXgva3Jl Zi5oOjcwIFtpbmxpbmVdDQo+ID4gICAgICAgIHhwcnRfcHV0KzB4MzgvMHg0MCBuZXQvc3VucnBj L3hwcnQuYzoxNTAxDQo+ID4gICAgICAgIHJwY190YXNrX3JlbGVhc2VfY2xpZW50KzB4Mjk5LzB4 NDMwIG5ldC9zdW5ycGMvY2xudC5jOjk4Ng0KPiA+ICAgICAgICBycGNfcmVsZWFzZV9yZXNvdXJj ZXNfdGFzaysweDdmLzB4YTAgbmV0L3N1bnJwYy9zY2hlZC5jOjEwMjANCj4gPiAgICAgICAgcnBj X3JlbGVhc2VfdGFzayBuZXQvc3VucnBjL3NjaGVkLmM6MTA1OSBbaW5saW5lXQ0KPiA+ICAgICAg ICBfX3JwY19leGVjdXRlKzB4NGQ5LzB4ZTcwIG5ldC9zdW5ycGMvc2NoZWQuYzo4MjQNCj4gPiAg ICAgICAgcnBjX2FzeW5jX3NjaGVkdWxlKzB4MTYvMHgyMCBuZXQvc3VucnBjL3NjaGVkLmM6ODQ4 DQo+ID4gICAgICAgIHByb2Nlc3Nfb25lX3dvcmsrMHhiZjAvMHgxYmMwIGtlcm5lbC93b3JrcXVl dWUuYzoyMTEyDQo+ID4gICAgICAgIHdvcmtlcl90aHJlYWQrMHgyMjMvMHgxOTkwIGtlcm5lbC93 b3JrcXVldWUuYzoyMjQ2DQo+ID4gICAgICAgIGt0aHJlYWQrMHgzOGIvMHg0NzAga2VybmVsL2t0 aHJlYWQuYzoyNDINCj4gPiAgICAgICAgcmV0X2Zyb21fZm9yaysweDJhLzB4NDAgYXJjaC94ODYv ZW50cnkvZW50cnlfNjQuUzo0MzENCj4gPiANCj4gPiBvdGhlciBpbmZvIHRoYXQgbWlnaHQgaGVs cCB1cyBkZWJ1ZyB0aGlzOg0KPiA+IA0KPiA+ICBQb3NzaWJsZSB1bnNhZmUgbG9ja2luZyBzY2Vu YXJpbzoNCj4gPiANCj4gPiAgICAgICAgQ1BVMCAgICAgICAgICAgICAgICAgICAgQ1BVMQ0KPiA+ ICAgICAgICAtLS0tICAgICAgICAgICAgICAgICAgICAtLS0tDQo+ID4gICBsb2NrKCgmdGFzay0+ dS50a193b3JrKSk7DQo+ID4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxvY2soInhw cnRpb2QiKTsNCj4gPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbG9jaygoJnRhc2st PnUudGtfd29yaykpOw0KPiA+ICAgbG9jaygieHBydGlvZCIpOw0KPiA+IA0KPiA+ICAqKiogREVB RExPQ0sgKioqDQo+ID4gDQo+ID4gMiBsb2NrcyBoZWxkIGJ5IGt3b3JrZXIvMDozLzM0MDA6DQo+ ID4gICMwOiAgKCJycGNpb2QiKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTQ3MWU1Zj5dIF9fd3Jp dGVfb25jZV9zaXplDQo+ID4gaW5jbHVkZS9saW51eC9jb21waWxlci5oOjMwNSBbaW5saW5lXQ0K PiA+ICAjMDogICgicnBjaW9kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0NzFlNWY+XSBhdG9t aWM2NF9zZXQNCj4gPiBhcmNoL3g4Ni9pbmNsdWRlL2FzbS9hdG9taWM2NF82NC5oOjMzIFtpbmxp bmVdDQo+ID4gICMwOiAgKCJycGNpb2QiKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTQ3MWU1Zj5d IGF0b21pY19sb25nX3NldA0KPiA+IGluY2x1ZGUvYXNtLWdlbmVyaWMvYXRvbWljLWxvbmcuaDo1 NiBbaW5saW5lXQ0KPiA+ICAjMDogICgicnBjaW9kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0 NzFlNWY+XSBzZXRfd29ya19kYXRhDQo+ID4ga2VybmVsL3dvcmtxdWV1ZS5jOjYxOCBbaW5saW5l XQ0KPiA+ICAjMDogICgicnBjaW9kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0NzFlNWY+XQ0K PiA+IHNldF93b3JrX3Bvb2xfYW5kX2NsZWFyX3BlbmRpbmcga2VybmVsL3dvcmtxdWV1ZS5jOjY0 NSBbaW5saW5lXQ0KPiA+ICAjMDogICgicnBjaW9kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0 NzFlNWY+XQ0KPiA+IHByb2Nlc3Nfb25lX3dvcmsrMHhhZGYvMHgxYmMwIGtlcm5lbC93b3JrcXVl dWUuYzoyMDgzDQo+ID4gICMxOiAgKCgmdGFzay0+dS50a193b3JrKSl7Ky4rLn0sIGF0OiBbPGZm ZmZmZmZmODE0NzFlYjI+XQ0KPiA+IHByb2Nlc3Nfb25lX3dvcmsrMHhiMzIvMHgxYmMwIGtlcm5l bC93b3JrcXVldWUuYzoyMDg3DQo+ID4gDQo+ID4gc3RhY2sgYmFja3RyYWNlOg0KPiA+IENQVTog MCBQSUQ6IDM0MDAgQ29tbToga3dvcmtlci8wOjMgTm90IHRhaW50ZWQgNC4xNC4wLXJjNS1tbTEr ICMyMA0KPiA+IEhhcmR3YXJlIG5hbWU6IEdvb2dsZSBHb29nbGUgQ29tcHV0ZSBFbmdpbmUvR29v Z2xlIENvbXB1dGUgRW5naW5lLA0KPiA+IEJJT1MNCj4gPiBHb29nbGUgMDEvMDEvMjAxMQ0KPiA+ IFdvcmtxdWV1ZTogcnBjaW9kIHJwY19hc3luY19zY2hlZHVsZQ0KPiA+IENhbGwgVHJhY2U6DQo+ ID4gIF9fZHVtcF9zdGFjayBsaWIvZHVtcF9zdGFjay5jOjE2IFtpbmxpbmVdDQo+ID4gIGR1bXBf c3RhY2srMHgxOTQvMHgyNTcgbGliL2R1bXBfc3RhY2suYzo1Mg0KPiA+ICBwcmludF9jaXJjdWxh cl9idWcuaXNyYS40MSsweDM0Mi8weDM2YQ0KPiA+IGtlcm5lbC9sb2NraW5nL2xvY2tkZXAuYzox MjU4DQo+ID4gIGNoZWNrX3ByZXZfYWRkIGtlcm5lbC9sb2NraW5nL2xvY2tkZXAuYzoxOTAxIFtp bmxpbmVdDQo+ID4gIGNoZWNrX3ByZXZzX2FkZCBrZXJuZWwvbG9ja2luZy9sb2NrZGVwLmM6MjAx OCBbaW5saW5lXQ0KPiA+ICB2YWxpZGF0ZV9jaGFpbiBrZXJuZWwvbG9ja2luZy9sb2NrZGVwLmM6 MjQ2MCBbaW5saW5lXQ0KPiA+ICBfX2xvY2tfYWNxdWlyZSsweDJmNTUvMHgzZDUwIGtlcm5lbC9s b2NraW5nL2xvY2tkZXAuYzozNDg3DQo+ID4gIGxvY2tfYWNxdWlyZSsweDFkNS8weDU4MCBrZXJu ZWwvbG9ja2luZy9sb2NrZGVwLmM6Mzk5MQ0KPiA+ICBzdGFydF9mbHVzaF93b3JrIGtlcm5lbC93 b3JrcXVldWUuYzoyODUxIFtpbmxpbmVdDQo+ID4gIGZsdXNoX3dvcmsrMHg1N2YvMHg4YTAga2Vy bmVsL3dvcmtxdWV1ZS5jOjI4ODINCj4gPiAgX19jYW5jZWxfd29ya190aW1lcisweDMwYS8weDdl MCBrZXJuZWwvd29ya3F1ZXVlLmM6Mjk1NA0KPiA+ICBjYW5jZWxfd29ya19zeW5jKzB4MTcvMHgy MCBrZXJuZWwvd29ya3F1ZXVlLmM6Mjk5MA0KPiA+ICB4cHJ0X2Rlc3Ryb3krMHhhMS8weDEzMCBu ZXQvc3VucnBjL3hwcnQuYzoxNDY3DQo+ID4gIHhwcnRfZGVzdHJveV9rcmVmIG5ldC9zdW5ycGMv eHBydC5jOjE0NzcgW2lubGluZV0NCj4gPiAga3JlZl9wdXQgaW5jbHVkZS9saW51eC9rcmVmLmg6 NzAgW2lubGluZV0NCj4gPiAgeHBydF9wdXQrMHgzOC8weDQwIG5ldC9zdW5ycGMveHBydC5jOjE1 MDENCj4gPiAgcnBjX3Rhc2tfcmVsZWFzZV9jbGllbnQrMHgyOTkvMHg0MzAgbmV0L3N1bnJwYy9j bG50LmM6OTg2DQo+ID4gIHJwY19yZWxlYXNlX3Jlc291cmNlc190YXNrKzB4N2YvMHhhMCBuZXQv c3VucnBjL3NjaGVkLmM6MTAyMA0KPiA+ICBycGNfcmVsZWFzZV90YXNrIG5ldC9zdW5ycGMvc2No ZWQuYzoxMDU5IFtpbmxpbmVdDQo+ID4gIF9fcnBjX2V4ZWN1dGUrMHg0ZDkvMHhlNzAgbmV0L3N1 bnJwYy9zY2hlZC5jOjgyNA0KPiA+ICBycGNfYXN5bmNfc2NoZWR1bGUrMHgxNi8weDIwIG5ldC9z dW5ycGMvc2NoZWQuYzo4NDgNCj4gPiAgcHJvY2Vzc19vbmVfd29yaysweGJmMC8weDFiYzAga2Vy bmVsL3dvcmtxdWV1ZS5jOjIxMTINCj4gPiAgd29ya2VyX3RocmVhZCsweDIyMy8weDE5OTAga2Vy bmVsL3dvcmtxdWV1ZS5jOjIyNDYNCj4gPiAga3RocmVhZCsweDM4Yi8weDQ3MCBrZXJuZWwva3Ro cmVhZC5jOjI0Mg0KPiA+ICByZXRfZnJvbV9mb3JrKzB4MmEvMHg0MCBhcmNoL3g4Ni9lbnRyeS9l bnRyeV82NC5TOjQzMQ0KPiANCj4gDQo+ICtzdW5ycGMgbWFpbnRhaW5lcnMNCg0KQSBmaXggZm9y IHRoaXMgaGFzIGFscmVhZHkgYmVlbiBtZXJnZWQuIFBsZWFzZSByZXRlc3Qgd2l0aCBhbiB1cCB0 bw0KZGF0ZSBrZXJuZWwuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50 IG1haW50YWluZXIsIFByaW1hcnlEYXRhDQp0cm9uZC5teWtsZWJ1c3RAcHJpbWFyeWRhdGEuY29t DQo= ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in flush_work (2) @ 2017-11-05 16:00 ` Trond Myklebust 0 siblings, 0 replies; 6+ messages in thread From: Trond Myklebust @ 2017-11-05 16:00 UTC (permalink / raw) To: dvyukov@google.com, bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.com Cc: linux-kernel@vger.kernel.org, bfields@fieldses.org, linux-nfs@vger.kernel.org, jlayton@poochiereds.net, jiangshanlai@gmail.com, anna.schumaker@netapp.com, netdev@vger.kernel.org, syzkaller-bugs@googlegroups.com, tj@kernel.org, davem@davemloft.net On Sun, 2017-11-05 at 11:53 +0300, Dmitry Vyukov wrote: > On Sun, Nov 5, 2017 at 11:41 AM, syzbot > <bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.c > om> > wrote: > > Hello, > > > > syzkaller hit the following crash on > > 0f611fb6dcc0d6d91b4e1fec911321f434a3b858 > > git://git.cmpxchg.org/linux-mmots.git/master > > compiler: gcc (GCC) 7.1.1 20170620 > > .config is attached > > Raw console output is attached. > > > > xs_tcp_setup_socket: connect returned unhandled error -113 > > xs_tcp_setup_socket: connect returned unhandled error -113 > > xs_tcp_setup_socket: connect returned unhandled error -113 > > > > ====================================================== > > WARNING: possible circular locking dependency detected > > 4.14.0-rc5-mm1+ #20 Not tainted > > ------------------------------------------------------ > > kworker/0:3/3400 is trying to acquire lock: > > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] start_flush_work > > kernel/workqueue.c:2850 [inline] > > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] flush_work+0x55a/0x8a0 > > kernel/workqueue.c:2882 > > > > but task is already holding lock: > > ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] > > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 > > > > which lock already depends on the new lock. > > > > > > the existing dependency chain (in reverse order) is: > > > > -> #1 ((&task->u.tk_work)){+.+.}: > > process_one_work+0xba2/0x1bc0 kernel/workqueue.c:2088 > > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > > kthread+0x38b/0x470 kernel/kthread.c:242 > > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > > > -> #0 ("xprtiod"){+.+.}: > > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 > > start_flush_work kernel/workqueue.c:2851 [inline] > > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 > > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 > > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 > > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 > > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] > > kref_put include/linux/kref.h:70 [inline] > > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 > > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 > > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 > > rpc_release_task net/sunrpc/sched.c:1059 [inline] > > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 > > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 > > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 > > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > > kthread+0x38b/0x470 kernel/kthread.c:242 > > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > > > other info that might help us debug this: > > > > Possible unsafe locking scenario: > > > > CPU0 CPU1 > > ---- ---- > > lock((&task->u.tk_work)); > > lock("xprtiod"); > > lock((&task->u.tk_work)); > > lock("xprtiod"); > > > > *** DEADLOCK *** > > > > 2 locks held by kworker/0:3/3400: > > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] __write_once_size > > include/linux/compiler.h:305 [inline] > > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic64_set > > arch/x86/include/asm/atomic64_64.h:33 [inline] > > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic_long_set > > include/asm-generic/atomic-long.h:56 [inline] > > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] set_work_data > > kernel/workqueue.c:618 [inline] > > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] > > set_work_pool_and_clear_pending kernel/workqueue.c:645 [inline] > > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] > > process_one_work+0xadf/0x1bc0 kernel/workqueue.c:2083 > > #1: ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] > > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 > > > > stack backtrace: > > CPU: 0 PID: 3400 Comm: kworker/0:3 Not tainted 4.14.0-rc5-mm1+ #20 > > Hardware name: Google Google Compute Engine/Google Compute Engine, > > BIOS > > Google 01/01/2011 > > Workqueue: rpciod rpc_async_schedule > > Call Trace: > > __dump_stack lib/dump_stack.c:16 [inline] > > dump_stack+0x194/0x257 lib/dump_stack.c:52 > > print_circular_bug.isra.41+0x342/0x36a > > kernel/locking/lockdep.c:1258 > > check_prev_add kernel/locking/lockdep.c:1901 [inline] > > check_prevs_add kernel/locking/lockdep.c:2018 [inline] > > validate_chain kernel/locking/lockdep.c:2460 [inline] > > __lock_acquire+0x2f55/0x3d50 kernel/locking/lockdep.c:3487 > > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 > > start_flush_work kernel/workqueue.c:2851 [inline] > > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 > > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 > > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 > > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 > > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] > > kref_put include/linux/kref.h:70 [inline] > > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 > > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 > > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 > > rpc_release_task net/sunrpc/sched.c:1059 [inline] > > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 > > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 > > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 > > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 > > kthread+0x38b/0x470 kernel/kthread.c:242 > > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 > > > +sunrpc maintainers A fix for this has already been merged. Please retest with an up to date kernel. -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in flush_work (2) 2017-11-05 16:00 ` Trond Myklebust (?) @ 2017-11-06 10:34 ` Dmitry Vyukov 2018-02-13 18:55 ` Dmitry Vyukov -1 siblings, 1 reply; 6+ messages in thread From: Dmitry Vyukov @ 2017-11-06 10:34 UTC (permalink / raw) To: Trond Myklebust Cc: bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.com, linux-kernel@vger.kernel.org, bfields@fieldses.org, linux-nfs@vger.kernel.org, jlayton@poochiereds.net, jiangshanlai@gmail.com, anna.schumaker@netapp.com, netdev@vger.kernel.org, syzkaller-bugs@googlegroups.com, tj@kernel.org, davem@davemloft.net On Sun, Nov 5, 2017 at 5:00 PM, Trond Myklebust <trondmy@primarydata.com> wrote: > > > On Sun, 2017-11-05 at 11:53 +0300, Dmitry Vyukov wrote: >> On Sun, Nov 5, 2017 at 11:41 AM, syzbot >> <bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.c >> om> >> wrote: >> > Hello, >> > >> > syzkaller hit the following crash on >> > 0f611fb6dcc0d6d91b4e1fec911321f434a3b858 >> > git://git.cmpxchg.org/linux-mmots.git/master >> > compiler: gcc (GCC) 7.1.1 20170620 >> > .config is attached >> > Raw console output is attached. >> > >> > xs_tcp_setup_socket: connect returned unhandled error -113 >> > xs_tcp_setup_socket: connect returned unhandled error -113 >> > xs_tcp_setup_socket: connect returned unhandled error -113 >> > >> > ====================================================== >> > WARNING: possible circular locking dependency detected >> > 4.14.0-rc5-mm1+ #20 Not tainted >> > ------------------------------------------------------ >> > kworker/0:3/3400 is trying to acquire lock: >> > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] start_flush_work >> > kernel/workqueue.c:2850 [inline] >> > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] flush_work+0x55a/0x8a0 >> > kernel/workqueue.c:2882 >> > >> > but task is already holding lock: >> > ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] >> > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 >> > >> > which lock already depends on the new lock. >> > >> > >> > the existing dependency chain (in reverse order) is: >> > >> > -> #1 ((&task->u.tk_work)){+.+.}: >> > process_one_work+0xba2/0x1bc0 kernel/workqueue.c:2088 >> > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 >> > kthread+0x38b/0x470 kernel/kthread.c:242 >> > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 >> > >> > -> #0 ("xprtiod"){+.+.}: >> > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 >> > start_flush_work kernel/workqueue.c:2851 [inline] >> > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 >> > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 >> > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 >> > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 >> > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] >> > kref_put include/linux/kref.h:70 [inline] >> > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 >> > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 >> > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 >> > rpc_release_task net/sunrpc/sched.c:1059 [inline] >> > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 >> > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 >> > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 >> > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 >> > kthread+0x38b/0x470 kernel/kthread.c:242 >> > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 >> > >> > other info that might help us debug this: >> > >> > Possible unsafe locking scenario: >> > >> > CPU0 CPU1 >> > ---- ---- >> > lock((&task->u.tk_work)); >> > lock("xprtiod"); >> > lock((&task->u.tk_work)); >> > lock("xprtiod"); >> > >> > *** DEADLOCK *** >> > >> > 2 locks held by kworker/0:3/3400: >> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] __write_once_size >> > include/linux/compiler.h:305 [inline] >> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic64_set >> > arch/x86/include/asm/atomic64_64.h:33 [inline] >> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic_long_set >> > include/asm-generic/atomic-long.h:56 [inline] >> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] set_work_data >> > kernel/workqueue.c:618 [inline] >> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] >> > set_work_pool_and_clear_pending kernel/workqueue.c:645 [inline] >> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] >> > process_one_work+0xadf/0x1bc0 kernel/workqueue.c:2083 >> > #1: ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] >> > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 >> > >> > stack backtrace: >> > CPU: 0 PID: 3400 Comm: kworker/0:3 Not tainted 4.14.0-rc5-mm1+ #20 >> > Hardware name: Google Google Compute Engine/Google Compute Engine, >> > BIOS >> > Google 01/01/2011 >> > Workqueue: rpciod rpc_async_schedule >> > Call Trace: >> > __dump_stack lib/dump_stack.c:16 [inline] >> > dump_stack+0x194/0x257 lib/dump_stack.c:52 >> > print_circular_bug.isra.41+0x342/0x36a >> > kernel/locking/lockdep.c:1258 >> > check_prev_add kernel/locking/lockdep.c:1901 [inline] >> > check_prevs_add kernel/locking/lockdep.c:2018 [inline] >> > validate_chain kernel/locking/lockdep.c:2460 [inline] >> > __lock_acquire+0x2f55/0x3d50 kernel/locking/lockdep.c:3487 >> > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 >> > start_flush_work kernel/workqueue.c:2851 [inline] >> > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 >> > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 >> > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 >> > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 >> > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] >> > kref_put include/linux/kref.h:70 [inline] >> > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 >> > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 >> > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 >> > rpc_release_task net/sunrpc/sched.c:1059 [inline] >> > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 >> > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 >> > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 >> > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 >> > kthread+0x38b/0x470 kernel/kthread.c:242 >> > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 >> >> >> +sunrpc maintainers > > A fix for this has already been merged. Please retest with an up to > date kernel. Hi, What's the fix? Please specify it in the following form: > syzbot will keep track of this bug report. > Once a fix for this bug is committed, please reply to this email with: > #syz fix: exact-commit-title > Note: all commands must start from beginning of the line. The bot tests HEAD of multiple branches and it needs to know what fixes what, because this still happens on other branches. Once the bot knows the fix, it will track when the fix reaches all tested branches. Thanks ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in flush_work (2) 2017-11-06 10:34 ` Dmitry Vyukov @ 2018-02-13 18:55 ` Dmitry Vyukov 0 siblings, 0 replies; 6+ messages in thread From: Dmitry Vyukov @ 2018-02-13 18:55 UTC (permalink / raw) To: Trond Myklebust Cc: bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.com, linux-kernel@vger.kernel.org, bfields@fieldses.org, linux-nfs@vger.kernel.org, jlayton@poochiereds.net, jiangshanlai@gmail.com, anna.schumaker@netapp.com, netdev@vger.kernel.org, syzkaller-bugs@googlegroups.com, tj@kernel.org, davem@davemloft.net On Mon, Nov 6, 2017 at 11:34 AM, Dmitry Vyukov <dvyukov@google.com> wrote: > On Sun, Nov 5, 2017 at 5:00 PM, Trond Myklebust <trondmy@primarydata.com> wrote: >> >> >> On Sun, 2017-11-05 at 11:53 +0300, Dmitry Vyukov wrote: >>> On Sun, Nov 5, 2017 at 11:41 AM, syzbot >>> <bot+d8fe95298ef830cd7d05e33eefa4a5a6f6f334d4@syzkaller.appspotmail.c >>> om> >>> wrote: >>> > Hello, >>> > >>> > syzkaller hit the following crash on >>> > 0f611fb6dcc0d6d91b4e1fec911321f434a3b858 >>> > git://git.cmpxchg.org/linux-mmots.git/master >>> > compiler: gcc (GCC) 7.1.1 20170620 >>> > .config is attached >>> > Raw console output is attached. >>> > >>> > xs_tcp_setup_socket: connect returned unhandled error -113 >>> > xs_tcp_setup_socket: connect returned unhandled error -113 >>> > xs_tcp_setup_socket: connect returned unhandled error -113 >>> > >>> > ====================================================== >>> > WARNING: possible circular locking dependency detected >>> > 4.14.0-rc5-mm1+ #20 Not tainted >>> > ------------------------------------------------------ >>> > kworker/0:3/3400 is trying to acquire lock: >>> > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] start_flush_work >>> > kernel/workqueue.c:2850 [inline] >>> > ("xprtiod"){+.+.}, at: [<ffffffff8146adda>] flush_work+0x55a/0x8a0 >>> > kernel/workqueue.c:2882 >>> > >>> > but task is already holding lock: >>> > ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] >>> > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 >>> > >>> > which lock already depends on the new lock. >>> > >>> > >>> > the existing dependency chain (in reverse order) is: >>> > >>> > -> #1 ((&task->u.tk_work)){+.+.}: >>> > process_one_work+0xba2/0x1bc0 kernel/workqueue.c:2088 >>> > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 >>> > kthread+0x38b/0x470 kernel/kthread.c:242 >>> > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 >>> > >>> > -> #0 ("xprtiod"){+.+.}: >>> > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 >>> > start_flush_work kernel/workqueue.c:2851 [inline] >>> > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 >>> > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 >>> > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 >>> > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 >>> > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] >>> > kref_put include/linux/kref.h:70 [inline] >>> > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 >>> > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 >>> > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 >>> > rpc_release_task net/sunrpc/sched.c:1059 [inline] >>> > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 >>> > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 >>> > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 >>> > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 >>> > kthread+0x38b/0x470 kernel/kthread.c:242 >>> > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 >>> > >>> > other info that might help us debug this: >>> > >>> > Possible unsafe locking scenario: >>> > >>> > CPU0 CPU1 >>> > ---- ---- >>> > lock((&task->u.tk_work)); >>> > lock("xprtiod"); >>> > lock((&task->u.tk_work)); >>> > lock("xprtiod"); >>> > >>> > *** DEADLOCK *** >>> > >>> > 2 locks held by kworker/0:3/3400: >>> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] __write_once_size >>> > include/linux/compiler.h:305 [inline] >>> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic64_set >>> > arch/x86/include/asm/atomic64_64.h:33 [inline] >>> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] atomic_long_set >>> > include/asm-generic/atomic-long.h:56 [inline] >>> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] set_work_data >>> > kernel/workqueue.c:618 [inline] >>> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] >>> > set_work_pool_and_clear_pending kernel/workqueue.c:645 [inline] >>> > #0: ("rpciod"){+.+.}, at: [<ffffffff81471e5f>] >>> > process_one_work+0xadf/0x1bc0 kernel/workqueue.c:2083 >>> > #1: ((&task->u.tk_work)){+.+.}, at: [<ffffffff81471eb2>] >>> > process_one_work+0xb32/0x1bc0 kernel/workqueue.c:2087 >>> > >>> > stack backtrace: >>> > CPU: 0 PID: 3400 Comm: kworker/0:3 Not tainted 4.14.0-rc5-mm1+ #20 >>> > Hardware name: Google Google Compute Engine/Google Compute Engine, >>> > BIOS >>> > Google 01/01/2011 >>> > Workqueue: rpciod rpc_async_schedule >>> > Call Trace: >>> > __dump_stack lib/dump_stack.c:16 [inline] >>> > dump_stack+0x194/0x257 lib/dump_stack.c:52 >>> > print_circular_bug.isra.41+0x342/0x36a >>> > kernel/locking/lockdep.c:1258 >>> > check_prev_add kernel/locking/lockdep.c:1901 [inline] >>> > check_prevs_add kernel/locking/lockdep.c:2018 [inline] >>> > validate_chain kernel/locking/lockdep.c:2460 [inline] >>> > __lock_acquire+0x2f55/0x3d50 kernel/locking/lockdep.c:3487 >>> > lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3991 >>> > start_flush_work kernel/workqueue.c:2851 [inline] >>> > flush_work+0x57f/0x8a0 kernel/workqueue.c:2882 >>> > __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2954 >>> > cancel_work_sync+0x17/0x20 kernel/workqueue.c:2990 >>> > xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 >>> > xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] >>> > kref_put include/linux/kref.h:70 [inline] >>> > xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 >>> > rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 >>> > rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 >>> > rpc_release_task net/sunrpc/sched.c:1059 [inline] >>> > __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 >>> > rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 >>> > process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2112 >>> > worker_thread+0x223/0x1990 kernel/workqueue.c:2246 >>> > kthread+0x38b/0x470 kernel/kthread.c:242 >>> > ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 >>> >>> >>> +sunrpc maintainers >> >> A fix for this has already been merged. Please retest with an up to >> date kernel. > > Hi, > > What's the fix? Please specify it in the following form: > >> syzbot will keep track of this bug report. >> Once a fix for this bug is committed, please reply to this email with: >> #syz fix: exact-commit-title >> Note: all commands must start from beginning of the line. > > The bot tests HEAD of multiple branches and it needs to know what > fixes what, because this still happens on other branches. Once the bot > knows the fix, it will track when the fix reaches all tested branches. Seems to be this one: #syz fix: SUNRPC: Destroy transport from the system workqueue ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-02-13 18:55 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <001a113ee9baf95598055d384ecb@google.com>
2017-11-05 8:53 ` possible deadlock in flush_work (2) Dmitry Vyukov
2017-11-05 8:53 ` Dmitry Vyukov
2017-11-05 16:00 ` Trond Myklebust
2017-11-05 16:00 ` Trond Myklebust
2017-11-06 10:34 ` Dmitry Vyukov
2018-02-13 18:55 ` Dmitry Vyukov
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.