From: Johannes Berg <johannes@sipsolutions.net>
To: Nicolai Stange <nicstange@gmail.com>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
"Paul E.McKenney" <paulmck@linux.vnet.ibm.com>,
gregkh <gregkh@linuxfoundation.org>
Subject: Re: deadlock in synchronize_srcu() in debugfs?
Date: Thu, 23 Mar 2017 16:47:04 +0100 [thread overview]
Message-ID: <1490284024.2766.12.camel@sipsolutions.net> (raw)
In-Reply-To: <87o9ws6m4s.fsf@gmail.com> (sfid-20170323_163621_602585_CBD64B58)
Hi,
> Not yet. How reproducible is this?
Apparently quite. I haven't tried myself - it happens during some
automated test that I need to analyse further.
> > We're observing that with our (backported, but very recent) driver
> > against 4.9 (and 4.10, I think),
>
> Do I understand it correctly that this driver has been backported
> from 4.11-rcX to 4.9/10
Yes.
> and that there isn't any issue with 4.11-rcX?
No, I can't say this, we haven't run that test.
> > but there are no backports of any debugfs things so the backport
> > itself doesn't seem like a likely problem.
>
> Right, there haven't been any SRCU related changes to debugfs after
> 4.8.
Right.
> > sysrq-w shows a lot of tasks blocked on various locks (e.g. RTNL),
> > but
> > the ultimate problem is the wireless stack getting blocked on
> > debugfs_remove_recursive(), in __synchronize_srcu(), in
> > wait_for_completion() (while holding lots of locks, hence the other
> > tasks getting stuck).
>
> Could you share a complete backtrace? For example, is the
> debugfs_remove_recursive() called from any debugfs file's fops and
> thus, possibly from within a SRCU read side critical section?
No, it's called from netlink:
[ 884.634857] wpa_supplicant D 0 1769 1005 0x00000000
[ 884.634874] 0000000000000000 ffff8ca50633d140 ffff8ca507b219c0 ffff8ca5455d4cc0
[ 884.634898] ffff8ca54f599d98 ffff97df431c36a0 ffffffff878dadf3 ffff8ca500000001
[ 884.634927] 81ed67337c8469e4 ffff8ca54f599d98 0000932a07b219c0 ffff8ca507b219c0
[ 884.634952] Call Trace:
[ 884.634969] [<ffffffff878dadf3>] ? __schedule+0x303/0xb00
[ 884.634985] [<ffffffff878db62d>] schedule+0x3d/0x90
[ 884.635002] [<ffffffff878e022c>] schedule_timeout+0x2fc/0x600
[ 884.635021] [<ffffffff870e8b06>] ? mark_held_locks+0x66/0x90
[ 884.635041] [<ffffffff878e16bc>] ? _raw_spin_unlock_irq+0x2c/0x40
[ 884.635059] [<ffffffff878dc8cc>] wait_for_completion+0xdc/0x110
[ 884.635073] [<ffffffff870bff90>] ? wake_up_q+0x80/0x80
[ 884.635091] [<ffffffff8710a46e>] __synchronize_srcu+0x11e/0x1c0
[ 884.635109] [<ffffffff87109510>] ? trace_raw_output_rcu_utilization+0x60/0x60
[ 884.635131] [<ffffffff8710a542>] synchronize_srcu+0x32/0x40
[ 884.635145] [<ffffffff873899ed>] debugfs_remove_recursive+0x17d/0x190
[ 884.635239] [<ffffffffc087b3be>] ieee80211_debugfs_key_remove+0x1e/0x30 [mac80211]
[ 884.635333] [<ffffffffc0840773>] __ieee80211_key_destroy+0x1b3/0x480 [mac80211]
[ 884.635440] [<ffffffffc0841807>] ieee80211_free_sta_keys+0x117/0x170 [mac80211]
[ 884.635524] [<ffffffffc0807b0c>] __sta_info_destroy_part2+0x4c/0x200 [mac80211]
[ 884.635597] [<ffffffffc0807fbd>] __sta_info_flush+0x10d/0x1a0 [mac80211]
[ 884.635706] [<ffffffffc086634b>] ieee80211_set_disassoc+0xcb/0x530 [mac80211]
[ 884.635802] [<ffffffffc086e3b6>] ieee80211_mgd_deauth+0x2e6/0x7b0 [mac80211]
[ 884.635901] [<ffffffffc08237c8>] ieee80211_deauth+0x18/0x20 [mac80211]
[ 884.636024] [<ffffffffc0673e8f>] cfg80211_mlme_deauth+0x14f/0x3b0 [cfg80211]
[ 884.636110] [<ffffffffc0649265>] nl80211_deauthenticate+0xe5/0x130 [cfg80211]
[ 884.636133] [<ffffffff877dc52c>] genl_family_rcv_msg+0x1bc/0x370
[ 884.636151] [<ffffffff877dc6e0>] ? genl_family_rcv_msg+0x370/0x370
[ 884.636262] [<ffffffff877dc760>] genl_rcv_msg+0x80/0xc0
[ 884.636275] [<ffffffff877dba87>] netlink_rcv_skb+0xa7/0xc0
[ 884.636289] [<ffffffff877dc148>] genl_rcv+0x28/0x40
[ 884.636303] [<ffffffff877db45b>] netlink_unicast+0x15b/0x210
[ 884.636318] [<ffffffff877db82a>] netlink_sendmsg+0x31a/0x3a0
[ 884.636335] [<ffffffff8777bb48>] sock_sendmsg+0x38/0x50
[ 884.636354] [<ffffffff8777c41c>] ___sys_sendmsg+0x26c/0x280
[ 884.636378] [<ffffffff8717b042>] ? ring_buffer_unlock_commit+0x32/0x290
[ 884.636393] [<ffffffff8718122e>] ? __buffer_unlock_commit+0x1e/0x40
[ 884.636407] [<ffffffff87181d12>] ? tracing_mark_write+0x162/0x2b0
[ 884.636423] [<ffffffff870e7419>] ? __lock_is_held+0x49/0x70
[ 884.636440] [<ffffffff8777d0a5>] __sys_sendmsg+0x45/0x80
[ 884.636459] [<ffffffff8777d0f2>] SyS_sendmsg+0x12/0x20
[ 884.636477] [<ffffffff878e1e45>] entry_SYSCALL_64_fastpath+0x23/0xc6
johannes
next prev parent reply other threads:[~2017-03-23 15:47 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-23 14:54 deadlock in synchronize_srcu() in debugfs? Johannes Berg
2017-03-23 15:29 ` Johannes Berg
2017-03-24 8:56 ` Johannes Berg
2017-03-24 9:24 ` Johannes Berg
2017-03-24 17:45 ` Paul E. McKenney
2017-03-24 18:51 ` Johannes Berg
2017-03-24 19:33 ` Paul E. McKenney
2017-03-24 20:20 ` Paul E. McKenney
2017-03-27 11:18 ` Johannes Berg
2017-03-23 15:36 ` Nicolai Stange
2017-03-23 15:47 ` Johannes Berg [this message]
2017-03-27 11:36 ` Johannes Berg
2017-03-30 7:32 ` Nicolai Stange
2017-03-30 7:55 ` Johannes Berg
2017-03-30 10:27 ` Nicolai Stange
2017-03-30 11:11 ` Johannes Berg
2017-03-31 9:03 ` Nicolai Stange
2017-03-31 9:44 ` Johannes Berg
2017-04-16 9:51 ` [RFC PATCH 0/9] debugfs: per-file removal protection Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 1/9] debugfs: add support for more elaborate ->d_fsdata Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 2/9] debugfs: implement per-file removal protection Nicolai Stange
2017-04-18 2:23 ` [lkp-robot] [debugfs] f3e7155d08: BUG:unable_to_handle_kernel kernel test robot
2017-04-18 2:23 ` kernel test robot
2017-04-23 18:37 ` Nicolai Stange
2017-04-23 18:37 ` Nicolai Stange
2017-04-24 6:36 ` Ye Xiaolong
2017-04-24 6:36 ` Ye Xiaolong
2017-04-16 9:51 ` [RFC PATCH 3/9] debugfs: debugfs_real_fops(): drop __must_hold sparse annotation Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 4/9] debugfs: convert to debugfs_file_get() and -put() Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 5/9] IB/hfi1: " Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 6/9] debugfs: purge obsolete SRCU based removal protection Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 7/9] debugfs: call debugfs_real_fops() only after debugfs_file_get() Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 8/9] debugfs: defer debugfs_fsdata allocation to first usage Nicolai Stange
2017-04-18 9:36 ` Johannes Berg
2017-05-02 20:05 ` Nicolai Stange
2017-05-03 5:43 ` Johannes Berg
2017-04-16 9:51 ` [RFC PATCH 9/9] debugfs: free debugfs_fsdata instances Nicolai Stange
2017-04-17 16:01 ` Paul E. McKenney
2017-04-18 9:39 ` Johannes Berg
2017-04-18 13:31 ` Paul E. McKenney
2017-04-18 13:40 ` Johannes Berg
2017-04-18 15:17 ` Paul E. McKenney
2017-04-18 15:20 ` Johannes Berg
2017-04-18 17:19 ` Paul E. McKenney
2017-03-23 15:37 ` deadlock in synchronize_srcu() in debugfs? Paul E. McKenney
2017-03-23 15:46 ` Johannes Berg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1490284024.2766.12.camel@sipsolutions.net \
--to=johannes@sipsolutions.net \
--cc=gregkh@linuxfoundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nicstange@gmail.com \
--cc=paulmck@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.