From: Johannes Berg <johannes@sipsolutions.net>
To: Nicolai Stange <nicstange@gmail.com>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
"Paul E.McKenney" <paulmck@linux.vnet.ibm.com>,
gregkh <gregkh@linuxfoundation.org>
Subject: Re: deadlock in synchronize_srcu() in debugfs?
Date: Thu, 23 Mar 2017 16:47:04 +0100 [thread overview]
Message-ID: <1490284024.2766.12.camel@sipsolutions.net> (raw)
In-Reply-To: <87o9ws6m4s.fsf@gmail.com> (sfid-20170323_163621_602585_CBD64B58)
Hi,
> Not yet. How reproducible is this?
Apparently quite. I haven't tried myself - it happens during some
automated test that I need to analyse further.
> > We're observing that with our (backported, but very recent) driver
> > against 4.9 (and 4.10, I think),
>
> Do I understand it correctly that this driver has been backported
> from 4.11-rcX to 4.9/10
Yes.
> and that there isn't any issue with 4.11-rcX?
No, I can't say this, we haven't run that test.
> > but there are no backports of any debugfs things so the backport
> > itself doesn't seem like a likely problem.
>
> Right, there haven't been any SRCU related changes to debugfs after
> 4.8.
Right.
> > sysrq-w shows a lot of tasks blocked on various locks (e.g. RTNL),
> > but
> > the ultimate problem is the wireless stack getting blocked on
> > debugfs_remove_recursive(), in __synchronize_srcu(), in
> > wait_for_completion() (while holding lots of locks, hence the other
> > tasks getting stuck).
>
> Could you share a complete backtrace? For example, is the
> debugfs_remove_recursive() called from any debugfs file's fops and
> thus, possibly from within a SRCU read side critical section?
No, it's called from netlink:
[ 884.634857] wpa_supplicant D 0 1769 1005 0x00000000
[ 884.634874] 0000000000000000 ffff8ca50633d140 ffff8ca507b219c0 ffff8ca5455d4cc0
[ 884.634898] ffff8ca54f599d98 ffff97df431c36a0 ffffffff878dadf3 ffff8ca500000001
[ 884.634927] 81ed67337c8469e4 ffff8ca54f599d98 0000932a07b219c0 ffff8ca507b219c0
[ 884.634952] Call Trace:
[ 884.634969] [<ffffffff878dadf3>] ? __schedule+0x303/0xb00
[ 884.634985] [<ffffffff878db62d>] schedule+0x3d/0x90
[ 884.635002] [<ffffffff878e022c>] schedule_timeout+0x2fc/0x600
[ 884.635021] [<ffffffff870e8b06>] ? mark_held_locks+0x66/0x90
[ 884.635041] [<ffffffff878e16bc>] ? _raw_spin_unlock_irq+0x2c/0x40
[ 884.635059] [<ffffffff878dc8cc>] wait_for_completion+0xdc/0x110
[ 884.635073] [<ffffffff870bff90>] ? wake_up_q+0x80/0x80
[ 884.635091] [<ffffffff8710a46e>] __synchronize_srcu+0x11e/0x1c0
[ 884.635109] [<ffffffff87109510>] ? trace_raw_output_rcu_utilization+0x60/0x60
[ 884.635131] [<ffffffff8710a542>] synchronize_srcu+0x32/0x40
[ 884.635145] [<ffffffff873899ed>] debugfs_remove_recursive+0x17d/0x190
[ 884.635239] [<ffffffffc087b3be>] ieee80211_debugfs_key_remove+0x1e/0x30 [mac80211]
[ 884.635333] [<ffffffffc0840773>] __ieee80211_key_destroy+0x1b3/0x480 [mac80211]
[ 884.635440] [<ffffffffc0841807>] ieee80211_free_sta_keys+0x117/0x170 [mac80211]
[ 884.635524] [<ffffffffc0807b0c>] __sta_info_destroy_part2+0x4c/0x200 [mac80211]
[ 884.635597] [<ffffffffc0807fbd>] __sta_info_flush+0x10d/0x1a0 [mac80211]
[ 884.635706] [<ffffffffc086634b>] ieee80211_set_disassoc+0xcb/0x530 [mac80211]
[ 884.635802] [<ffffffffc086e3b6>] ieee80211_mgd_deauth+0x2e6/0x7b0 [mac80211]
[ 884.635901] [<ffffffffc08237c8>] ieee80211_deauth+0x18/0x20 [mac80211]
[ 884.636024] [<ffffffffc0673e8f>] cfg80211_mlme_deauth+0x14f/0x3b0 [cfg80211]
[ 884.636110] [<ffffffffc0649265>] nl80211_deauthenticate+0xe5/0x130 [cfg80211]
[ 884.636133] [<ffffffff877dc52c>] genl_family_rcv_msg+0x1bc/0x370
[ 884.636151] [<ffffffff877dc6e0>] ? genl_family_rcv_msg+0x370/0x370
[ 884.636262] [<ffffffff877dc760>] genl_rcv_msg+0x80/0xc0
[ 884.636275] [<ffffffff877dba87>] netlink_rcv_skb+0xa7/0xc0
[ 884.636289] [<ffffffff877dc148>] genl_rcv+0x28/0x40
[ 884.636303] [<ffffffff877db45b>] netlink_unicast+0x15b/0x210
[ 884.636318] [<ffffffff877db82a>] netlink_sendmsg+0x31a/0x3a0
[ 884.636335] [<ffffffff8777bb48>] sock_sendmsg+0x38/0x50
[ 884.636354] [<ffffffff8777c41c>] ___sys_sendmsg+0x26c/0x280
[ 884.636378] [<ffffffff8717b042>] ? ring_buffer_unlock_commit+0x32/0x290
[ 884.636393] [<ffffffff8718122e>] ? __buffer_unlock_commit+0x1e/0x40
[ 884.636407] [<ffffffff87181d12>] ? tracing_mark_write+0x162/0x2b0
[ 884.636423] [<ffffffff870e7419>] ? __lock_is_held+0x49/0x70
[ 884.636440] [<ffffffff8777d0a5>] __sys_sendmsg+0x45/0x80
[ 884.636459] [<ffffffff8777d0f2>] SyS_sendmsg+0x12/0x20
[ 884.636477] [<ffffffff878e1e45>] entry_SYSCALL_64_fastpath+0x23/0xc6
johannes
next prev parent reply other threads:[~2017-03-23 15:47 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-23 14:54 deadlock in synchronize_srcu() in debugfs? Johannes Berg
2017-03-23 15:29 ` Johannes Berg
2017-03-24 8:56 ` Johannes Berg
2017-03-24 9:24 ` Johannes Berg
2017-03-24 17:45 ` Paul E. McKenney
2017-03-24 18:51 ` Johannes Berg
2017-03-24 19:33 ` Paul E. McKenney
2017-03-24 20:20 ` Paul E. McKenney
2017-03-27 11:18 ` Johannes Berg
2017-03-23 15:36 ` Nicolai Stange
2017-03-23 15:47 ` Johannes Berg [this message]
2017-03-27 11:36 ` Johannes Berg
2017-03-30 7:32 ` Nicolai Stange
2017-03-30 7:55 ` Johannes Berg
2017-03-30 10:27 ` Nicolai Stange
2017-03-30 11:11 ` Johannes Berg
2017-03-31 9:03 ` Nicolai Stange
2017-03-31 9:44 ` Johannes Berg
2017-04-16 9:51 ` [RFC PATCH 0/9] debugfs: per-file removal protection Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 1/9] debugfs: add support for more elaborate ->d_fsdata Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 2/9] debugfs: implement per-file removal protection Nicolai Stange
2017-04-18 2:23 ` [lkp-robot] [debugfs] f3e7155d08: BUG:unable_to_handle_kernel kernel test robot
2017-04-23 18:37 ` Nicolai Stange
2017-04-24 6:36 ` Ye Xiaolong
2017-04-16 9:51 ` [RFC PATCH 3/9] debugfs: debugfs_real_fops(): drop __must_hold sparse annotation Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 4/9] debugfs: convert to debugfs_file_get() and -put() Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 5/9] IB/hfi1: " Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 6/9] debugfs: purge obsolete SRCU based removal protection Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 7/9] debugfs: call debugfs_real_fops() only after debugfs_file_get() Nicolai Stange
2017-04-16 9:51 ` [RFC PATCH 8/9] debugfs: defer debugfs_fsdata allocation to first usage Nicolai Stange
2017-04-18 9:36 ` Johannes Berg
2017-05-02 20:05 ` Nicolai Stange
2017-05-03 5:43 ` Johannes Berg
2017-04-16 9:51 ` [RFC PATCH 9/9] debugfs: free debugfs_fsdata instances Nicolai Stange
2017-04-17 16:01 ` Paul E. McKenney
2017-04-18 9:39 ` Johannes Berg
2017-04-18 13:31 ` Paul E. McKenney
2017-04-18 13:40 ` Johannes Berg
2017-04-18 15:17 ` Paul E. McKenney
2017-04-18 15:20 ` Johannes Berg
2017-04-18 17:19 ` Paul E. McKenney
2017-03-23 15:37 ` deadlock in synchronize_srcu() in debugfs? Paul E. McKenney
2017-03-23 15:46 ` Johannes Berg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1490284024.2766.12.camel@sipsolutions.net \
--to=johannes@sipsolutions.net \
--cc=gregkh@linuxfoundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nicstange@gmail.com \
--cc=paulmck@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).