Lockup with ath5k, rtnl_mutex related?

Lockup with ath5k, rtnl_mutex related?

[Ben Greear]

I'm (also) trying to debug an issue with an ath5k system that deadlocks.

We have sysrq output for the held locks, process backtraces, etc.

One thing that has me confused is that the locks-held output shows
a large number of processes owning the rtnl_mutex.  Any idea
how to tell what process *really* owns it?


Oct  4 14:57:46 localhost kernel: SysRq : Show Locks Held
Oct  4 14:57:46 localhost kernel:
Oct  4 14:57:46 localhost kernel: Showing all locks held in the system:
Oct  4 14:57:46 localhost kernel: 4 locks held by kworker/u:0/5:
Oct  4 14:57:46 localhost kernel: #0:  ((wiphy_name(local->hw.wiphy))){+.+.+.}, at: [<c0443d2d>] process_one_work+0x173/0x2c3
Oct  4 14:57:46 localhost kernel: #1:  ((&local->work_work)){+.+.+.}, at: [<c0443d2d>] process_one_work+0x173/0x2c3
Oct  4 14:57:46 localhost kernel: #2:  (&ifmgd->mtx){+.+.+.}, at: [<f8a9b7bd>] ieee80211_assoc_done+0x9b/0xf6 [mac80211]
Oct  4 14:57:46 localhost kernel: #3:  (&local->iflist_mtx){+.+...}, at: [<f8aaa384>] ieee80211_set_channel_type+0x20/0xe3 [mac80211]
Oct  4 14:57:46 localhost kernel: 3 locks held by kworker/0:2/51:
Oct  4 14:57:46 localhost kernel: #0:  (events){+.+.+.}, at: [<c0443d2d>] process_one_work+0x173/0x2c3
Oct  4 14:57:46 localhost kernel: #1:  ((linkwatch_work).work){+.+.+.}, at: [<c0443d2d>] process_one_work+0x173/0x2c3
Oct  4 14:57:46 localhost kernel: #2:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: 1 lock held by hald/1316:
Oct  4 14:57:46 localhost kernel: #0:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: 1 lock held by ntpd/1388:
Oct  4 14:57:46 localhost kernel: #0:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: 1 lock held by mingetty/1497:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by mingetty/1499:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by mingetty/1501:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by mingetty/1503:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by mingetty/1505:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by mingetty/1511:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by bash/1802:
Oct  4 14:57:46 localhost kernel: #0:  (&tty->atomic_read_lock){+.+...}, at: [<c05f63f1>] n_tty_read+0x1d1/0x5ed
Oct  4 14:57:46 localhost kernel: 1 lock held by gnuserver/2195:
Oct  4 14:57:46 localhost kernel: #0:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: 3 locks held by kworker/0:0/4000:
Oct  4 14:57:46 localhost kernel: #0:  (events){+.+.+.}, at: [<c0443d2d>] process_one_work+0x173/0x2c3
Oct  4 14:57:46 localhost kernel: #1:  (wireless_nlevent_work){+.+.+.}, at: [<c0443d2d>] process_one_work+0x173/0x2c3
Oct  4 14:57:46 localhost kernel: #2:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: 4 locks held by ip/4035:
Oct  4 14:57:46 localhost kernel: #0:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: #1:  (&ifmgd->mtx){+.+.+.}, at: [<f8a9b7bd>] ieee80211_assoc_done+0x9b/0xf6 [mac80211]
Oct  4 14:57:46 localhost kernel: #2:  (&local->iflist_mtx){+.+...}, at: [<f8a9b5ca>] ieee80211_assoc_success+0x68c/0x7e4 [mac80211]
Oct  4 14:57:46 localhost kernel: #3:  (&ifmgd->mtx/1){+.+...}, at: [<f8aa8d37>] ieee80211_recalc_smps+0xb6/0x151 [mac80211]
Oct  4 14:57:46 localhost kernel: 1 lock held by iwconfig/4040:
Oct  4 14:57:46 localhost kernel: #0:  (rtnl_mutex){+.+.+.}, at: [<c06db8ea>] rtnl_lock+0xf/0x11
Oct  4 14:57:46 localhost kernel: 2 locks held by bash/5275:
Oct  4 14:57:46 localhost kernel: #0:  (sysrq_key_table_lock){......}, at: [<c06052d4>] __handle_sysrq+0x18/0x110
Oct  4 14:57:46 localhost kernel: #1:  (tasklist_lock){.+.+..}, at: [<c0455c6f>] debug_show_all_locks+0x41/0x142


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Lockup with ath5k, rtnl_mutex related?

[Johannes Berg]

On Mon, 2010-10-04 at 16:34 -0700, Ben Greear wrote:
> I'm (also) trying to debug an issue with an ath5k system that deadlocks.
> 
> We have sysrq output for the held locks, process backtraces, etc.
> 
> One thing that has me confused is that the locks-held output shows
> a large number of processes owning the rtnl_mutex.  Any idea
> how to tell what process *really* owns it?

Can't really -- the mutex code acquires the lockdep dep_map before it
even manages to lock, so that it can detect recursive locking.

johannes

[PATCH] mac80211: fix deadlock with multiple interfaces

[Johannes Berg]

From: Johannes Berg <johannes.berg@intel.com>

The locking around ieee80211_recalc_smps is
buggy -- it cannot acquire another interface's
mutex while the iflist mutex is held because
another code path could be holding the iface
mutex and trying to acquire the iflist mutex.

But the locking is also unnecessary, we only
check "ifmgd->associated" as a bool, and don't
use the pointer (in check_mgd_smps).

Reported-by: Ben Greear <greearb@candelatech.com>
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
---
 net/mac80211/cfg.c         |    2 +-
 net/mac80211/ieee80211_i.h |    3 +--
 net/mac80211/main.c        |    2 +-
 net/mac80211/mlme.c        |    2 +-
 net/mac80211/util.c        |   20 +++-----------------
 5 files changed, 7 insertions(+), 22 deletions(-)

--- wirelmac80211: fix deadlock with multiple interfacesess-testing.orig/net/mac80211/cfg.c	2010-10-05 10:37:21.000000000 +0200
+++ wireless-testing/net/mac80211/cfg.c	2010-10-05 10:37:29.000000000 +0200
@@ -1380,7 +1380,7 @@ int __ieee80211_request_smps(struct ieee
 	if (!sdata->u.mgd.associated ||
 	    sdata->vif.bss_conf.channel_type == NL80211_CHAN_NO_HT) {
 		mutex_lock(&sdata->local->iflist_mtx);
-		ieee80211_recalc_smps(sdata->local, sdata);
+		ieee80211_recalc_smps(sdata->local);
 		mutex_unlock(&sdata->local->iflist_mtx);
 		return 0;
 	}
--- wireless-testing.orig/net/mac80211/ieee80211_i.h	2010-10-05 10:37:11.000000000 +0200
+++ wireless-testing/net/mac80211/ieee80211_i.h	2010-10-05 10:37:16.000000000 +0200
@@ -1292,8 +1292,7 @@ u32 ieee80211_sta_get_rates(struct ieee8
 			    enum ieee80211_band band);
 int __ieee80211_request_smps(struct ieee80211_sub_if_data *sdata,
 			     enum ieee80211_smps_mode smps_mode);
-void ieee80211_recalc_smps(struct ieee80211_local *local,
-			   struct ieee80211_sub_if_data *forsdata);
+void ieee80211_recalc_smps(struct ieee80211_local *local);
 
 size_t ieee80211_ie_split(const u8 *ies, size_t ielen,
 			  const u8 *ids, int n_ids, size_t offset);
--- wireless-testing.orig/net/mac80211/main.c	2010-10-05 10:37:36.000000000 +0200
+++ wireless-testing/net/mac80211/main.c	2010-10-05 10:37:43.000000000 +0200
@@ -329,7 +329,7 @@ static void ieee80211_recalc_smps_work(s
 		container_of(work, struct ieee80211_local, recalc_smps);
 
 	mutex_lock(&local->iflist_mtx);
-	ieee80211_recalc_smps(local, NULL);
+	ieee80211_recalc_smps(local);
 	mutex_unlock(&local->iflist_mtx);
 }
 
--- wireless-testing.orig/net/mac80211/mlme.c	2010-10-05 10:37:36.000000000 +0200
+++ wireless-testing/net/mac80211/mlme.c	2010-10-05 10:37:48.000000000 +0200
@@ -913,7 +913,7 @@ static void ieee80211_set_associated(str
 
 	mutex_lock(&local->iflist_mtx);
 	ieee80211_recalc_ps(local, -1);
-	ieee80211_recalc_smps(local, sdata);
+	ieee80211_recalc_smps(local);
 	mutex_unlock(&local->iflist_mtx);
 
 	netif_tx_start_all_queues(sdata->dev);
--- wireless-testing.orig/net/mac80211/util.c	2010-10-05 10:36:21.000000000 +0200
+++ wireless-testing/net/mac80211/util.c	2010-10-05 10:37:06.000000000 +0200
@@ -1297,16 +1297,12 @@ static int check_mgd_smps(struct ieee802
 }
 
 /* must hold iflist_mtx */
-void ieee80211_recalc_smps(struct ieee80211_local *local,
-			   struct ieee80211_sub_if_data *forsdata)
+void ieee80211_recalc_smps(struct ieee80211_local *local)
 {
 	struct ieee80211_sub_if_data *sdata;
 	enum ieee80211_smps_mode smps_mode = IEEE80211_SMPS_OFF;
 	int count = 0;
 
-	if (forsdata)
-		lockdep_assert_held(&forsdata->u.mgd.mtx);
-
 	lockdep_assert_held(&local->iflist_mtx);
 
 	/*
@@ -1324,18 +1320,8 @@ void ieee80211_recalc_smps(struct ieee80
 			continue;
 		if (sdata->vif.type != NL80211_IFTYPE_STATION)
 			goto set;
-		if (sdata != forsdata) {
-			/*
-			 * This nested is ok -- we are holding the iflist_mtx
-			 * so can't get here twice or so. But it's required
-			 * since normally we acquire it first and then the
-			 * iflist_mtx.
-			 */
-			mutex_lock_nested(&sdata->u.mgd.mtx, SINGLE_DEPTH_NESTING);
-			count += check_mgd_smps(&sdata->u.mgd, &smps_mode);
-			mutex_unlock(&sdata->u.mgd.mtx);
-		} else
-			count += check_mgd_smps(&sdata->u.mgd, &smps_mode);
+
+		count += check_mgd_smps(&sdata->u.mgd, &smps_mode);
 
 		if (count > 1) {
 			smps_mode = IEEE80211_SMPS_OFF;