Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059)

Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059)

[Sebastian Andrzej Siewior]

On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:
> Hi,
Hi,

> The issue persists up to v4.11.12-rt10
Does
	CONFIG_RWLOCK_RT_READER_BIASED=y
make it go away?

> Stacktrace:
> 
> Aug 29 17:28:04 localhost kernel: [   46.483810] ------------[ cut here ]------------
> Aug 29 17:28:04 localhost kernel: [   46.483812] kernel BUG at kernel/locking/rtmutex.c:1059!

this is a deadlock on RT however !RT has a hidden problem which not
yelled at by lockdep. We have the following call path:

| hci_send_monitor_ctrl_event()
|   read_lock(&hci_sk_list.lock);
|
|   hci_send_to_channel()
|      read_lock(&hci_sk_list.lock);

So both functions acquire the same read_lock. If a write comes along
between the first read_lock() and second read_lock() then we have a
deadlock because the write_lock() will lock down further readers from
acquiring the read-lock until the writer completed its task.

This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add
support for sending MGMT commands and events to monitor").

Sebastian

Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059)

[Thomas Gleixner]

On Thu, 31 Aug 2017, Sebastian Andrzej Siewior wrote:

> On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:
> > Hi,
> Hi,
> 
> > The issue persists up to v4.11.12-rt10
> Does
> 	CONFIG_RWLOCK_RT_READER_BIASED=y
> make it go away?
> 
> > Stacktrace:
> > 
> > Aug 29 17:28:04 localhost kernel: [   46.483810] ------------[ cut here ]------------
> > Aug 29 17:28:04 localhost kernel: [   46.483812] kernel BUG at kernel/locking/rtmutex.c:1059!
> 
> this is a deadlock on RT however !RT has a hidden problem which not
> yelled at by lockdep. We have the following call path:
> 
> | hci_send_monitor_ctrl_event()
> |   read_lock(&hci_sk_list.lock);
> |
> |   hci_send_to_channel()
> |      read_lock(&hci_sk_list.lock);
> 
> So both functions acquire the same read_lock. If a write comes along
> between the first read_lock() and second read_lock() then we have a
> deadlock because the write_lock() will lock down further readers from
> acquiring the read-lock until the writer completed its task.
> 
> This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add
> support for sending MGMT commands and events to monitor").

Just for clarification: This is a mainline issue because qrlock based
rwlocks are writer fair, while the traditional rwlocks are reader
biased. Lockdep does not yell about it yet, because the wrapper function
use read-recursive mode.

Thanks,

	tglx

Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059)

[Mart van de Wege]

On Thu, 31 Aug 2017 16:52:12 +0200
Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote:

> On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:
> > Hi,  
> Hi,
> 
> > The issue persists up to v4.11.12-rt10  
> Does
> 	CONFIG_RWLOCK_RT_READER_BIASED=y
> make it go away?
> 
Yes, that does make it go away. -rt8 now boots cleanly, and works fine.
Building -rt10 now to be sure.

> > Stacktrace:
> > 
> > Aug 29 17:28:04 localhost kernel: [   46.483810] ------------[ cut
> > here ]------------ Aug 29 17:28:04 localhost kernel: [   46.483812]
> > kernel BUG at kernel/locking/rtmutex.c:1059!  
> 
> this is a deadlock on RT however !RT has a hidden problem which not
> yelled at by lockdep. We have the following call path:
> 
> | hci_send_monitor_ctrl_event()
> |   read_lock(&hci_sk_list.lock);
> |
> |   hci_send_to_channel()
> |      read_lock(&hci_sk_list.lock);
> 
> So both functions acquire the same read_lock. If a write comes along
> between the first read_lock() and second read_lock() then we have a
> deadlock because the write_lock() will lock down further readers from
> acquiring the read-lock until the writer completed its task.
> 
> This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add
> support for sending MGMT commands and events to monitor").
> 
> Sebastian

Re: Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059)

[Mart van de Wege]

On Fri, 1 Sep 2017 10:51:07 +0200
Mart van de Wege <mvdwege@gmail.com> wrote:

> On Thu, 31 Aug 2017 16:52:12 +0200
> Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote:
> 
> > On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:  
> > > Hi,    
> > Hi,
> >   
> > > The issue persists up to v4.11.12-rt10    
> > Does
> > 	CONFIG_RWLOCK_RT_READER_BIASED=y
> > make it go away?
> >   
> Yes, that does make it go away. -rt8 now boots cleanly, and works
> fine. Building -rt10 now to be sure.
> 
Confirm that as of 4.11.12-rt11 the deadlock still goes away when
enabling CONFIG_RWLOCK_RT_READER_BIASED

Mart

[PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel()

[Sebastian Andrzej Siewior]

Mart reported a deadlock in -RT in the call path:
  hci_send_monitor_ctrl_event() -> hci_send_to_channel()

because both functions acquire the same read lock hci_sk_list.lock. This
is also a mainline issue because the qrwlock implementation is writer
fair (the traditional rwlock implementation is reader biased).

To avoid the deadlock there is now __hci_send_to_channel() which expects
the readlock to be held.

Cc: Marcel Holtmann <marcel@holtmann.org>
Cc: Johan Hedberg <johan.hedberg@intel.com>
Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor")
Reported-by: Mart van de Wege <mvdwege@gmail.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 net/bluetooth/hci_sock.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index 638bf0e1a2e3..f1ee820d871c 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c
@@ -251,15 +251,13 @@ void hci_send_to_sock(struct hci_dev *hdev, struct sk_buff *skb)
 }
 
 /* Send frame to sockets with specific channel */
-void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
-			 int flag, struct sock *skip_sk)
+static void __hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
+				  int flag, struct sock *skip_sk)
 {
 	struct sock *sk;
 
 	BT_DBG("channel %u len %d", channel, skb->len);
 
-	read_lock(&hci_sk_list.lock);
-
 	sk_for_each(sk, &hci_sk_list.head) {
 		struct sk_buff *nskb;
 
@@ -285,6 +283,13 @@ void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
 			kfree_skb(nskb);
 	}
 
+}
+
+void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
+			 int flag, struct sock *skip_sk)
+{
+	read_lock(&hci_sk_list.lock);
+	__hci_send_to_channel(channel, skb, flag, skip_sk);
 	read_unlock(&hci_sk_list.lock);
 }
 
@@ -388,8 +393,8 @@ void hci_send_monitor_ctrl_event(struct hci_dev *hdev, u16 event,
 		hdr->index = index;
 		hdr->len = cpu_to_le16(skb->len - HCI_MON_HDR_SIZE);
 
-		hci_send_to_channel(HCI_CHANNEL_MONITOR, skb,
-				    HCI_SOCK_TRUSTED, NULL);
+		__hci_send_to_channel(HCI_CHANNEL_MONITOR, skb,
+				      HCI_SOCK_TRUSTED, NULL);
 		kfree_skb(skb);
 	}
 
-- 
2.14.1

Re: [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel()

[Mart van de Wege]

Confirm that 4.11.12-rt14 with this fix boots normally.

On Thu, 21 Sep 2017 15:51:23 +0200
Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote:

> Mart reported a deadlock in -RT in the call path:
>   hci_send_monitor_ctrl_event() -> hci_send_to_channel()
> 
> because both functions acquire the same read lock hci_sk_list.lock.
> This is also a mainline issue because the qrwlock implementation is
> writer fair (the traditional rwlock implementation is reader biased).
> 
> To avoid the deadlock there is now __hci_send_to_channel() which
> expects the readlock to be held.
> 
> Cc: Marcel Holtmann <marcel@holtmann.org>
> Cc: Johan Hedberg <johan.hedberg@intel.com>
> Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT
> commands and events to monitor") Reported-by: Mart van de Wege
> <mvdwege@gmail.com> Signed-off-by: Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> ---
>  net/bluetooth/hci_sock.c | 17 +++++++++++------
>  1 file changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
> index 638bf0e1a2e3..f1ee820d871c 100644
> --- a/net/bluetooth/hci_sock.c
> +++ b/net/bluetooth/hci_sock.c
> @@ -251,15 +251,13 @@ void hci_send_to_sock(struct hci_dev *hdev,
> struct sk_buff *skb) }
>  
>  /* Send frame to sockets with specific channel */
> -void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
> -			 int flag, struct sock *skip_sk)
> +static void __hci_send_to_channel(unsigned short channel, struct
> sk_buff *skb,
> +				  int flag, struct sock *skip_sk)
>  {
>  	struct sock *sk;
>  
>  	BT_DBG("channel %u len %d", channel, skb->len);
>  
> -	read_lock(&hci_sk_list.lock);
> -
>  	sk_for_each(sk, &hci_sk_list.head) {
>  		struct sk_buff *nskb;
>  
> @@ -285,6 +283,13 @@ void hci_send_to_channel(unsigned short channel,
> struct sk_buff *skb, kfree_skb(nskb);
>  	}
>  
> +}
> +
> +void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
> +			 int flag, struct sock *skip_sk)
> +{
> +	read_lock(&hci_sk_list.lock);
> +	__hci_send_to_channel(channel, skb, flag, skip_sk);
>  	read_unlock(&hci_sk_list.lock);
>  }
>  
> @@ -388,8 +393,8 @@ void hci_send_monitor_ctrl_event(struct hci_dev
> *hdev, u16 event, hdr->index = index;
>  		hdr->len = cpu_to_le16(skb->len - HCI_MON_HDR_SIZE);
>  
> -		hci_send_to_channel(HCI_CHANNEL_MONITOR, skb,
> -				    HCI_SOCK_TRUSTED, NULL);
> +		__hci_send_to_channel(HCI_CHANNEL_MONITOR, skb,
> +				      HCI_SOCK_TRUSTED, NULL);
>  		kfree_skb(skb);
>  	}
>  

Re: [PATCH] Bluetooth: avoid recursive locking in hci_send_to_channel()

[Marcel Holtmann]

Hi Sebastian,

> Mart reported a deadlock in -RT in the call path:
>  hci_send_monitor_ctrl_event() -> hci_send_to_channel()
> 
> because both functions acquire the same read lock hci_sk_list.lock. This
> is also a mainline issue because the qrwlock implementation is writer
> fair (the traditional rwlock implementation is reader biased).
> 
> To avoid the deadlock there is now __hci_send_to_channel() which expects
> the readlock to be held.
> 
> Cc: Marcel Holtmann <marcel@holtmann.org>
> Cc: Johan Hedberg <johan.hedberg@intel.com>
> Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor")
> Reported-by: Mart van de Wege <mvdwege@gmail.com>
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> ---
> net/bluetooth/hci_sock.c | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)

patch has been applied to bluetooth-next tree.

Regards

Marcel