Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 20 Jan 2009 10:16:38 -0800 (PST)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12515
> 
>            Summary: possible circular locking #0:  (sk_lock-AF_PACKET){--
>                     ..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4
>            Product: Networking
>            Version: 2.5
>      KernelVersion: 2.6.29-rc1-git4
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: IPV4
>         AssignedTo: shemminger@linux-foundation.org
>         ReportedBy: mmokrejs@ribosome.natur.cuni.cz
> 
> 
> Latest working kernel version:
> Earliest failing kernel version:
> Distribution: Gentoo Linux
> Hardware Environment: ASUS L3C/S laptop
> Software Environment:
> Problem Description:
> 
> Steps to reproduce: I have been reinstalling some apps while started tcpdump.
> It immediately hit the bug.
> 

More info at the link.  Vegard did some analysis, had a shot at fixing
it, but it seems that he missed.

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Martin MOKREJŠ]

There has been some discussion under subject
"Re: [PATCH] net: fix setsockopt() locking errors"
and I hope it is archived somewhere via netdev@vger.kernel.org .
If not, Jarek Poplawski and Peter Zijlstra have some clues
what to do and I am waiting for Vegard Nossum to give me
another patch to test.


Here is some part of the discussion ,hopefully the most important
part:

On Tue, Jan 27, 2009 at 09:52:49AM +0100, Peter Zijlstra wrote:
> > On Tue, 2009-01-27 at 08:45 +0000, Jarek Poplawski wrote:
>> > > On Mon, Jan 26, 2009 at 10:30:30PM +0100, Martin MOKREJŠ wrote:
>>> > > > The patch really did not help:
>>> > > > http://bugzilla.kernel.org/show_bug.cgi?id=12515#c5
>>> > > > Martin
>> > > 
>> > > Actually, there is a little change: the warning triggerd in another
>> > > place (sock_setsockopt() -> sk_attach_filter()). So we could go deeper
>> > > with these changes, but I'm not sure this is the right way to fix.
>> > > 
>> > > It looks like the scenario is very old, but probably wasn't reported
>> > > (maybe there is some lockdep improvement):
> > 
> > Yes, they likely are very old, and yes we added a lockdep annotation to
> > copy_to/from_user() to catch these.
> > 
>> > > A) sys_mmap2() -> mm->mmap_sem -> packet_mmap() -> sk_lock
>> > > B) sock_setsockopt() -> sk_lock -> copy_from_user() -> mm->mmap_sem
>> > > 
>> > > packet_mmap() (net/packet/af_packet.c) seems to be the only place in
>> > > net to implement mmap method, and using this lock order btw. On the
>> > > other hand copy_from_user() could be more popular under sk_lock, and
>> > > I'm not sure these changes are necessary.
>> > > 
>> > > Since I don't know enough neither sock/packet nor sys_mmap, I guess
>> > > some advice would be precious. It looks like Peter Zijlstra solved
>> > > similar problems in nfs, so I CC him.
> > 
> > The NFS/sunrpc case was special in that it did copy_to/from_kernel, that
> > is, it never actually touched user memory -- we taught the might_fault()
> > annotation about that.
> > 
> > Can't you simply do the copy_from_user() before you take the sk_lock?
> > 

Since it's really needed, and Vegard started doing it like this, I
guess he will try to add the missing pieces.

Thanks again,
Jarek P.





Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Tue, 20 Jan 2009 10:16:38 -0800 (PST)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
>> http://bugzilla.kernel.org/show_bug.cgi?id=12515
>>
>>            Summary: possible circular locking #0:  (sk_lock-AF_PACKET){--
>>                     ..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4
>>            Product: Networking
>>            Version: 2.5
>>      KernelVersion: 2.6.29-rc1-git4
>>           Platform: All
>>         OS/Version: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: IPV4
>>         AssignedTo: shemminger@linux-foundation.org
>>         ReportedBy: mmokrejs@ribosome.natur.cuni.cz
>>
>>
>> Latest working kernel version:
>> Earliest failing kernel version:
>> Distribution: Gentoo Linux
>> Hardware Environment: ASUS L3C/S laptop
>> Software Environment:
>> Problem Description:
>>
>> Steps to reproduce: I have been reinstalling some apps while started tcpdump.
>> It immediately hit the bug.
>>
> 
> More info at the link.  Vegard did some analysis, had a shot at fixing
> it, but it seems that he missed.

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Martin MOKREJŠ]

Martin MOKREJŠ wrote:
> There has been some discussion under subject
> "Re: [PATCH] net: fix setsockopt() locking errors"
> and I hope it is archived somewhere via netdev@vger.kernel.org .
> If not, Jarek Poplawski and Peter Zijlstra have some clues
> what to do and I am waiting for Vegard Nossum to give me
> another patch to test.

I have tested the patch below in bugzilla and cannot reproduce anymore.
dmesg(1) output in bugzilla.

Martin


------- Comment #8 from vegard.nossum@gmail.com  2009-01-27 14:38 -------
Created an attachment (id=20018)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=20018&action=view)
[PATCH] net: fix setsockopt() locking errors #2

Andrew, I just missed a case because the actual call to copy_from_user was in a
different file altogether.

There might be more, but I didn't really have the time to look for everything
yet.

In the meantime, I give you this patch, which includes the SO_ATTACH_FILTER
socket option, which is what was reported to fail with the first patch applied.

I also don't REALLY think this is a regression, it's just lockdep that got
smarter (or the insertion of a might_fault() something something made it more
likely to show up -- I think this was info from Peter, but I don't remember
accurately).



> 
> 
> Here is some part of the discussion ,hopefully the most important
> part:
> 
> On Tue, Jan 27, 2009 at 09:52:49AM +0100, Peter Zijlstra wrote:
>>> On Tue, 2009-01-27 at 08:45 +0000, Jarek Poplawski wrote:
>>>>> On Mon, Jan 26, 2009 at 10:30:30PM +0100, Martin MOKREJŠ wrote:
>>>>>>> The patch really did not help:
>>>>>>> http://bugzilla.kernel.org/show_bug.cgi?id=12515#c5
>>>>>>> Martin
>>>>> Actually, there is a little change: the warning triggerd in another
>>>>> place (sock_setsockopt() -> sk_attach_filter()). So we could go deeper
>>>>> with these changes, but I'm not sure this is the right way to fix.
>>>>>
>>>>> It looks like the scenario is very old, but probably wasn't reported
>>>>> (maybe there is some lockdep improvement):
>>> Yes, they likely are very old, and yes we added a lockdep annotation to
>>> copy_to/from_user() to catch these.
>>>
>>>>> A) sys_mmap2() -> mm->mmap_sem -> packet_mmap() -> sk_lock
>>>>> B) sock_setsockopt() -> sk_lock -> copy_from_user() -> mm->mmap_sem
>>>>>
>>>>> packet_mmap() (net/packet/af_packet.c) seems to be the only place in
>>>>> net to implement mmap method, and using this lock order btw. On the
>>>>> other hand copy_from_user() could be more popular under sk_lock, and
>>>>> I'm not sure these changes are necessary.
>>>>>
>>>>> Since I don't know enough neither sock/packet nor sys_mmap, I guess
>>>>> some advice would be precious. It looks like Peter Zijlstra solved
>>>>> similar problems in nfs, so I CC him.
>>> The NFS/sunrpc case was special in that it did copy_to/from_kernel, that
>>> is, it never actually touched user memory -- we taught the might_fault()
>>> annotation about that.
>>>
>>> Can't you simply do the copy_from_user() before you take the sk_lock?
>>>
> 
> Since it's really needed, and Vegard started doing it like this, I
> guess he will try to add the missing pieces.
> 
> Thanks again,
> Jarek P.
> 
> 
> 
> 
> 
> Andrew Morton wrote:
>> (switched to email.  Please respond via emailed reply-to-all, not via the
>> bugzilla web interface).
>>
>> On Tue, 20 Jan 2009 10:16:38 -0800 (PST)
>> bugme-daemon@bugzilla.kernel.org wrote:
>>
>>> http://bugzilla.kernel.org/show_bug.cgi?id=12515
>>>
>>>            Summary: possible circular locking #0:  (sk_lock-AF_PACKET){--
>>>                     ..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4
>>>            Product: Networking
>>>            Version: 2.5
>>>      KernelVersion: 2.6.29-rc1-git4
>>>           Platform: All
>>>         OS/Version: Linux
>>>               Tree: Mainline
>>>             Status: NEW
>>>           Severity: normal
>>>           Priority: P1
>>>          Component: IPV4
>>>         AssignedTo: shemminger@linux-foundation.org
>>>         ReportedBy: mmokrejs@ribosome.natur.cuni.cz
>>>
>>>
>>> Latest working kernel version:
>>> Earliest failing kernel version:
>>> Distribution: Gentoo Linux
>>> Hardware Environment: ASUS L3C/S laptop
>>> Software Environment:
>>> Problem Description:
>>>
>>> Steps to reproduce: I have been reinstalling some apps while started tcpdump.
>>> It immediately hit the bug.
>>>
>> More info at the link.  Vegard did some analysis, had a shot at fixing
>> it, but it seems that he missed.

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Herbert Xu]

On Fri, Jan 30, 2009 at 05:12:50PM +1100, Herbert Xu wrote:
> 
> Well, doing the copy under sk_lock is pretty common through all
> protocols.  So I think it'd be safer to change the other path,
> which is doing the odd thing here, i.e., ->mmap() grabbing the
> socket lock while holding mmap_sem.
> 
> In fact, it would appear that we don't really need the socket lock
> in ->mmap() since it only needs to ensure that pg_vec* doesn't
> get yanked or changed.  So this patch should work:
> 
> packet: Avoid lock_sock in mmap handler

Dave pointed out that a spin lock is illegal for this purpose
as vm_insert_page can do a GFP_KERNEL allocation.  So I've added
a mutex for this.

I've also widened the critical section in packet_set_ring since
we need the mapped check to be within it.

packet: Avoid lock_sock in mmap handler

As the mmap handler gets called under mmap_sem, and we may grab
mmap_sem elsewhere under the socket lock to access user data, we
should avoid grabbing the socket lock in the mmap handler.

Since the only thing we care about in the mmap handler is for
pg_vec* to be invariant, i.e., to exclude packet_set_ring, we
can achieve this by simply using sk_receive_queue.lock.

I resisted the temptation to create a new spin lock because the
mmap path isn't exactly common.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 5f94db2..9454d4a 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -77,6 +77,7 @@
 #include <linux/poll.h>
 #include <linux/module.h>
 #include <linux/init.h>
+#include <linux/mutex.h>
 
 #ifdef CONFIG_INET
 #include <net/inet_common.h>
@@ -175,6 +176,7 @@ struct packet_sock {
 #endif
 	struct packet_type	prot_hook;
 	spinlock_t		bind_lock;
+	struct mutex		pg_vec_lock;
 	unsigned int		running:1,	/* prot_hook is attached*/
 				auxdata:1,
 				origdev:1;
@@ -1069,6 +1071,7 @@ static int packet_create(struct net *net, struct socket *sock, int protocol)
 	 */
 
 	spin_lock_init(&po->bind_lock);
+	mutex_init(&po->pg_vec_lock);
 	po->prot_hook.func = packet_rcv;
 
 	if (sock->type == SOCK_PACKET)
@@ -1865,6 +1868,7 @@ static int packet_set_ring(struct sock *sk, struct tpacket_req *req, int closing
 	synchronize_net();
 
 	err = -EBUSY;
+	mutex_lock(&po->pg_vec_lock);
 	if (closing || atomic_read(&po->mapped) == 0) {
 		err = 0;
 #define XC(a, b) ({ __typeof__ ((a)) __t; __t = (a); (a) = (b); __t; })
@@ -1886,6 +1890,7 @@ static int packet_set_ring(struct sock *sk, struct tpacket_req *req, int closing
 		if (atomic_read(&po->mapped))
 			printk(KERN_DEBUG "packet_mmap: vma is busy: %d\n", atomic_read(&po->mapped));
 	}
+	mutex_unlock(&po->pg_vec_lock);
 
 	spin_lock(&po->bind_lock);
 	if (was_running && !po->running) {
@@ -1918,7 +1923,7 @@ static int packet_mmap(struct file *file, struct socket *sock, struct vm_area_st
 
 	size = vma->vm_end - vma->vm_start;
 
-	lock_sock(sk);
+	mutex_lock(&po->pg_vec_lock);
 	if (po->pg_vec == NULL)
 		goto out;
 	if (size != po->pg_vec_len*po->pg_vec_pages*PAGE_SIZE)
@@ -1941,7 +1946,7 @@ static int packet_mmap(struct file *file, struct socket *sock, struct vm_area_st
 	err = 0;
 
 out:
-	release_sock(sk);
+	mutex_unlock(&po->pg_vec_lock);
 	return err;
 }
 #endif

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Jarek Poplawski]

On Fri, Jan 30, 2009 at 11:49:47PM +1100, Herbert Xu wrote:
...
> Dave pointed out that a spin lock is illegal for this purpose
> as vm_insert_page can do a GFP_KERNEL allocation.  So I've added
> a mutex for this.
...
> I resisted the temptation to create a new spin lock because the
> mmap path isn't exactly common.

So, Dave is stronger than the temptation...

Jarek P.

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Martin MOKREJŠ]

This patch works for me.
Thanks
M.

Herbert Xu wrote:
> On Fri, Jan 30, 2009 at 05:12:50PM +1100, Herbert Xu wrote:
>> Well, doing the copy under sk_lock is pretty common through all
>> protocols.  So I think it'd be safer to change the other path,
>> which is doing the odd thing here, i.e., ->mmap() grabbing the
>> socket lock while holding mmap_sem.
>>
>> In fact, it would appear that we don't really need the socket lock
>> in ->mmap() since it only needs to ensure that pg_vec* doesn't
>> get yanked or changed.  So this patch should work:
>>
>> packet: Avoid lock_sock in mmap handler
> 
> Dave pointed out that a spin lock is illegal for this purpose
> as vm_insert_page can do a GFP_KERNEL allocation.  So I've added
> a mutex for this.
> 
> I've also widened the critical section in packet_set_ring since
> we need the mapped check to be within it.
> 
> packet: Avoid lock_sock in mmap handler
> 
> As the mmap handler gets called under mmap_sem, and we may grab
> mmap_sem elsewhere under the socket lock to access user data, we
> should avoid grabbing the socket lock in the mmap handler.
> 
> Since the only thing we care about in the mmap handler is for
> pg_vec* to be invariant, i.e., to exclude packet_set_ring, we
> can achieve this by simply using sk_receive_queue.lock.
> 
> I resisted the temptation to create a new spin lock because the
> mmap path isn't exactly common.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
> 
> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 5f94db2..9454d4a 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -77,6 +77,7 @@
>  #include <linux/poll.h>
>  #include <linux/module.h>
>  #include <linux/init.h>
> +#include <linux/mutex.h>
>  
>  #ifdef CONFIG_INET
>  #include <net/inet_common.h>
> @@ -175,6 +176,7 @@ struct packet_sock {
>  #endif
>  	struct packet_type	prot_hook;
>  	spinlock_t		bind_lock;
> +	struct mutex		pg_vec_lock;
>  	unsigned int		running:1,	/* prot_hook is attached*/
>  				auxdata:1,
>  				origdev:1;
> @@ -1069,6 +1071,7 @@ static int packet_create(struct net *net, struct socket *sock, int protocol)
>  	 */
>  
>  	spin_lock_init(&po->bind_lock);
> +	mutex_init(&po->pg_vec_lock);
>  	po->prot_hook.func = packet_rcv;
>  
>  	if (sock->type == SOCK_PACKET)
> @@ -1865,6 +1868,7 @@ static int packet_set_ring(struct sock *sk, struct tpacket_req *req, int closing
>  	synchronize_net();
>  
>  	err = -EBUSY;
> +	mutex_lock(&po->pg_vec_lock);
>  	if (closing || atomic_read(&po->mapped) == 0) {
>  		err = 0;
>  #define XC(a, b) ({ __typeof__ ((a)) __t; __t = (a); (a) = (b); __t; })
> @@ -1886,6 +1890,7 @@ static int packet_set_ring(struct sock *sk, struct tpacket_req *req, int closing
>  		if (atomic_read(&po->mapped))
>  			printk(KERN_DEBUG "packet_mmap: vma is busy: %d\n", atomic_read(&po->mapped));
>  	}
> +	mutex_unlock(&po->pg_vec_lock);
>  
>  	spin_lock(&po->bind_lock);
>  	if (was_running && !po->running) {
> @@ -1918,7 +1923,7 @@ static int packet_mmap(struct file *file, struct socket *sock, struct vm_area_st
>  
>  	size = vma->vm_end - vma->vm_start;
>  
> -	lock_sock(sk);
> +	mutex_lock(&po->pg_vec_lock);
>  	if (po->pg_vec == NULL)
>  		goto out;
>  	if (size != po->pg_vec_len*po->pg_vec_pages*PAGE_SIZE)
> @@ -1941,7 +1946,7 @@ static int packet_mmap(struct file *file, struct socket *sock, struct vm_area_st
>  	err = 0;
>  
>  out:
> -	release_sock(sk);
> +	mutex_unlock(&po->pg_vec_lock);
>  	return err;
>  }
>  #endif

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[Herbert Xu]

On Fri, Jan 30, 2009 at 01:56:17PM +0000, Jarek Poplawski wrote:
>
> > I resisted the temptation to create a new spin lock because the
> > mmap path isn't exactly common.
> 
> So, Dave is stronger than the temptation...

Heh, that sentence should be removed from the description.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[David Miller]

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Sat, 31 Jan 2009 09:01:28 +1100

> On Fri, Jan 30, 2009 at 01:56:17PM +0000, Jarek Poplawski wrote:
> >
> > > I resisted the temptation to create a new spin lock because the
> > > mmap path isn't exactly common.
> > 
> > So, Dave is stronger than the temptation...
> 
> Heh, that sentence should be removed from the description.

I might leave it in there for laughs :-)

Re: [Bugme-new] [Bug 12515] New: possible circular locking #0: (sk_lock-AF_PACKET){--..}, at: [<c1279838>] sock_setsockopt+0x12b/0x4a4

[David Miller]

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Fri, 30 Jan 2009 23:49:47 +1100

> packet: Avoid lock_sock in mmap handler
> 
> As the mmap handler gets called under mmap_sem, and we may grab
> mmap_sem elsewhere under the socket lock to access user data, we
> should avoid grabbing the socket lock in the mmap handler.
> 
> Since the only thing we care about in the mmap handler is for
> pg_vec* to be invariant, i.e., to exclude packet_set_ring, we
> can achieve this by simply using sk_receive_queue.lock.
> 
> I resisted the temptation to create a new spin lock because the
> mmap path isn't exactly common.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Looks good, applied.

I rewrote the commit message as follows so that it actually
matches what happens in the fix ;-)

--------------------
packet: Avoid lock_sock in mmap handler

As the mmap handler gets called under mmap_sem, and we may grab
mmap_sem elsewhere under the socket lock to access user data, we
should avoid grabbing the socket lock in the mmap handler.

Since the only thing we care about in the mmap handler is for
pg_vec* to be invariant, i.e., to exclude packet_set_ring, we
can achieve this by simply using a new mutex.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: David S. Miller <davem@davemloft.net>