net-next/unix: BUG: using smp_processor_id() in preemptible

net-next/unix: BUG: using smp_processor_id() in preemptible

[Ilpo Järvinen]

I got plenty of these from sock_prot_inuse_add:

BUG: using smp_processor_id() in preemptible [00000000] code: rcS/1146

caller is sock_prot_inuse_add+0x24/0x42
Pid: 1146, comm: rcS Not tainted 2.6.28-rc6-01121-g89a2f15 #76
Call Trace:
 [<ffffffff80365767>] debug_smp_processor_id+0xd3/0xe8
 [<ffffffff80485a0f>] sock_prot_inuse_add+0x24/0x42
 [<ffffffff80514cdd>] unix_create1+0x161/0x176
 [<ffffffff80514d52>] unix_create+0x60/0x6b
 [<ffffffff80483f66>] __sock_create+0x144/0x1bd
 [<ffffffff80483edb>] ? __sock_create+0xb9/0x1bd
 [<ffffffff8048402d>] sock_create+0x2d/0x2f
 [<ffffffff80484274>] sys_socket+0x29/0x5b
 [<ffffffff8020c10a>] system_call_fastpath+0x16/0x1b


-- 
 i.

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[Eric Dumazet]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

Ilpo Järvinen a écrit :
> I got plenty of these from sock_prot_inuse_add:
> 
> BUG: using smp_processor_id() in preemptible [00000000] code: rcS/1146
> 
> caller is sock_prot_inuse_add+0x24/0x42
> Pid: 1146, comm: rcS Not tainted 2.6.28-rc6-01121-g89a2f15 #76
> Call Trace:
>  [<ffffffff80365767>] debug_smp_processor_id+0xd3/0xe8
>  [<ffffffff80485a0f>] sock_prot_inuse_add+0x24/0x42
>  [<ffffffff80514cdd>] unix_create1+0x161/0x176
>  [<ffffffff80514d52>] unix_create+0x60/0x6b
>  [<ffffffff80483f66>] __sock_create+0x144/0x1bd
>  [<ffffffff80483edb>] ? __sock_create+0xb9/0x1bd
>  [<ffffffff8048402d>] sock_create+0x2d/0x2f
>  [<ffffffff80484274>] sys_socket+0x29/0x5b
>  [<ffffffff8020c10a>] system_call_fastpath+0x16/0x1b
> 
> 

Thanks Ilpo for this report

I guess the following is necessary.

Once Christopher and Rusty work on percpu variables is finished,
the preempt_enable()/disable() wont be necessary anymore, so
its a temporary workaround anyway.

[PATCH] net: make sock_prot_inuse_add() preempt safe

Ilpo Järvinen reported that commit a8076d8db98de6da61394b2e942320e4612643ac
(net: af_unix should update its inuse counter) was triggering
a warning in smp_processor_id(), being called in a preemptible code.

Fix is to make sock_prot_inuse_add() safe in this regard. This fix
can be reverted when new percpu infrastructure is ready, allowing
a cpu to safely do a increment/decrement on a percpu var.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

[-- Attachment #2: prot_inuse.patch --]
[-- Type: text/plain, Size: 822 bytes --]

diff --git a/net/core/sock.c b/net/core/sock.c
index a4e840e..d4f5ad7 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1942,8 +1942,8 @@ static DECLARE_BITMAP(proto_inuse_idx, PROTO_INUSE_NR);
 #ifdef CONFIG_NET_NS
 void sock_prot_inuse_add(struct net *net, struct proto *prot, int val)
 {
-	int cpu = smp_processor_id();
-	per_cpu_ptr(net->core.inuse, cpu)->val[prot->inuse_idx] += val;
+	per_cpu_ptr(net->core.inuse, get_cpu())->val[prot->inuse_idx] += val;
+	put_cpu();
 }
 EXPORT_SYMBOL_GPL(sock_prot_inuse_add);
 
@@ -1989,7 +1989,9 @@ static DEFINE_PER_CPU(struct prot_inuse, prot_inuse);
 
 void sock_prot_inuse_add(struct net *net, struct proto *prot, int val)
 {
+	preempt_disable();
 	__get_cpu_var(prot_inuse).val[prot->inuse_idx] += val;
+	preempt_enable();
 }
 EXPORT_SYMBOL_GPL(sock_prot_inuse_add);
 

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1656 bytes --]

On Sun, 23 Nov 2008, Eric Dumazet wrote:

> Ilpo Järvinen a écrit :
> > I got plenty of these from sock_prot_inuse_add:
> > 
> > BUG: using smp_processor_id() in preemptible [00000000] code: rcS/1146
> > 
> > caller is sock_prot_inuse_add+0x24/0x42
> > Pid: 1146, comm: rcS Not tainted 2.6.28-rc6-01121-g89a2f15 #76
> > Call Trace:
> >  [<ffffffff80365767>] debug_smp_processor_id+0xd3/0xe8
> >  [<ffffffff80485a0f>] sock_prot_inuse_add+0x24/0x42
> >  [<ffffffff80514cdd>] unix_create1+0x161/0x176
> >  [<ffffffff80514d52>] unix_create+0x60/0x6b
> >  [<ffffffff80483f66>] __sock_create+0x144/0x1bd
> >  [<ffffffff80483edb>] ? __sock_create+0xb9/0x1bd
> >  [<ffffffff8048402d>] sock_create+0x2d/0x2f
> >  [<ffffffff80484274>] sys_socket+0x29/0x5b
> >  [<ffffffff8020c10a>] system_call_fastpath+0x16/0x1b
> > 
> > 
> 
> Thanks Ilpo for this report
> 
> I guess the following is necessary.
> 
> Once Christopher and Rusty work on percpu variables is finished,
> the preempt_enable()/disable() wont be necessary anymore, so
> its a temporary workaround anyway.
> 
> [PATCH] net: make sock_prot_inuse_add() preempt safe
> 
> Ilpo Järvinen reported that commit a8076d8db98de6da61394b2e942320e4612643ac
> (net: af_unix should update its inuse counter) was triggering
> a warning in smp_processor_id(), being called in a preemptible code.
> 
> Fix is to make sock_prot_inuse_add() safe in this regard. This fix
> can be reverted when new percpu infrastructure is ready, allowing
> a cpu to safely do a increment/decrement on a percpu var.
> 
> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

Works.

Tested-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>

-- 
 i.

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Sun, 23 Nov 2008 04:32:30 +0100

> [PATCH] net: make sock_prot_inuse_add() preempt safe
> 
> Ilpo Järvinen reported that commit a8076d8db98de6da61394b2e942320e4612643ac
> (net: af_unix should update its inuse counter) was triggering
> a warning in smp_processor_id(), being called in a preemptible code.
> 
> Fix is to make sock_prot_inuse_add() safe in this regard. This fix
> can be reverted when new percpu infrastructure is ready, allowing
> a cpu to safely do a increment/decrement on a percpu var.
> 
> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

Eric, you added this bug by starting to use this interface in
situations where BH's were not disabled.

Ever existing use adhered to that rule.

If you therefore want to call this interface in new locations,
you have to make sure those locations follow the rule too.

Making it expensive for all the existing cases which were already
safe, is not the way to fix this.  And saying some future not-merged
change justifies this added cost is besides the point.

What is valid is getting rid of the BH disables you need to add
to net/unix/af_unix.c et al.'s call sites, once those percpu patches
are added.  Not the other way around.

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Sun, 23 Nov 2008 17:20:14 -0800 (PST)

> From: Eric Dumazet <dada1@cosmosbay.com>
> Date: Sun, 23 Nov 2008 04:32:30 +0100
> 
> > [PATCH] net: make sock_prot_inuse_add() preempt safe
 ...
> Eric, you added this bug by starting to use this interface in
> situations where BH's were not disabled.
> 
> Ever existing use adhered to that rule.
> 
> If you therefore want to call this interface in new locations,
> you have to make sure those locations follow the rule too.

Here is what I commited to fix this bug.

net: Make sure BHs are disabled in sock_prot_inuse_add()

The rule of calling sock_prot_inuse_add() is that BHs must
be disabled.  Some new calls were added where this was not
true and this tiggers warnings as reported by Ilpo.

Fix this by adding explicit BH disabling around those call sites.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 net/netlink/af_netlink.c |    3 +++
 net/sctp/socket.c        |    4 ++++
 net/unix/af_unix.c       |    2 ++
 3 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index a2071dc..c7d7657 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -452,7 +452,10 @@ static int netlink_create(struct net *net, struct socket *sock, int protocol)
 	if (err < 0)
 		goto out_module;
 
+	local_bh_disable();
 	sock_prot_inuse_add(net, &netlink_proto, 1);
+	local_bh_enable();
+
 	nlk = nlk_sk(sock->sk);
 	nlk->module = module;
 out:
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index 0c70eff..f03af84 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -3614,7 +3614,11 @@ SCTP_STATIC int sctp_init_sock(struct sock *sk)
 
 	SCTP_DBG_OBJCNT_INC(sock);
 	atomic_inc(&sctp_sockets_allocated);
+
+	local_bh_disable();
 	sock_prot_inuse_add(sock_net(sk), sk->sk_prot, 1);
+	local_bh_enable();
+
 	return 0;
 }
 
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index e1ca8f7..a45a9f7 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -361,7 +361,9 @@ static void unix_sock_destructor(struct sock *sk)
 		unix_release_addr(u->addr);
 
 	atomic_dec(&unix_nr_socks);
+	local_bh_disable();
 	sock_prot_inuse_add(sock_net(sk), sk->sk_prot, -1);
+	local_bh_enable();
 #ifdef UNIX_REFCNT_DEBUG
 	printk(KERN_DEBUG "UNIX %p is destroyed, %d are still alive.\n", sk,
 		atomic_read(&unix_nr_socks));
-- 
1.5.6.5

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[Eric Dumazet]

David Miller a écrit :
> From: David Miller <davem@davemloft.net>
> Date: Sun, 23 Nov 2008 17:20:14 -0800 (PST)
> 
>> From: Eric Dumazet <dada1@cosmosbay.com>
>> Date: Sun, 23 Nov 2008 04:32:30 +0100
>>
>>> [PATCH] net: make sock_prot_inuse_add() preempt safe
>  ...
>> Eric, you added this bug by starting to use this interface in
>> situations where BH's were not disabled.
>>
>> Ever existing use adhered to that rule.
>>
>> If you therefore want to call this interface in new locations,
>> you have to make sure those locations follow the rule too.
> 
> Here is what I commited to fix this bug.
> 
> net: Make sure BHs are disabled in sock_prot_inuse_add()
> 
> The rule of calling sock_prot_inuse_add() is that BHs must
> be disabled.  Some new calls were added where this was not
> true and this tiggers warnings as reported by Ilpo.
> 
> Fix this by adding explicit BH disabling around those call sites.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
>  net/netlink/af_netlink.c |    3 +++
>  net/sctp/socket.c        |    4 ++++
>  net/unix/af_unix.c       |    2 ++
>  3 files changed, 9 insertions(+), 0 deletions(-)

Hello David

Thanks for working on this during my sleep ;)

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[Eric Dumazet]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

David Miller a écrit :
> From: David Miller <davem@davemloft.net>
> Date: Sun, 23 Nov 2008 17:20:14 -0800 (PST)
> 
>> From: Eric Dumazet <dada1@cosmosbay.com>
>> Date: Sun, 23 Nov 2008 04:32:30 +0100
>>
>>> [PATCH] net: make sock_prot_inuse_add() preempt safe
>  ...
>> Eric, you added this bug by starting to use this interface in
>> situations where BH's were not disabled.
>>
>> Ever existing use adhered to that rule.
>>
>> If you therefore want to call this interface in new locations,
>> you have to make sure those locations follow the rule too.
> 
> Here is what I commited to fix this bug.
> 
> net: Make sure BHs are disabled in sock_prot_inuse_add()
> 
> The rule of calling sock_prot_inuse_add() is that BHs must
> be disabled.  Some new calls were added where this was not
> true and this tiggers warnings as reported by Ilpo.
> 
> Fix this by adding explicit BH disabling around those call sites.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
>  net/netlink/af_netlink.c |    3 +++
>  net/sctp/socket.c        |    4 ++++
>  net/unix/af_unix.c       |    2 ++
>  3 files changed, 9 insertions(+), 0 deletions(-)

I believe some bits are missing

Ilpo report was about unix_create1() being preemptable for example

Thanks

[PATCH] net: Make sure BHs are disabled in sock_prot_inuse_add()

The rule of calling sock_prot_inuse_add() is that BHs must
be disabled.  Some new calls were added where this was not
true and this tiggers warnings as reported by Ilpo.

Fix this by adding explicit BH disabling around those call sites,
or moving sock_prot_inuse_add() call inside an existing BH disabled
section.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
---

 net/ipv4/inet_hashtables.c |    2 +-
 net/packet/af_packet.c     |    4 ++--
 net/unix/af_unix.c         |    6 ++++--
 3 files changed, 7 insertions(+), 5 deletions(-)

[-- Attachment #2: prot_inuse.patch --]
[-- Type: text/plain, Size: 1810 bytes --]

diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
index 11fcb87..6a1045d 100644
--- a/net/ipv4/inet_hashtables.c
+++ b/net/ipv4/inet_hashtables.c
@@ -402,9 +402,9 @@ void inet_unhash(struct sock *sk)
 
 	spin_lock_bh(lock);
 	done =__sk_nulls_del_node_init_rcu(sk);
-	spin_unlock_bh(lock);
 	if (done)
 		sock_prot_inuse_add(sock_net(sk), sk->sk_prot, -1);
+	spin_unlock_bh(lock);
 }
 EXPORT_SYMBOL_GPL(inet_unhash);
 
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index b4870a3..5f94db2 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -872,6 +872,7 @@ static int packet_release(struct socket *sock)
 
 	write_lock_bh(&net->packet.sklist_lock);
 	sk_del_node_init(sk);
+	sock_prot_inuse_add(net, sk->sk_prot, -1);
 	write_unlock_bh(&net->packet.sklist_lock);
 
 	/*
@@ -910,7 +911,6 @@ static int packet_release(struct socket *sock)
 	skb_queue_purge(&sk->sk_receive_queue);
 	sk_refcnt_debug_release(sk);
 
-	sock_prot_inuse_add(net, sk->sk_prot, -1);
 	sock_put(sk);
 	return 0;
 }
@@ -1085,8 +1085,8 @@ static int packet_create(struct net *net, struct socket *sock, int protocol)
 
 	write_lock_bh(&net->packet.sklist_lock);
 	sk_add_node(sk, &net->packet.sklist);
-	write_unlock_bh(&net->packet.sklist_lock);
 	sock_prot_inuse_add(net, &packet_proto, 1);
+	write_unlock_bh(&net->packet.sklist_lock);
 	return(0);
 out:
 	return err;
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index a45a9f7..3a35a6e 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -615,9 +615,11 @@ static struct sock *unix_create1(struct net *net, struct socket *sock)
 out:
 	if (sk == NULL)
 		atomic_dec(&unix_nr_socks);
-	else
+	else {
+		local_bh_disable();
 		sock_prot_inuse_add(sock_net(sk), sk->sk_prot, 1);
-
+		local_bh_enable();
+	}
 	return sk;
 }
 

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Mon, 24 Nov 2008 09:01:49 +0100

> David Miller a écrit :
> > From: David Miller <davem@davemloft.net>
> > Date: Sun, 23 Nov 2008 17:20:14 -0800 (PST)
> > 
> >> From: Eric Dumazet <dada1@cosmosbay.com>
> >> Date: Sun, 23 Nov 2008 04:32:30 +0100
> >>
> >>> [PATCH] net: make sock_prot_inuse_add() preempt safe
> >  ...
> >> Eric, you added this bug by starting to use this interface in
> >> situations where BH's were not disabled.
> >>
> >> Ever existing use adhered to that rule.
> >>
> >> If you therefore want to call this interface in new locations,
> >> you have to make sure those locations follow the rule too.
> > Here is what I commited to fix this bug.
> > net: Make sure BHs are disabled in sock_prot_inuse_add()
> > The rule of calling sock_prot_inuse_add() is that BHs must
> > be disabled.  Some new calls were added where this was not
> > true and this tiggers warnings as reported by Ilpo.
> > Fix this by adding explicit BH disabling around those call sites.
> > Signed-off-by: David S. Miller <davem@davemloft.net>
> > ---
> >  net/netlink/af_netlink.c |    3 +++
> >  net/sctp/socket.c        |    4 ++++
> >  net/unix/af_unix.c       |    2 ++
> >  3 files changed, 9 insertions(+), 0 deletions(-)
> 
> I believe some bits are missing
> 
> Ilpo report was about unix_create1() being preemptable for example

Thanks for catching that oversight, applied.

Re: net-next/unix: BUG: using smp_processor_id() in preemptible

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1727 bytes --]

On Mon, 24 Nov 2008, Eric Dumazet wrote:

> David Miller a écrit :
> > From: David Miller <davem@davemloft.net>
> > Date: Sun, 23 Nov 2008 17:20:14 -0800 (PST)
> > 
> > > From: Eric Dumazet <dada1@cosmosbay.com>
> > > Date: Sun, 23 Nov 2008 04:32:30 +0100
> > >
> > > > [PATCH] net: make sock_prot_inuse_add() preempt safe
> >  ...
> > > Eric, you added this bug by starting to use this interface in
> > > situations where BH's were not disabled.
> > >
> > > Ever existing use adhered to that rule.
> > >
> > > If you therefore want to call this interface in new locations,
> > > you have to make sure those locations follow the rule too.
> > 
> > Here is what I commited to fix this bug.
> > 
> > net: Make sure BHs are disabled in sock_prot_inuse_add()
> > 
> > The rule of calling sock_prot_inuse_add() is that BHs must
> > be disabled.  Some new calls were added where this was not
> > true and this tiggers warnings as reported by Ilpo.
> > 
> > Fix this by adding explicit BH disabling around those call sites.
> > 
> > Signed-off-by: David S. Miller <davem@davemloft.net>
> > ---
> >  net/netlink/af_netlink.c |    3 +++
> >  net/sctp/socket.c        |    4 ++++
> >  net/unix/af_unix.c       |    2 ++
> >  3 files changed, 9 insertions(+), 0 deletions(-)
> 
> I believe some bits are missing
> 
> Ilpo report was about unix_create1() being preemptable for example

Yes, basically these two sites I got (during boot, I didn't look much 
further if I could have gotten them from somewhere else too):

$ grep -A1 '[]] sock_prot_inuse_add[+]' dmesg.log  | grep "fffff" | grep -v sock_prot_inuse_add | sort | uniq
 [<ffffffff80514cdd>] unix_create1+0x161/0x176
 [<ffffffff805151de>] unix_sock_destructor+0xb6/0xbc


-- 
 i.