[PATCH v3.14-rt] netpoll: guard the access to dev->npinfo with rcu_read_lock/unlock_bh() for CONFIG_PREEMPT_RT_FULL=y

[PATCH v3.14-rt] netpoll: guard the access to dev->npinfo with rcu_read_lock/unlock_bh() for CONFIG_PREEMPT_RT_FULL=y

[Kevin Hao]

For vanilla kernel we don't need to invoke rcu_read_lock/unlock_bh
explicitly to mark an RCU-bh critical section in the softirq context
because bh is already disabled in this case. But for a rt kernel,
the commit ("rcu: Merge RCU-bh into RCU-preempt") implements the
RCU-bh in term of RCU-preempt. So we have to use
rcu_read_lock/unlock_bh() to mark an RCU-bh critical section even in
a softirq context. Otherwise we will get a call trace like this:
  include/linux/netpoll.h:90 suspicious rcu_dereference_check() usage!
  other info that might help us debug this:
  rcu_scheduler_active = 1, debug_locks = 0
  1 lock held by irq/177-eth0_g0/129:
   #0:  (&per_cpu(local_softirq_locks[i], __cpu).lock){+.+...}, at: [<8002f544>] do_current_softirqs+0x12c/0x5ec

  stack backtrace:
  CPU: 0 PID: 129 Comm: irq/177-eth0_g0 Not tainted 3.14.23 #11
  [<80018c0c>] (unwind_backtrace) from [<800138b0>] (show_stack+0x20/0x24)
  [<800138b0>] (show_stack) from [<8075c3bc>] (dump_stack+0x84/0xd0)
  [<8075c3bc>] (dump_stack) from [<8008111c>] (lockdep_rcu_suspicious+0xe8/0x11c)
  [<8008111c>] (lockdep_rcu_suspicious) from [<805e94e8>] (dev_gro_receive+0x240/0x724)
  [<805e94e8>] (dev_gro_receive) from [<805e9c34>] (napi_gro_receive+0x3c/0x1e8)
  [<805e9c34>] (napi_gro_receive) from [<804b01ac>] (gfar_clean_rx_ring+0x2d4/0x624)
  [<804b01ac>] (gfar_clean_rx_ring) from [<804b078c>] (gfar_poll_rx_sq+0x58/0xe8)
  [<804b078c>] (gfar_poll_rx_sq) from [<805eada8>] (net_rx_action+0x1c8/0x418)
  [<805eada8>] (net_rx_action) from [<8002f62c>] (do_current_softirqs+0x214/0x5ec)
  [<8002f62c>] (do_current_softirqs) from [<8002fa88>] (__local_bh_enable+0x84/0x9c)
  [<8002fa88>] (__local_bh_enable) from [<8002fab8>] (local_bh_enable+0x18/0x1c)
  [<8002fab8>] (local_bh_enable) from [<80093924>] (irq_forced_thread_fn+0x50/0x74)
  [<80093924>] (irq_forced_thread_fn) from [<80093c30>] (irq_thread+0x158/0x1c4)
  [<80093c30>] (irq_thread) from [<800555b8>] (kthread+0xd4/0xe8)
  [<800555b8>] (kthread) from [<8000ee88>] (ret_from_fork+0x14/0x20)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
---
 include/linux/netpoll.h | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/include/linux/netpoll.h b/include/linux/netpoll.h
index fbfdb9d8d3a7..039dd5df0ef7 100644
--- a/include/linux/netpoll.h
+++ b/include/linux/netpoll.h
@@ -87,9 +87,21 @@ static inline void netpoll_send_skb(struct netpoll *np, struct sk_buff *skb)
 #ifdef CONFIG_NETPOLL
 static inline bool netpoll_rx_on(struct sk_buff *skb)
 {
-	struct netpoll_info *npinfo = rcu_dereference_bh(skb->dev->npinfo);
+	struct netpoll_info *npinfo;
+	bool ret;
+
+#ifdef CONFIG_PREEMPT_RT_FULL
+	rcu_read_lock_bh();
+#endif
+
+	npinfo = rcu_dereference_bh(skb->dev->npinfo);
+	ret = npinfo && (!list_empty(&npinfo->rx_np) || npinfo->rx_flags);
+
+#ifdef CONFIG_PREEMPT_RT_FULL
+	rcu_read_unlock_bh();
+#endif
 
-	return npinfo && (!list_empty(&npinfo->rx_np) || npinfo->rx_flags);
+	return ret;
 }
 
 static inline bool netpoll_rx(struct sk_buff *skb)
-- 
1.9.3

Re: [PATCH v3.14-rt] netpoll: guard the access to dev->npinfo with rcu_read_lock/unlock_bh() for CONFIG_PREEMPT_RT_FULL=y

[Nicholas Mc Guire]

On Wed, 03 Dec 2014, Kevin Hao wrote:

> For vanilla kernel we don't need to invoke rcu_read_lock/unlock_bh
> explicitly to mark an RCU-bh critical section in the softirq context
> because bh is already disabled in this case. But for a rt kernel,
> the commit ("rcu: Merge RCU-bh into RCU-preempt") implements the
> RCU-bh in term of RCU-preempt. So we have to use
> rcu_read_lock/unlock_bh() to mark an RCU-bh critical section even in
> +
> +#ifdef CONFIG_PREEMPT_RT_FULL
> +	rcu_read_lock_bh();
> +#endif
> +
> +	npinfo = rcu_dereference_bh(skb->dev->npinfo);
> +	ret = npinfo && (!list_empty(&npinfo->rx_np) || npinfo->rx_flags);
> +
> +#ifdef CONFIG_PREEMPT_RT_FULL
> +	rcu_read_unlock_bh();
> +#endif

Is that not actually a bug indepedent of RT ? 
without the rcu_read_lock/unlock who says that the 
rcu_dereference is still valid at this point ? 
I though that if bh are already disabled you still
need the read_lock. disabled bh would allow to "downgrad" 
the rcu_read_lock_bh to rcu_read_lock but you still need it.

thx!
hofrat

Re: [PATCH v3.14-rt] netpoll: guard the access to dev->npinfo with rcu_read_lock/unlock_bh() for CONFIG_PREEMPT_RT_FULL=y

[Paul E. McKenney]

On Wed, Dec 03, 2014 at 01:50:20PM +0100, Nicholas Mc Guire wrote:
> On Wed, 03 Dec 2014, Kevin Hao wrote:
> 
> > For vanilla kernel we don't need to invoke rcu_read_lock/unlock_bh
> > explicitly to mark an RCU-bh critical section in the softirq context
> > because bh is already disabled in this case. But for a rt kernel,
> > the commit ("rcu: Merge RCU-bh into RCU-preempt") implements the
> > RCU-bh in term of RCU-preempt. So we have to use
> > rcu_read_lock/unlock_bh() to mark an RCU-bh critical section even in
> > +
> > +#ifdef CONFIG_PREEMPT_RT_FULL
> > +	rcu_read_lock_bh();
> > +#endif
> > +
> > +	npinfo = rcu_dereference_bh(skb->dev->npinfo);
> > +	ret = npinfo && (!list_empty(&npinfo->rx_np) || npinfo->rx_flags);
> > +
> > +#ifdef CONFIG_PREEMPT_RT_FULL
> > +	rcu_read_unlock_bh();
> > +#endif
> 
> Is that not actually a bug indepedent of RT ? 
> without the rcu_read_lock/unlock who says that the 
> rcu_dereference is still valid at this point ? 
> I though that if bh are already disabled you still
> need the read_lock. disabled bh would allow to "downgrad" 
> the rcu_read_lock_bh to rcu_read_lock but you still need it.

In vanilla kernels, anything that disables BH acts as rcu_read_lock_bh().
So yes, you can have cases where rcu_read_lock_bh() is needed only in
the -rt kernel.

							Thanx, Paul

Re: [PATCH v3.14-rt] netpoll: guard the access to dev->npinfo with rcu_read_lock/unlock_bh() for CONFIG_PREEMPT_RT_FULL=y

[Sebastian Andrzej Siewior]

* Paul E. McKenney | 2014-12-03 13:19:11 [-0800]:

>> Is that not actually a bug indepedent of RT ? 
>> without the rcu_read_lock/unlock who says that the 
>> rcu_dereference is still valid at this point ? 
>> I though that if bh are already disabled you still
>> need the read_lock. disabled bh would allow to "downgrad" 
>> the rcu_read_lock_bh to rcu_read_lock but you still need it.
>
>In vanilla kernels, anything that disables BH acts as rcu_read_lock_bh().
>So yes, you can have cases where rcu_read_lock_bh() is needed only in
>the -rt kernel.

But it won't hurt mainline using rcu_read_lock_bh() around
rcu_dereference_bh() right?
I am not going to apply this because that code is gone shortly after
v3.14 was released. The fm10k however does the same thing so atleast RCU
knows when to scream :)

>
>							Thanx, Paul

Sebastian

Re: [PATCH v3.14-rt] netpoll: guard the access to dev->npinfo with rcu_read_lock/unlock_bh() for CONFIG_PREEMPT_RT_FULL=y

[Paul E. McKenney]

On Tue, Feb 17, 2015 at 03:29:53PM +0100, Sebastian Andrzej Siewior wrote:
> * Paul E. McKenney | 2014-12-03 13:19:11 [-0800]:
> 
> >> Is that not actually a bug indepedent of RT ? 
> >> without the rcu_read_lock/unlock who says that the 
> >> rcu_dereference is still valid at this point ? 
> >> I though that if bh are already disabled you still
> >> need the read_lock. disabled bh would allow to "downgrad" 
> >> the rcu_read_lock_bh to rcu_read_lock but you still need it.
> >
> >In vanilla kernels, anything that disables BH acts as rcu_read_lock_bh().
> >So yes, you can have cases where rcu_read_lock_bh() is needed only in
> >the -rt kernel.
> 
> But it won't hurt mainline using rcu_read_lock_bh() around
> rcu_dereference_bh() right?

Using rcu_read_lock_bh() around rcu_dereference_bh() is completely
legal, yes.

> I am not going to apply this because that code is gone shortly after
> v3.14 was released. The fm10k however does the same thing so atleast RCU
> knows when to scream :)

That is good, because it has been a good long time since I could
reasonably review all code in the kernel that uses RCU.  ;-)

							Thanx, Paul

> Sebastian
>