for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

[Breno Leitao]

Hello,

We're seeing CONFIG_PROVE_RCU_LIST warnings when for_each_netdev_rcu()
is called with RTNL held. While RTNL provides sufficient locking, the
RCU list checker isn't aware of this relationship, leading to false
positives like:

	WARNING: suspicious RCU usage
	net/core/dev.c:1143 RCU-list traversed in non-reader section!!

The initial discussion popped up in:

	https://lore.kernel.org/all/20250205-flying-coucal-of-influence-0dcbc3@leitao/

I've attempted a solution by modifying for_each_netdev_rcu():

	diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
	index 2a59034a5fa2f..59b18b58fa927 100644
	--- a/include/linux/netdevice.h
	+++ b/include/linux/netdevice.h
	@@ -3210,13 +3210,14 @@ netdev_notifier_info_to_extack(const struct netdev_notifier_info *info)
	int call_netdevice_notifiers(unsigned long val, struct net_device *dev);
	int call_netdevice_notifiers_info(unsigned long val,
					struct netdev_notifier_info *info);
	+bool lockdep_rtnl_net_is_held(struct net *net);

	#define for_each_netdev(net, d)		\
			list_for_each_entry(d, &(net)->dev_base_head, dev_list)
	#define for_each_netdev_reverse(net, d)	\
			list_for_each_entry_reverse(d, &(net)->dev_base_head, dev_list)
	#define for_each_netdev_rcu(net, d)		\
	-		list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list)
	+		list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list, lockdep_rtnl_net_is_held(net))
	#define for_each_netdev_safe(net, d, n)	\
			list_for_each_entry_safe(d, n, &(net)->dev_base_head, dev_list)
	#define for_each_netdev_continue(net, d)		\

However, I have concerns about using lockdep_rtnl_net_is_held() since it
has a dependency on CONFIG_DEBUG_NET_SMALL_RTNL.

Are there better approaches to silence these warnings when RTNL is held?
Any suggestions would be appreciated.

Thanks
--breno

Re: for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

[Kuniyuki Iwashima]

From: Breno Leitao <leitao@debian.org>
Date: Thu, 6 Feb 2025 07:51:55 -0800
> Hello,
> 
> We're seeing CONFIG_PROVE_RCU_LIST warnings when for_each_netdev_rcu()
> is called with RTNL held. While RTNL provides sufficient locking, the
> RCU list checker isn't aware of this relationship, leading to false
> positives like:
> 
> 	WARNING: suspicious RCU usage
> 	net/core/dev.c:1143 RCU-list traversed in non-reader section!!
> 
> The initial discussion popped up in:
> 
> 	https://lore.kernel.org/all/20250205-flying-coucal-of-influence-0dcbc3@leitao/
> 
> I've attempted a solution by modifying for_each_netdev_rcu():
> 
> 	diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> 	index 2a59034a5fa2f..59b18b58fa927 100644
> 	--- a/include/linux/netdevice.h
> 	+++ b/include/linux/netdevice.h
> 	@@ -3210,13 +3210,14 @@ netdev_notifier_info_to_extack(const struct netdev_notifier_info *info)
> 	int call_netdevice_notifiers(unsigned long val, struct net_device *dev);
> 	int call_netdevice_notifiers_info(unsigned long val,
> 					struct netdev_notifier_info *info);
> 	+bool lockdep_rtnl_net_is_held(struct net *net);
> 
> 	#define for_each_netdev(net, d)		\
> 			list_for_each_entry(d, &(net)->dev_base_head, dev_list)
> 	#define for_each_netdev_reverse(net, d)	\
> 			list_for_each_entry_reverse(d, &(net)->dev_base_head, dev_list)
> 	#define for_each_netdev_rcu(net, d)		\
> 	-		list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list)
> 	+		list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list, lockdep_rtnl_net_is_held(net))
> 	#define for_each_netdev_safe(net, d, n)	\
> 			list_for_each_entry_safe(d, n, &(net)->dev_base_head, dev_list)
> 	#define for_each_netdev_continue(net, d)		\
> 
> However, I have concerns about using lockdep_rtnl_net_is_held() since it
> has a dependency on CONFIG_DEBUG_NET_SMALL_RTNL.
> 
> Are there better approaches to silence these warnings when RTNL is held?
> Any suggestions would be appreciated.

We can't use lockdep_rtnl_net_is_held() there yet because most users are
not converted to per-netns RTNL, so it will complain loudly.

Re: for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

[Breno Leitao]

Hello Kuniyuki,

On Fri, Feb 07, 2025 at 12:38:22PM +0900, Kuniyuki Iwashima wrote:
> From: Breno Leitao <leitao@debian.org>
> Date: Thu, 6 Feb 2025 07:51:55 -0800

> > Are there better approaches to silence these warnings when RTNL is held?
> > Any suggestions would be appreciated.
> 
> We can't use lockdep_rtnl_net_is_held() there yet because most users are
> not converted to per-netns RTNL, so it will complain loudly.

Right, so, I understand the best approach is to leverage
lockdep_rtnl_is_held() only right now. Something as:

	diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
	index 1dcc76af75203..0deee1313f23a 100644
	--- a/include/linux/netdevice.h
	+++ b/include/linux/netdevice.h
	@@ -3217,7 +3217,8 @@ int call_netdevice_notifiers_info(unsigned long val,
	#define for_each_netdev_reverse(net, d)        \
			list_for_each_entry_reverse(d, &(net)->dev_base_head, dev_list)
	#define for_each_netdev_rcu(net, d)            \
	-               list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list)
	+               list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list, \
	+                                       lockdep_rtnl_is_held())
	#define for_each_netdev_safe(net, d, n)        \
			list_for_each_entry_safe(d, n, &(net)->dev_base_head, dev_list)
	#define for_each_netdev_continue(net, d)               \

Which brings another problem:

lockdep_rtnl_is_held() is defined in include/linux/rtnetlink.h, so,
we'll need to include 'linux/rtnetlink.h' in linux/netdevice.h, which
doesn't seem correct (!?).

Otherwise drivers using for_each_netdev_rcu() will not be able to find
lockdep_rtnl_is_held().

I suppose we will need to move some of definitions around, but, I am
confident in which way.

--breno

Re: for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

[Eric Dumazet]

On Fri, Feb 7, 2025 at 11:46 AM Breno Leitao <leitao@debian.org> wrote:
>
> Hello Kuniyuki,
>
> On Fri, Feb 07, 2025 at 12:38:22PM +0900, Kuniyuki Iwashima wrote:
> > From: Breno Leitao <leitao@debian.org>
> > Date: Thu, 6 Feb 2025 07:51:55 -0800
>
> > > Are there better approaches to silence these warnings when RTNL is held?
> > > Any suggestions would be appreciated.
> >
> > We can't use lockdep_rtnl_net_is_held() there yet because most users are
> > not converted to per-netns RTNL, so it will complain loudly.
>
> Right, so, I understand the best approach is to leverage
> lockdep_rtnl_is_held() only right now. Something as:
>
>         diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
>         index 1dcc76af75203..0deee1313f23a 100644
>         --- a/include/linux/netdevice.h
>         +++ b/include/linux/netdevice.h
>         @@ -3217,7 +3217,8 @@ int call_netdevice_notifiers_info(unsigned long val,
>         #define for_each_netdev_reverse(net, d)        \
>                         list_for_each_entry_reverse(d, &(net)->dev_base_head, dev_list)
>         #define for_each_netdev_rcu(net, d)            \
>         -               list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list)
>         +               list_for_each_entry_rcu(d, &(net)->dev_base_head, dev_list, \
>         +                                       lockdep_rtnl_is_held())
>         #define for_each_netdev_safe(net, d, n)        \
>                         list_for_each_entry_safe(d, n, &(net)->dev_base_head, dev_list)
>         #define for_each_netdev_continue(net, d)               \
>
> Which brings another problem:
>
> lockdep_rtnl_is_held() is defined in include/linux/rtnetlink.h, so,
> we'll need to include 'linux/rtnetlink.h' in linux/netdevice.h, which
> doesn't seem correct (!?).
>
> Otherwise drivers using for_each_netdev_rcu() will not be able to find
> lockdep_rtnl_is_held().
>
> I suppose we will need to move some of definitions around, but, I am
> confident in which way.

Note that we have different accessors like rtnl_dereference() and
rcu_dereference_rtnl()
It helps to differentiate expectations, and as self describing code.

I would not change  for_each_netdev_rcu(), and instead add a new
dev_getbyhwaddr_rtnl()
function for contexts holding RTNL.

Alternatively, add one rcu_read_lock()/rcu_read_unlock() pair as some
dev_getbyhwaddr_rcu() callers already do.

Re: for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

[Breno Leitao]

Hello Eric,

On Fri, Feb 07, 2025 at 11:56:53AM +0100, Eric Dumazet wrote:

> > I suppose we will need to move some of definitions around, but, I am
> > NOT confident in which way.
> 
> Note that we have different accessors like rtnl_dereference() and
> rcu_dereference_rtnl()

Makes sense. I suppose that would be a for_each_netdev_rtnl().

> It helps to differentiate expectations, and as self describing code.

The problem with this approach, is that we don't know what lock the
caller of dev_getbyhwaddr_rcu() is using, thus, we cannot leverage
a possible for_each_netdev_rtnl() inside dev_getbyhwaddr_rcu().

> I would not change  for_each_netdev_rcu(), and instead add a new
> dev_getbyhwaddr_rtnl()
> function for contexts holding RTNL.

Initially, I had reservations about this approach, but after further
consideration, it seems that creating separate variants of
dev_getbyhwaddr() might be the most effective solution.

By doing so, we can introduce dev_getbyhwaddr_rcu() and
dev_getbyhwaddr_rtnl(), each tailored to specific locking mechanisms.

To explore this idea further, I'll create a proof-of-concept
implementation to see how these new functions would look in practice.

This will help us determine whether this approach is indeed the best way
forward. Thanks for the suggestion.

> Alternatively, add one rcu_read_lock()/rcu_read_unlock() pair as some
> dev_getbyhwaddr_rcu() callers already do.

Fair, we can do that as well, but, it seemed weird to me to have
something like:

	rtnl_lock();
	rcu_read_lock();
	dev_getbyhwaddr_rcu();

Thanks for chiming in
--breno

Re: for_each_netdev_rcu() protected by RTNL and CONFIG_PROVE_RCU_LIST

[Breno Leitao]

On Fri, Feb 07, 2025 at 03:26:14AM -0800, Breno Leitao wrote:

> > Note that we have different accessors like rtnl_dereference() and
> > rcu_dereference_rtnl()

> To explore this idea further, I'll create a proof-of-concept
> implementation to see how these new functions would look in practice.

I hacked a bit, and I have an RFC for further discussion:

https://lore.kernel.org/all/20250207-arm_fix_selftest-v1-1-487518d2fd1c@debian.org/

Thanks for the suggestion,
--breno