Re: [PATCH net] team: fix deadlock during setting of MTU

[Ivan Vecera <ivecera@redhat.com>]

On Thu, 14 Jan 2021 16:34:24 -0800
Saeed Mahameed <saeed@kernel.org> wrote:

> On Thu, 2021-01-14 at 12:55 +0100, Ivan Vecera wrote:
> > Team driver protects port list traversal in team_change_mtu()
> > by its team->lock mutex. This causes a deadlock with certain
> > devices that calls netdev_update_features() during their
> > .ndo_change_mtu() callback. In this case netdev_update_features()
> > calls team's netdevice notifier team_device_event() that in case
> > of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.
> > 
> > Example (r8169 case):
> > ...
> > [ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
> > [ 6391.358602]  team_device_event+0x9d/0x160 [team]
> > [ 6391.363756]  notifier_call_chain+0x47/0x70
> > [ 6391.368329]  netdev_update_features+0x56/0x60
> > [ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
> > [ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
> > [ 6391.387022]  dev_set_mtu+0x52/0x90
> > [ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
> > [ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
> > [ 6391.399963]  do_setlink+0x231/0xf50
> > ...
> > 
> > To fix the problem the port list traversal in team_change_mtu() can
> > be protected by RCU read lock. In case of failure the failing port
> > is marked and unwind code-path is done also under RCU read lock
> > protection (but not in reverse order).
> > 
> > Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device")
> > Cc: Jiri Pirko <jiri@resnulli.us>
> > Signed-off-by: Ivan Vecera <ivecera@redhat.com>
> > ---
> >  drivers/net/team/team.c | 20 ++++++++++----------
> >  1 file changed, 10 insertions(+), 10 deletions(-)
> > 
> > diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
> > index c19dac21c468..69d4b28beb17 100644
> > --- a/drivers/net/team/team.c
> > +++ b/drivers/net/team/team.c
> > @@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct
> > net_device *dev, void *p)
> >  static int team_change_mtu(struct net_device *dev, int new_mtu)
> >  {
> >  	struct team *team = netdev_priv(dev);
> > -	struct team_port *port;
> > +	struct team_port *port, *fail_port;
> >  	int err;
> >  
> > -	/*
> > -	 * Alhough this is reader, it's guarded by team lock. It's not
> > possible
> > -	 * to traverse list in reverse under rcu_read_lock
> > -	 */
> > -	mutex_lock(&team->lock);
> > +	rcu_read_lock();
> >  	team->port_mtu_change_allowed = true;
> > -	list_for_each_entry(port, &team->port_list, list) {
> > +	list_for_each_entry_rcu(port, &team->port_list, list) {
> >  		err = dev_set_mtu(port->dev, new_mtu);  
> 
> some netdevs will need to sleep in their set_mtu, and you can't sleep
> under rcu!
> 
> according to your explanation in the commit message the team->lock
> mutex will be also taken under this rcu lock, so this is bad even
> if dev_set_mtu does not sleep.
> 
Hmm, you are right... btw do we need to take this mutex at this place?

team_change_mtu() is protected by RTNL, team_device_event() as a netdevice
notifier as well... and team_{add,del}_slave() that modify port list
also.

Thoughts?

Thanks,
Ivan