Q: what protects dev->napi_list?

Q: what protects dev->napi_list?

[Cong Wang]

Hi,

Sylvain reported a netpoll CPU stall
http://marc.info/?l=linux-netdev&m=134563282530588&w=2

I tried to provide some fix for it:
http://marc.info/?l=linux-netdev&m=134571069921429&w=2

When reviewing that code, I noticed a problem, it seems dev->napi_list
is not protected by any lock? What if the device driver calls
netif_napi_del() meanwhile we are iterating &dev->napi_list in
poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
called with the RTNL lock held during driver init/uninit, but again
poll_napi() doesn't have RTNL lock.

Am I missing anything?

Thanks!

Re: Q: what protects dev->napi_list?

[Eric Dumazet]

On Fri, 2012-08-24 at 17:46 +0800, Cong Wang wrote:
> Hi,
> 
> Sylvain reported a netpoll CPU stall
> http://marc.info/?l=linux-netdev&m=134563282530588&w=2
> 
> I tried to provide some fix for it:
> http://marc.info/?l=linux-netdev&m=134571069921429&w=2
> 
> When reviewing that code, I noticed a problem, it seems dev->napi_list
> is not protected by any lock? What if the device driver calls
> netif_napi_del() meanwhile we are iterating &dev->napi_list in
> poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
> called with the RTNL lock held during driver init/uninit, but again
> poll_napi() doesn't have RTNL lock.
> 

Of course poll_napi() cant try to get RTNL (its a mutex by the way)

There are no problems, since :

netif_napi_add() is called at device open time (before napi_poll() can
use it)

netif_napi_del() at device dismantle time (after making sure napi_poll()
wont use the device again)

Re: Q: what protects dev->napi_list?

[Cong Wang]

On Fri, 2012-08-24 at 12:12 +0200, Eric Dumazet wrote:
> On Fri, 2012-08-24 at 17:46 +0800, Cong Wang wrote:
> > Hi,
> > 
> > Sylvain reported a netpoll CPU stall
> > http://marc.info/?l=linux-netdev&m=134563282530588&w=2
> > 
> > I tried to provide some fix for it:
> > http://marc.info/?l=linux-netdev&m=134571069921429&w=2
> > 
> > When reviewing that code, I noticed a problem, it seems dev->napi_list
> > is not protected by any lock? What if the device driver calls
> > netif_napi_del() meanwhile we are iterating &dev->napi_list in
> > poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
> > called with the RTNL lock held during driver init/uninit, but again
> > poll_napi() doesn't have RTNL lock.
> > 
> 
> Of course poll_napi() cant try to get RTNL (its a mutex by the way)
> 
> There are no problems, since :
> 
> netif_napi_add() is called at device open time (before napi_poll() can
> use it)
> 
> netif_napi_del() at device dismantle time (after making sure napi_poll()
> wont use the device again)

Yeah, but bnx2 driver calls it at other time too, for example
bnx2_change_ring_size() which in turn could be called by
bnx2_set_channels().

Re: Q: what protects dev->napi_list?

[Eric Dumazet]

On Fri, 2012-08-24 at 18:39 +0800, Cong Wang wrote:
> On Fri, 2012-08-24 at 12:12 +0200, Eric Dumazet wrote:
> > On Fri, 2012-08-24 at 17:46 +0800, Cong Wang wrote:
> > > Hi,
> > > 
> > > Sylvain reported a netpoll CPU stall
> > > http://marc.info/?l=linux-netdev&m=134563282530588&w=2
> > > 
> > > I tried to provide some fix for it:
> > > http://marc.info/?l=linux-netdev&m=134571069921429&w=2
> > > 
> > > When reviewing that code, I noticed a problem, it seems dev->napi_list
> > > is not protected by any lock? What if the device driver calls
> > > netif_napi_del() meanwhile we are iterating &dev->napi_list in
> > > poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
> > > called with the RTNL lock held during driver init/uninit, but again
> > > poll_napi() doesn't have RTNL lock.
> > > 
> > 
> > Of course poll_napi() cant try to get RTNL (its a mutex by the way)
> > 
> > There are no problems, since :
> > 
> > netif_napi_add() is called at device open time (before napi_poll() can
> > use it)
> > 
> > netif_napi_del() at device dismantle time (after making sure napi_poll()
> > wont use the device again)
> 
> Yeah, but bnx2 driver calls it at other time too, for example
> bnx2_change_ring_size() which in turn could be called by
> bnx2_set_channels().

Then at this point, device is stopped, or should be.

Re: Q: what protects dev->napi_list?

[Cong Wang]

On Fri, 2012-08-24 at 13:29 +0200, Eric Dumazet wrote:
> On Fri, 2012-08-24 at 18:39 +0800, Cong Wang wrote:
> > 
> > Yeah, but bnx2 driver calls it at other time too, for example
> > bnx2_change_ring_size() which in turn could be called by
> > bnx2_set_channels().
> 
> Then at this point, device is stopped, or should be.
> 

But poll_napi() is still iterating the dev->napi_list, could anyone
stops it?

IOW, the following race condition may happen:

static void poll_napi(struct net_device *dev)
{               
        struct napi_struct *napi;
        int budget = 16;
                
        WARN_ON_ONCE(!irqs_disabled());
                
        list_for_each_entry(napi, &dev->napi_list, dev_list) {
                local_irq_enable();

				//<-- Another process may call
				//bnx2_change_ring_size() to del
				//napi from dev, on other CPU.

                local_irq_disable();
        }
}

Re: Q: what protects dev->napi_list?

[Eric Dumazet]

On Sat, 2012-08-25 at 14:08 +0800, Cong Wang wrote:
> On Fri, 2012-08-24 at 13:29 +0200, Eric Dumazet wrote:
> > On Fri, 2012-08-24 at 18:39 +0800, Cong Wang wrote:
> > > 
> > > Yeah, but bnx2 driver calls it at other time too, for example
> > > bnx2_change_ring_size() which in turn could be called by
> > > bnx2_set_channels().
> > 
> > Then at this point, device is stopped, or should be.
> > 
> 
> But poll_napi() is still iterating the dev->napi_list, could anyone
> stops it?
> 
> IOW, the following race condition may happen:
> 
> static void poll_napi(struct net_device *dev)
> {               
>         struct napi_struct *napi;
>         int budget = 16;
>                 
>         WARN_ON_ONCE(!irqs_disabled());
>                 
>         list_for_each_entry(napi, &dev->napi_list, dev_list) {
>                 local_irq_enable();
> 
> 				//<-- Another process may call
> 				//bnx2_change_ring_size() to del
> 				//napi from dev, on other CPU.
> 
>                 local_irq_disable();
>         }
> }
> 
> 

Are you trying to say netpoll is buggy ?

Thats the first time I ear that !

Just kidding.

I repeat again : If bnx2_change_ring_size() is called while netconsole
is able to enter poll_napi(), then netpoll is buggy and should be fixed.

napi_list is manipulated under a mutex protection, and there is no way
netpoll can use a mutex. Same mutex protection for all other netdevice
management, so even calling start_xmit() should be forbidden in this
state.

Therefore, netpoll must be disabled while a device is reconfigured.