* Q: what protects dev->napi_list?
@ 2012-08-24 9:46 Cong Wang
2012-08-24 10:12 ` Eric Dumazet
0 siblings, 1 reply; 6+ messages in thread
From: Cong Wang @ 2012-08-24 9:46 UTC (permalink / raw)
To: netdev; +Cc: Eric Dumazet, Sylvain Munaut, David Miller
Hi,
Sylvain reported a netpoll CPU stall
http://marc.info/?l=linux-netdev&m=134563282530588&w=2
I tried to provide some fix for it:
http://marc.info/?l=linux-netdev&m=134571069921429&w=2
When reviewing that code, I noticed a problem, it seems dev->napi_list
is not protected by any lock? What if the device driver calls
netif_napi_del() meanwhile we are iterating &dev->napi_list in
poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
called with the RTNL lock held during driver init/uninit, but again
poll_napi() doesn't have RTNL lock.
Am I missing anything?
Thanks!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Q: what protects dev->napi_list?
2012-08-24 9:46 Q: what protects dev->napi_list? Cong Wang
@ 2012-08-24 10:12 ` Eric Dumazet
2012-08-24 10:39 ` Cong Wang
0 siblings, 1 reply; 6+ messages in thread
From: Eric Dumazet @ 2012-08-24 10:12 UTC (permalink / raw)
To: Cong Wang; +Cc: netdev, Sylvain Munaut, David Miller
On Fri, 2012-08-24 at 17:46 +0800, Cong Wang wrote:
> Hi,
>
> Sylvain reported a netpoll CPU stall
> http://marc.info/?l=linux-netdev&m=134563282530588&w=2
>
> I tried to provide some fix for it:
> http://marc.info/?l=linux-netdev&m=134571069921429&w=2
>
> When reviewing that code, I noticed a problem, it seems dev->napi_list
> is not protected by any lock? What if the device driver calls
> netif_napi_del() meanwhile we are iterating &dev->napi_list in
> poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
> called with the RTNL lock held during driver init/uninit, but again
> poll_napi() doesn't have RTNL lock.
>
Of course poll_napi() cant try to get RTNL (its a mutex by the way)
There are no problems, since :
netif_napi_add() is called at device open time (before napi_poll() can
use it)
netif_napi_del() at device dismantle time (after making sure napi_poll()
wont use the device again)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Q: what protects dev->napi_list?
2012-08-24 10:12 ` Eric Dumazet
@ 2012-08-24 10:39 ` Cong Wang
2012-08-24 11:29 ` Eric Dumazet
0 siblings, 1 reply; 6+ messages in thread
From: Cong Wang @ 2012-08-24 10:39 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, Sylvain Munaut, David Miller
On Fri, 2012-08-24 at 12:12 +0200, Eric Dumazet wrote:
> On Fri, 2012-08-24 at 17:46 +0800, Cong Wang wrote:
> > Hi,
> >
> > Sylvain reported a netpoll CPU stall
> > http://marc.info/?l=linux-netdev&m=134563282530588&w=2
> >
> > I tried to provide some fix for it:
> > http://marc.info/?l=linux-netdev&m=134571069921429&w=2
> >
> > When reviewing that code, I noticed a problem, it seems dev->napi_list
> > is not protected by any lock? What if the device driver calls
> > netif_napi_del() meanwhile we are iterating &dev->napi_list in
> > poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
> > called with the RTNL lock held during driver init/uninit, but again
> > poll_napi() doesn't have RTNL lock.
> >
>
> Of course poll_napi() cant try to get RTNL (its a mutex by the way)
>
> There are no problems, since :
>
> netif_napi_add() is called at device open time (before napi_poll() can
> use it)
>
> netif_napi_del() at device dismantle time (after making sure napi_poll()
> wont use the device again)
Yeah, but bnx2 driver calls it at other time too, for example
bnx2_change_ring_size() which in turn could be called by
bnx2_set_channels().
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Q: what protects dev->napi_list?
2012-08-24 10:39 ` Cong Wang
@ 2012-08-24 11:29 ` Eric Dumazet
2012-08-25 6:08 ` Cong Wang
0 siblings, 1 reply; 6+ messages in thread
From: Eric Dumazet @ 2012-08-24 11:29 UTC (permalink / raw)
To: Cong Wang; +Cc: netdev, Sylvain Munaut, David Miller
On Fri, 2012-08-24 at 18:39 +0800, Cong Wang wrote:
> On Fri, 2012-08-24 at 12:12 +0200, Eric Dumazet wrote:
> > On Fri, 2012-08-24 at 17:46 +0800, Cong Wang wrote:
> > > Hi,
> > >
> > > Sylvain reported a netpoll CPU stall
> > > http://marc.info/?l=linux-netdev&m=134563282530588&w=2
> > >
> > > I tried to provide some fix for it:
> > > http://marc.info/?l=linux-netdev&m=134571069921429&w=2
> > >
> > > When reviewing that code, I noticed a problem, it seems dev->napi_list
> > > is not protected by any lock? What if the device driver calls
> > > netif_napi_del() meanwhile we are iterating &dev->napi_list in
> > > poll_napi()? It seems netif_napi_del()/netif_napi_add() are usually
> > > called with the RTNL lock held during driver init/uninit, but again
> > > poll_napi() doesn't have RTNL lock.
> > >
> >
> > Of course poll_napi() cant try to get RTNL (its a mutex by the way)
> >
> > There are no problems, since :
> >
> > netif_napi_add() is called at device open time (before napi_poll() can
> > use it)
> >
> > netif_napi_del() at device dismantle time (after making sure napi_poll()
> > wont use the device again)
>
> Yeah, but bnx2 driver calls it at other time too, for example
> bnx2_change_ring_size() which in turn could be called by
> bnx2_set_channels().
Then at this point, device is stopped, or should be.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Q: what protects dev->napi_list?
2012-08-24 11:29 ` Eric Dumazet
@ 2012-08-25 6:08 ` Cong Wang
2012-08-25 6:49 ` Eric Dumazet
0 siblings, 1 reply; 6+ messages in thread
From: Cong Wang @ 2012-08-25 6:08 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, Sylvain Munaut, David Miller
On Fri, 2012-08-24 at 13:29 +0200, Eric Dumazet wrote:
> On Fri, 2012-08-24 at 18:39 +0800, Cong Wang wrote:
> >
> > Yeah, but bnx2 driver calls it at other time too, for example
> > bnx2_change_ring_size() which in turn could be called by
> > bnx2_set_channels().
>
> Then at this point, device is stopped, or should be.
>
But poll_napi() is still iterating the dev->napi_list, could anyone
stops it?
IOW, the following race condition may happen:
static void poll_napi(struct net_device *dev)
{
struct napi_struct *napi;
int budget = 16;
WARN_ON_ONCE(!irqs_disabled());
list_for_each_entry(napi, &dev->napi_list, dev_list) {
local_irq_enable();
//<-- Another process may call
//bnx2_change_ring_size() to del
//napi from dev, on other CPU.
local_irq_disable();
}
}
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Q: what protects dev->napi_list?
2012-08-25 6:08 ` Cong Wang
@ 2012-08-25 6:49 ` Eric Dumazet
0 siblings, 0 replies; 6+ messages in thread
From: Eric Dumazet @ 2012-08-25 6:49 UTC (permalink / raw)
To: Cong Wang; +Cc: netdev, Sylvain Munaut, David Miller
On Sat, 2012-08-25 at 14:08 +0800, Cong Wang wrote:
> On Fri, 2012-08-24 at 13:29 +0200, Eric Dumazet wrote:
> > On Fri, 2012-08-24 at 18:39 +0800, Cong Wang wrote:
> > >
> > > Yeah, but bnx2 driver calls it at other time too, for example
> > > bnx2_change_ring_size() which in turn could be called by
> > > bnx2_set_channels().
> >
> > Then at this point, device is stopped, or should be.
> >
>
> But poll_napi() is still iterating the dev->napi_list, could anyone
> stops it?
>
> IOW, the following race condition may happen:
>
> static void poll_napi(struct net_device *dev)
> {
> struct napi_struct *napi;
> int budget = 16;
>
> WARN_ON_ONCE(!irqs_disabled());
>
> list_for_each_entry(napi, &dev->napi_list, dev_list) {
> local_irq_enable();
>
> //<-- Another process may call
> //bnx2_change_ring_size() to del
> //napi from dev, on other CPU.
>
> local_irq_disable();
> }
> }
>
>
Are you trying to say netpoll is buggy ?
Thats the first time I ear that !
Just kidding.
I repeat again : If bnx2_change_ring_size() is called while netconsole
is able to enter poll_napi(), then netpoll is buggy and should be fixed.
napi_list is manipulated under a mutex protection, and there is no way
netpoll can use a mutex. Same mutex protection for all other netdevice
management, so even calling start_xmit() should be forbidden in this
state.
Therefore, netpoll must be disabled while a device is reconfigured.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-08-25 6:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-24 9:46 Q: what protects dev->napi_list? Cong Wang
2012-08-24 10:12 ` Eric Dumazet
2012-08-24 10:39 ` Cong Wang
2012-08-24 11:29 ` Eric Dumazet
2012-08-25 6:08 ` Cong Wang
2012-08-25 6:49 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).