* BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags @ 2013-10-22 1:04 Alexei Starovoitov 2013-10-22 11:52 ` Nicolas Dichtel 2013-10-23 3:53 ` Cong Wang 0 siblings, 2 replies; 6+ messages in thread From: Alexei Starovoitov @ 2013-10-22 1:04 UTC (permalink / raw) To: Nicolas Dichtel; +Cc: netdev Hi Nicolas, after commit 991fb3f74c "dev: always advertise rx_flags changes via netlink" I'm seeing 'sleeping in atomic' bug. Steps to reproduce: ip tuntap add dev tap1 mode tap ifconfig tap1 up tcpdump -nei tap1 and in different terminal: ip tuntap del dev tap1 mode tap [ 271.627994] device tap1 left promiscuous mode [ 271.639897] BUG: sleeping function called from invalid context at mm/slub.c:940 [ 271.664491] in_atomic(): 1, irqs_disabled(): 0, pid: 3394, name: ip [ 271.677525] INFO: lockdep is turned off. [ 271.690503] CPU: 0 PID: 3394 Comm: ip Tainted: G W 3.12.0-rc3+ #73 [ 271.703996] Hardware name: System manufacturer System Product Name/P8Z77 WS, BIOS 3007 07/26/2012 [ 271.731254] ffffffff81a58506 ffff8807f0d57a58 ffffffff817544e5 ffff88082fa0f428 [ 271.760261] ffff8808071f5f40 ffff8807f0d57a88 ffffffff8108bad1 ffffffff81110ff8 [ 271.790683] 0000000000000010 00000000000000d0 00000000000000d0 ffff8807f0d57af8 [ 271.822332] Call Trace: [ 271.838234] [<ffffffff817544e5>] dump_stack+0x55/0x76 [ 271.854446] [<ffffffff8108bad1>] __might_sleep+0x181/0x240 [ 271.870836] [<ffffffff81110ff8>] ? rcu_irq_exit+0x68/0xb0 [ 271.887076] [<ffffffff811a80be>] kmem_cache_alloc_node+0x4e/0x2a0 [ 271.903368] [<ffffffff810b4ddc>] ? vprintk_emit+0x1dc/0x5a0 [ 271.919716] [<ffffffff81614d67>] ? __alloc_skb+0x57/0x2a0 [ 271.936088] [<ffffffff810b4de0>] ? vprintk_emit+0x1e0/0x5a0 [ 271.952504] [<ffffffff81614d67>] __alloc_skb+0x57/0x2a0 [ 271.968902] [<ffffffff8163a0b2>] rtmsg_ifinfo+0x52/0x100 [ 271.985302] [<ffffffff8162ac6d>] __dev_notify_flags+0xad/0xc0 [ 272.001642] [<ffffffff8162ad0c>] __dev_set_promiscuity+0x8c/0x1c0 [ 272.017917] [<ffffffff81731ea5>] ? packet_notifier+0x5/0x380 [ 272.033961] [<ffffffff8162b109>] dev_set_promiscuity+0x29/0x50 [ 272.049855] [<ffffffff8172e937>] packet_dev_mc+0x87/0xc0 [ 272.065494] [<ffffffff81732052>] packet_notifier+0x1b2/0x380 [ 272.080915] [<ffffffff81731ea5>] ? packet_notifier+0x5/0x380 [ 272.096009] [<ffffffff81761c66>] notifier_call_chain+0x66/0x150 [ 272.110803] [<ffffffff8108503e>] __raw_notifier_call_chain+0xe/0x10 [ 272.125468] [<ffffffff81085056>] raw_notifier_call_chain+0x16/0x20 [ 272.139984] [<ffffffff81620190>] call_netdevice_notifiers_info+0x40/0x70 [ 272.154523] [<ffffffff816201d6>] call_netdevice_notifiers+0x16/0x20 [ 272.168552] [<ffffffff816224c5>] rollback_registered_many+0x145/0x240 [ 272.182263] [<ffffffff81622641>] rollback_registered+0x31/0x40 [ 272.195369] [<ffffffff816229c8>] unregister_netdevice_queue+0x58/0x90 [ 272.208230] [<ffffffff81547ca0>] __tun_detach+0x140/0x340 [ 272.220686] [<ffffffff81547ed6>] tun_chr_close+0x36/0x60 packet_notifier() does rcu_read_lock() before calling into packet_dev_mc() . Not sure how to fix it cleanly, other than disabling a notify here. Any suggestion? Thanks Alexei ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags 2013-10-22 1:04 BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags Alexei Starovoitov @ 2013-10-22 11:52 ` Nicolas Dichtel 2013-10-23 3:53 ` Cong Wang 1 sibling, 0 replies; 6+ messages in thread From: Nicolas Dichtel @ 2013-10-22 11:52 UTC (permalink / raw) To: Alexei Starovoitov; +Cc: netdev Le 22/10/2013 03:04, Alexei Starovoitov a écrit : > Hi Nicolas, > > after commit 991fb3f74c "dev: always advertise rx_flags changes via netlink" > I'm seeing 'sleeping in atomic' bug. > > Steps to reproduce: > ip tuntap add dev tap1 mode tap > ifconfig tap1 up > tcpdump -nei tap1 > and in different terminal: > ip tuntap del dev tap1 mode tap > > [ 271.627994] device tap1 left promiscuous mode > [ 271.639897] BUG: sleeping function called from invalid context at > mm/slub.c:940 > [ 271.664491] in_atomic(): 1, irqs_disabled(): 0, pid: 3394, name: ip > [ 271.677525] INFO: lockdep is turned off. > [ 271.690503] CPU: 0 PID: 3394 Comm: ip Tainted: G W 3.12.0-rc3+ #73 > [ 271.703996] Hardware name: System manufacturer System Product > Name/P8Z77 WS, BIOS 3007 07/26/2012 > [ 271.731254] ffffffff81a58506 ffff8807f0d57a58 ffffffff817544e5 > ffff88082fa0f428 > [ 271.760261] ffff8808071f5f40 ffff8807f0d57a88 ffffffff8108bad1 > ffffffff81110ff8 > [ 271.790683] 0000000000000010 00000000000000d0 00000000000000d0 > ffff8807f0d57af8 > [ 271.822332] Call Trace: > [ 271.838234] [<ffffffff817544e5>] dump_stack+0x55/0x76 > [ 271.854446] [<ffffffff8108bad1>] __might_sleep+0x181/0x240 > [ 271.870836] [<ffffffff81110ff8>] ? rcu_irq_exit+0x68/0xb0 > [ 271.887076] [<ffffffff811a80be>] kmem_cache_alloc_node+0x4e/0x2a0 > [ 271.903368] [<ffffffff810b4ddc>] ? vprintk_emit+0x1dc/0x5a0 > [ 271.919716] [<ffffffff81614d67>] ? __alloc_skb+0x57/0x2a0 > [ 271.936088] [<ffffffff810b4de0>] ? vprintk_emit+0x1e0/0x5a0 > [ 271.952504] [<ffffffff81614d67>] __alloc_skb+0x57/0x2a0 > [ 271.968902] [<ffffffff8163a0b2>] rtmsg_ifinfo+0x52/0x100 > [ 271.985302] [<ffffffff8162ac6d>] __dev_notify_flags+0xad/0xc0 > [ 272.001642] [<ffffffff8162ad0c>] __dev_set_promiscuity+0x8c/0x1c0 > [ 272.017917] [<ffffffff81731ea5>] ? packet_notifier+0x5/0x380 > [ 272.033961] [<ffffffff8162b109>] dev_set_promiscuity+0x29/0x50 > [ 272.049855] [<ffffffff8172e937>] packet_dev_mc+0x87/0xc0 > [ 272.065494] [<ffffffff81732052>] packet_notifier+0x1b2/0x380 > [ 272.080915] [<ffffffff81731ea5>] ? packet_notifier+0x5/0x380 > [ 272.096009] [<ffffffff81761c66>] notifier_call_chain+0x66/0x150 > [ 272.110803] [<ffffffff8108503e>] __raw_notifier_call_chain+0xe/0x10 > [ 272.125468] [<ffffffff81085056>] raw_notifier_call_chain+0x16/0x20 > [ 272.139984] [<ffffffff81620190>] call_netdevice_notifiers_info+0x40/0x70 > [ 272.154523] [<ffffffff816201d6>] call_netdevice_notifiers+0x16/0x20 > [ 272.168552] [<ffffffff816224c5>] rollback_registered_many+0x145/0x240 > [ 272.182263] [<ffffffff81622641>] rollback_registered+0x31/0x40 > [ 272.195369] [<ffffffff816229c8>] unregister_netdevice_queue+0x58/0x90 > [ 272.208230] [<ffffffff81547ca0>] __tun_detach+0x140/0x340 > [ 272.220686] [<ffffffff81547ed6>] tun_chr_close+0x36/0x60 > > packet_notifier() does rcu_read_lock() before calling into packet_dev_mc() . > > Not sure how to fix it cleanly, other than disabling a notify here. > Any suggestion? I don't reproduce it. Can you send me your .config? I will look more deeply at the code. Regards, Nicolas ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags 2013-10-22 1:04 BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags Alexei Starovoitov 2013-10-22 11:52 ` Nicolas Dichtel @ 2013-10-23 3:53 ` Cong Wang 2013-10-23 5:34 ` Alexei Starovoitov 1 sibling, 1 reply; 6+ messages in thread From: Cong Wang @ 2013-10-23 3:53 UTC (permalink / raw) To: netdev On Tue, 22 Oct 2013 at 01:04 GMT, Alexei Starovoitov <ast@plumgrid.com> wrote: > > packet_notifier() does rcu_read_lock() before calling into packet_dev_mc() . > > Not sure how to fix it cleanly, other than disabling a notify here. > Any suggestion? > Passing a gfp flag to rtmsg_ifinfo() seems a right fix for me, but I don't know if there is other better way to fix it. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags 2013-10-23 3:53 ` Cong Wang @ 2013-10-23 5:34 ` Alexei Starovoitov 2013-10-23 7:48 ` Nicolas Dichtel 0 siblings, 1 reply; 6+ messages in thread From: Alexei Starovoitov @ 2013-10-23 5:34 UTC (permalink / raw) To: Cong Wang; +Cc: netdev, nicolas.dichtel On Tue, Oct 22, 2013 at 8:53 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote: > On Tue, 22 Oct 2013 at 01:04 GMT, Alexei Starovoitov <ast@plumgrid.com> wrote: >> >> packet_notifier() does rcu_read_lock() before calling into packet_dev_mc() . >> >> Not sure how to fix it cleanly, other than disabling a notify here. >> Any suggestion? >> > > Passing a gfp flag to rtmsg_ifinfo() seems a right fix for me, but I don't > know if there is other better way to fix it. Indeed. rtnl_notify() already accepts gfp_t. The following diff fixes it for me: --- include/linux/rtnetlink.h | 3 ++- net/core/dev.c | 2 +- net/core/rtnetlink.c | 12 +++++++++--- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/include/linux/rtnetlink.h b/include/linux/rtnetlink.h index f28544b..0180523 100644 --- a/include/linux/rtnetlink.h +++ b/include/linux/rtnetlink.h @@ -15,7 +15,8 @@ extern int rtnetlink_put_metrics(struct sk_buff *skb, u32 *metrics); extern int rtnl_put_cacheinfo(struct sk_buff *skb, struct dst_entry *dst, u32 id, long expires, u32 error); -extern void rtmsg_ifinfo(int type, struct net_device *dev, unsigned change); +void __rtmsg_ifinfo(int type, struct net_device *dev, unsigned change, gfp_t flags); +void rtmsg_ifinfo(int type, struct net_device *dev, unsigned change); /* RTNL is used as a global lock for all changes to network configuration */ extern void rtnl_lock(void); diff --git a/net/core/dev.c b/net/core/dev.c index 0918aad..59b90fe 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5257,7 +5257,7 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags, unsigned int changes = dev->flags ^ old_flags; if (gchanges) - rtmsg_ifinfo(RTM_NEWLINK, dev, gchanges); + __rtmsg_ifinfo(RTM_NEWLINK, dev, gchanges, GFP_ATOMIC); if (changes & IFF_UP) { if (dev->flags & IFF_UP) diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c index 4aedf03..5931af9 100644 --- a/net/core/rtnetlink.c +++ b/net/core/rtnetlink.c @@ -1984,14 +1984,15 @@ static int rtnl_dump_all(struct sk_buff *skb, struct netlink_callback *cb) return skb->len; } -void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change) +void __rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change, + gfp_t flags) { struct net *net = dev_net(dev); struct sk_buff *skb; int err = -ENOBUFS; size_t if_info_size; - skb = nlmsg_new((if_info_size = if_nlmsg_size(dev, 0)), GFP_KERNEL); + skb = nlmsg_new((if_info_size = if_nlmsg_size(dev, 0)), flags); if (skb == NULL) goto errout; @@ -2002,12 +2003,17 @@ void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change) kfree_skb(skb); goto errout; } - rtnl_notify(skb, net, 0, RTNLGRP_LINK, NULL, GFP_KERNEL); + rtnl_notify(skb, net, 0, RTNLGRP_LINK, NULL, flags); return; errout: if (err < 0) rtnl_set_sk_err(net, RTNLGRP_LINK, err); } + +void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change) +{ + __rtmsg_ifinfo(type, dev, change, GFP_KERNEL); +} EXPORT_SYMBOL(rtmsg_ifinfo); static int nlmsg_populate_fdb_fill(struct sk_buff *skb, -- Nicolas, I've sent you my .config. Any better ideas? Thanks Alex ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags 2013-10-23 5:34 ` Alexei Starovoitov @ 2013-10-23 7:48 ` Nicolas Dichtel 2013-10-23 12:18 ` Nicolas Dichtel 0 siblings, 1 reply; 6+ messages in thread From: Nicolas Dichtel @ 2013-10-23 7:48 UTC (permalink / raw) To: Alexei Starovoitov, Cong Wang; +Cc: netdev Le 23/10/2013 07:34, Alexei Starovoitov a écrit : > On Tue, Oct 22, 2013 at 8:53 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote: >> On Tue, 22 Oct 2013 at 01:04 GMT, Alexei Starovoitov <ast@plumgrid.com> wrote: >>> >>> packet_notifier() does rcu_read_lock() before calling into packet_dev_mc() . >>> >>> Not sure how to fix it cleanly, other than disabling a notify here. >>> Any suggestion? >>> >> >> Passing a gfp flag to rtmsg_ifinfo() seems a right fix for me, but I don't >> know if there is other better way to fix it. > > Indeed. rtnl_notify() already accepts gfp_t. > > The following diff fixes it for me: > --- > include/linux/rtnetlink.h | 3 ++- > net/core/dev.c | 2 +- > net/core/rtnetlink.c | 12 +++++++++--- > 3 files changed, 12 insertions(+), 5 deletions(-) > > diff --git a/include/linux/rtnetlink.h b/include/linux/rtnetlink.h > index f28544b..0180523 100644 > --- a/include/linux/rtnetlink.h > +++ b/include/linux/rtnetlink.h > @@ -15,7 +15,8 @@ extern int rtnetlink_put_metrics(struct sk_buff > *skb, u32 *metrics); > extern int rtnl_put_cacheinfo(struct sk_buff *skb, struct dst_entry *dst, > u32 id, long expires, u32 error); > > -extern void rtmsg_ifinfo(int type, struct net_device *dev, unsigned change); > +void __rtmsg_ifinfo(int type, struct net_device *dev, unsigned > change, gfp_t flags); Just nitpiking: putting the type and the name on the same line is better 'unsigned change'. > +void rtmsg_ifinfo(int type, struct net_device *dev, unsigned change); > > /* RTNL is used as a global lock for all changes to network configuration */ > extern void rtnl_lock(void); > diff --git a/net/core/dev.c b/net/core/dev.c > index 0918aad..59b90fe 100644 > --- a/net/core/dev.c > +++ b/net/core/dev.c > @@ -5257,7 +5257,7 @@ void __dev_notify_flags(struct net_device *dev, > unsigned int old_flags, > unsigned int changes = dev->flags ^ old_flags; > > if (gchanges) > - rtmsg_ifinfo(RTM_NEWLINK, dev, gchanges); > + __rtmsg_ifinfo(RTM_NEWLINK, dev, gchanges, GFP_ATOMIC); > > if (changes & IFF_UP) { > if (dev->flags & IFF_UP) > diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c > index 4aedf03..5931af9 100644 > --- a/net/core/rtnetlink.c > +++ b/net/core/rtnetlink.c > @@ -1984,14 +1984,15 @@ static int rtnl_dump_all(struct sk_buff *skb, > struct netlink_callback *cb) > return skb->len; > } > > -void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change) > +void __rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change, > + gfp_t flags) 'gfp_t flags' should be aligned with 'int type' > { > struct net *net = dev_net(dev); > struct sk_buff *skb; > int err = -ENOBUFS; > size_t if_info_size; > > - skb = nlmsg_new((if_info_size = if_nlmsg_size(dev, 0)), GFP_KERNEL); > + skb = nlmsg_new((if_info_size = if_nlmsg_size(dev, 0)), flags); > if (skb == NULL) > goto errout; > > @@ -2002,12 +2003,17 @@ void rtmsg_ifinfo(int type, struct net_device > *dev, unsigned int change) > kfree_skb(skb); > goto errout; > } > - rtnl_notify(skb, net, 0, RTNLGRP_LINK, NULL, GFP_KERNEL); > + rtnl_notify(skb, net, 0, RTNLGRP_LINK, NULL, flags); > return; > errout: > if (err < 0) > rtnl_set_sk_err(net, RTNLGRP_LINK, err); > } > + > +void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change) > +{ > + __rtmsg_ifinfo(type, dev, change, GFP_KERNEL); > +} > EXPORT_SYMBOL(rtmsg_ifinfo); > > static int nlmsg_populate_fdb_fill(struct sk_buff *skb, > -- > > Nicolas, I've sent you my .config. I got some pb with my testbed, hence I still didn't reproduce the bug. I will make more test today, but I think this is the right patch. > Any better ideas? No and the patch is right for me (__dev_set_promiscuity() call audit_log(..., GFP_ATOMIC) already). ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags 2013-10-23 7:48 ` Nicolas Dichtel @ 2013-10-23 12:18 ` Nicolas Dichtel 0 siblings, 0 replies; 6+ messages in thread From: Nicolas Dichtel @ 2013-10-23 12:18 UTC (permalink / raw) To: Alexei Starovoitov, Cong Wang; +Cc: netdev Le 23/10/2013 09:48, Nicolas Dichtel a écrit : > Le 23/10/2013 07:34, Alexei Starovoitov a écrit : >> On Tue, Oct 22, 2013 at 8:53 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote: >>> On Tue, 22 Oct 2013 at 01:04 GMT, Alexei Starovoitov <ast@plumgrid.com> wrote: >>>> [snip] >> Nicolas, I've sent you my .config. > I got some pb with my testbed, hence I still didn't reproduce the bug. > I will make more test today, but I think this is the right patch. Still not reproduced with your config. But as I said, I think your patch is the right thing to do. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-10-23 12:18 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-10-22 1:04 BUG: scheduling while atomic dev_set_promiscuity->__dev_notify_flags Alexei Starovoitov 2013-10-22 11:52 ` Nicolas Dichtel 2013-10-23 3:53 ` Cong Wang 2013-10-23 5:34 ` Alexei Starovoitov 2013-10-23 7:48 ` Nicolas Dichtel 2013-10-23 12:18 ` Nicolas Dichtel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox