NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi again, a bit more detail:

> I'm trying to use the netconsole to feed kernel message to the outside
> but this lead to a stall ...
>
> This only happens in a fairly specific configuration where you have a
> bridge over vlan over bonding.
> I tested with only (bridge over vlan) and (vlan over bonding) and
> those work fine.
>
> [snip ... see original mail for all details]

I was previously testing under Xen.

For this round of test, I tried the kernel natively. And I also
included Dave Miller pending series ( e0e3cea4... ) since there was
patch related to netconsole and bridging / ...
So in the end, it's a 3.6-rc2 + Dave Miller tree (commit  e0e3cea4 ) +
pf malloc patch  + ip pmtu patch from Eric Dumazet.

I am now seeing more debug when I load netconsole in that config:

[   88.705138] netpoll: netconsole: local port 8888
[   88.705140] netpoll: netconsole: local IP 10.208.1.30
[   88.705141] netpoll: netconsole: interface 'mgmt'
[   88.705142] netpoll: netconsole: remote port 8000
[   88.705143] netpoll: netconsole: remote IP 10.208.1.3
[   88.705144] netpoll: netconsole: remote ethernet address 00:16:3e:1a:37:37
[   88.705469] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000008
[   88.705475] IP: [<ffffffffa0006653>] bnx2_start_xmit+0x20b/0x539 [bnx2]
[   88.705476] PGD 0
[   88.705478] Oops: 0002 [#1] PREEMPT SMP
[   88.705509] Modules linked in: netconsole(+) configfs nfsd
auth_rpcgss nfs_acl nfs lockd fscache sunrpc bridge 8021q garp stp llc
bonding ext2 iTCO_wdt iTCO_vendor_support lpc_ich mfd_core coretemp
joydev kvm evdev crc32c_intel ghash_clmulni_intel aesni_intel
aes_x86_64 aes_generic acpi_power_meter psmouse serio_raw dcdbas
processor ablk_helper i7core_edac pcspkr cryptd edac_core microcode
button hid_generic ext4 crc16 jbd2 mbcache dm_mod raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor xor async_tx
raid6_pq raid1 raid0 multipath linear md_mod sr_mod usbhid cdrom hid
ses sd_mod enclosure crc_t10dif usb_storage ata_generic pata_acpi uas
uhci_hcd megaraid_sas ata_piix ehci_hcd libata usbcore scsi_mod
usb_common bnx2
[   88.705511] CPU 2
[   88.705512] Pid: 3017, comm: modprobe Not tainted
3.6.0-rc2-00092-g9040592-dirty #6 Dell Inc. PowerEdge R610/0F0XJ6
[   88.705515] RIP: 0010:[<ffffffffa0006653>]  [<ffffffffa0006653>]
bnx2_start_xmit+0x20b/0x539 [bnx2]
[   88.705516] RSP: 0018:ffff88061e8fda28  EFLAGS: 00010002
[   88.705517] RAX: 0000000000000000 RBX: ffff8803200f2300 RCX: 0000000000000000
[   88.705519] RDX: 0000000320a95c02 RSI: 0000000000000003 RDI: ffff8800cb36f000
[   88.705519] RBP: ffff88031f814000 R08: 0000000000000054 R09: 0000000000000000
[   88.705520] R10: 000000000000ffff R11: 0000000000000000 R12: ffff8803215d52c0
[   88.705521] R13: ffff8803210e13c0 R14: 0000000000010008 R15: 0000000000000000
[   88.705522] FS:  00007fe9d0854700(0000) GS:ffff88062fc20000(0000)
knlGS:0000000000000000
[   88.705523] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   88.705524] CR2: 0000000000000008 CR3: 0000000619ccb000 CR4: 00000000000007e0
[   88.705525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   88.705526] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   88.705528] Process modprobe (pid: 3017, threadinfo
ffff88061e8fc000, task ffff8806205e8000)
[   88.705528] Stack:
[   88.705530]  ffff88062ffecd80 0000000320a95c02 0000000000000054
ffffffff00000000
[   88.705532]  0000000000000041 ffff8803215d55f8 ffff88031f8167d8
ffffffff00000000
[   88.705534]  0000000000000000 0000000100000000 ffff88062ffedb08
ffff8803200f2300
[   88.705534] Call Trace:
[   88.705542]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
[   88.705546]  [<ffffffffa007fc4c>] ? bond_dev_queue_xmit+0x62/0x7f [bonding]
[   88.705549]  [<ffffffffa0084588>] ? bond_3ad_xmit_xor+0xe7/0x10c [bonding]
[   88.705552]  [<ffffffffa007fffd>] ? bond_start_xmit+0x394/0x3ff [bonding]
[   88.705554]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
[   88.705558]  [<ffffffffa004afd5>] ?
vlan_dev_hard_start_xmit+0xab/0xf6 [8021q]
[   88.705559]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
[   88.705564]  [<ffffffffa00938e8>] ? __br_deliver+0x93/0xbe [bridge]
[   88.705567]  [<ffffffffa009237d>] ? br_dev_xmit+0x14a/0x16b [bridge]
[   88.705569]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
[   88.705570]  [<ffffffff81280372>] ? find_skb.isra.23+0x31/0x78
[   88.705572]  [<ffffffff81280bbe>] ? netpoll_send_skb+0x2c/0x39
[   88.705574]  [<ffffffffa00a222a>] ? write_msg+0x98/0xf3 [netconsole]
[   88.705579]  [<ffffffff81037db2>] ?
call_console_drivers.constprop.17+0x6e/0x7d
[   88.705580]  [<ffffffff81038248>] ? console_unlock+0x2ab/0x351
[   88.705582]  [<ffffffff81039112>] ? register_console+0x273/0x303
[   88.705584]  [<ffffffffa00fa182>] ? init_netconsole+0x182/0x210 [netconsole]
[   88.705586]  [<ffffffffa00fa000>] ? 0xffffffffa00f9fff
[   88.705588]  [<ffffffff81002085>] ? do_one_initcall+0x75/0x12c
[   88.705590]  [<ffffffff81077b35>] ? sys_init_module+0x80/0x1c5
[   88.705593]  [<ffffffff813319b9>] ? system_call_fastpath+0x16/0x1b
[   88.705606] Code: 41 c1 e1 10 48 89 d6 48 6b c8 18 48 c1 e0 04 48
c1 ee 20 49 03 8c 24 50 03 00 00 45 09 c8 44 89 4c 24 38 c7 44 24 24
00 00 00 00 <48> 89 51 08 48 89 19 49 03 84 24 48 03 00 00 89 50 04 44
89 f2
[   88.705608] RIP  [<ffffffffa0006653>] bnx2_start_xmit+0x20b/0x539 [bnx2]
[   88.705609]  RSP <ffff88061e8fda28>
[   88.705609] CR2: 0000000000000008
[   88.705611] ---[ end trace 24b75fe520341c20 ]---
[   88.705985] note: modprobe[3017] exited with preempt_count 6
[   88.706135] Dead loop on virtual device mgmt, fix it urgently!
[   88.706201] Dead loop on virtual device mgmt, fix it urgently!
[  148.557967] INFO: rcu_preempt detected stalls on CPUs/tasks: {}
(detected by 0, t=60002 jiffies)
[  148.557967] INFO: Stall ended before state dump start
[  328.112761] INFO: rcu_preempt detected stalls on CPUs/tasks: {}
(detected by 2, t=240007 jiffies)
[  328.112761] INFO: Stall ended before state dump start


And when trying on another machine that has Intel network cards, it
just completely freezes the machine ... nothing even gets printed on
the screen or anywhere I can see.

Also note that this also doesn't work in 3.5.1 so it's not a new
behavior. 3.2.x don't support netconsole over vlan at all so can't
test on it.

Cheers,

    Sylvain Munaut

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Eric Dumazet]

On Wed, 2012-08-22 at 12:53 +0200, Sylvain Munaut wrote:
> Hi again, a bit more detail:
> 
> > I'm trying to use the netconsole to feed kernel message to the outside
> > but this lead to a stall ...
> >
> > This only happens in a fairly specific configuration where you have a
> > bridge over vlan over bonding.
> > I tested with only (bridge over vlan) and (vlan over bonding) and
> > those work fine.
> >
> > [snip ... see original mail for all details]
> 
> I was previously testing under Xen.
> 
> For this round of test, I tried the kernel natively. And I also
> included Dave Miller pending series ( e0e3cea4... ) since there was
> patch related to netconsole and bridging / ...
> So in the end, it's a 3.6-rc2 + Dave Miller tree (commit  e0e3cea4 ) +
> pf malloc patch  + ip pmtu patch from Eric Dumazet.
> 
> I am now seeing more debug when I load netconsole in that config:
> 
> [   88.705138] netpoll: netconsole: local port 8888
> [   88.705140] netpoll: netconsole: local IP 10.208.1.30
> [   88.705141] netpoll: netconsole: interface 'mgmt'
> [   88.705142] netpoll: netconsole: remote port 8000
> [   88.705143] netpoll: netconsole: remote IP 10.208.1.3
> [   88.705144] netpoll: netconsole: remote ethernet address 00:16:3e:1a:37:37
> [   88.705469] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000008
> [   88.705475] IP: [<ffffffffa0006653>] bnx2_start_xmit+0x20b/0x539 [bnx2]
> [   88.705476] PGD 0
> [   88.705478] Oops: 0002 [#1] PREEMPT SMP
> [   88.705509] Modules linked in: netconsole(+) configfs nfsd
> auth_rpcgss nfs_acl nfs lockd fscache sunrpc bridge 8021q garp stp llc
> bonding ext2 iTCO_wdt iTCO_vendor_support lpc_ich mfd_core coretemp
> joydev kvm evdev crc32c_intel ghash_clmulni_intel aesni_intel
> aes_x86_64 aes_generic acpi_power_meter psmouse serio_raw dcdbas
> processor ablk_helper i7core_edac pcspkr cryptd edac_core microcode
> button hid_generic ext4 crc16 jbd2 mbcache dm_mod raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor xor async_tx
> raid6_pq raid1 raid0 multipath linear md_mod sr_mod usbhid cdrom hid
> ses sd_mod enclosure crc_t10dif usb_storage ata_generic pata_acpi uas
> uhci_hcd megaraid_sas ata_piix ehci_hcd libata usbcore scsi_mod
> usb_common bnx2
> [   88.705511] CPU 2
> [   88.705512] Pid: 3017, comm: modprobe Not tainted
> 3.6.0-rc2-00092-g9040592-dirty #6 Dell Inc. PowerEdge R610/0F0XJ6
> [   88.705515] RIP: 0010:[<ffffffffa0006653>]  [<ffffffffa0006653>]
> bnx2_start_xmit+0x20b/0x539 [bnx2]
> [   88.705516] RSP: 0018:ffff88061e8fda28  EFLAGS: 00010002
> [   88.705517] RAX: 0000000000000000 RBX: ffff8803200f2300 RCX: 0000000000000000
> [   88.705519] RDX: 0000000320a95c02 RSI: 0000000000000003 RDI: ffff8800cb36f000
> [   88.705519] RBP: ffff88031f814000 R08: 0000000000000054 R09: 0000000000000000
> [   88.705520] R10: 000000000000ffff R11: 0000000000000000 R12: ffff8803215d52c0
> [   88.705521] R13: ffff8803210e13c0 R14: 0000000000010008 R15: 0000000000000000
> [   88.705522] FS:  00007fe9d0854700(0000) GS:ffff88062fc20000(0000)
> knlGS:0000000000000000
> [   88.705523] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [   88.705524] CR2: 0000000000000008 CR3: 0000000619ccb000 CR4: 00000000000007e0
> [   88.705525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   88.705526] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [   88.705528] Process modprobe (pid: 3017, threadinfo
> ffff88061e8fc000, task ffff8806205e8000)
> [   88.705528] Stack:
> [   88.705530]  ffff88062ffecd80 0000000320a95c02 0000000000000054
> ffffffff00000000
> [   88.705532]  0000000000000041 ffff8803215d55f8 ffff88031f8167d8
> ffffffff00000000
> [   88.705534]  0000000000000000 0000000100000000 ffff88062ffedb08
> ffff8803200f2300
> [   88.705534] Call Trace:
> [   88.705542]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
> [   88.705546]  [<ffffffffa007fc4c>] ? bond_dev_queue_xmit+0x62/0x7f [bonding]
> [   88.705549]  [<ffffffffa0084588>] ? bond_3ad_xmit_xor+0xe7/0x10c [bonding]
> [   88.705552]  [<ffffffffa007fffd>] ? bond_start_xmit+0x394/0x3ff [bonding]
> [   88.705554]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
> [   88.705558]  [<ffffffffa004afd5>] ?
> vlan_dev_hard_start_xmit+0xab/0xf6 [8021q]
> [   88.705559]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
> [   88.705564]  [<ffffffffa00938e8>] ? __br_deliver+0x93/0xbe [bridge]
> [   88.705567]  [<ffffffffa009237d>] ? br_dev_xmit+0x14a/0x16b [bridge]
> [   88.705569]  [<ffffffff81280a76>] ? netpoll_send_skb_on_dev+0x201/0x31d
> [   88.705570]  [<ffffffff81280372>] ? find_skb.isra.23+0x31/0x78
> [   88.705572]  [<ffffffff81280bbe>] ? netpoll_send_skb+0x2c/0x39
> [   88.705574]  [<ffffffffa00a222a>] ? write_msg+0x98/0xf3 [netconsole]
> [   88.705579]  [<ffffffff81037db2>] ?
> call_console_drivers.constprop.17+0x6e/0x7d
> [   88.705580]  [<ffffffff81038248>] ? console_unlock+0x2ab/0x351
> [   88.705582]  [<ffffffff81039112>] ? register_console+0x273/0x303
> [   88.705584]  [<ffffffffa00fa182>] ? init_netconsole+0x182/0x210 [netconsole]
> [   88.705586]  [<ffffffffa00fa000>] ? 0xffffffffa00f9fff
> [   88.705588]  [<ffffffff81002085>] ? do_one_initcall+0x75/0x12c
> [   88.705590]  [<ffffffff81077b35>] ? sys_init_module+0x80/0x1c5
> [   88.705593]  [<ffffffff813319b9>] ? system_call_fastpath+0x16/0x1b
> [   88.705606] Code: 41 c1 e1 10 48 89 d6 48 6b c8 18 48 c1 e0 04 48
> c1 ee 20 49 03 8c 24 50 03 00 00 45 09 c8 44 89 4c 24 38 c7 44 24 24
> 00 00 00 00 <48> 89 51 08 48 89 19 49 03 84 24 48 03 00 00 89 50 04 44
> 89 f2
> [   88.705608] RIP  [<ffffffffa0006653>] bnx2_start_xmit+0x20b/0x539 [bnx2]
> [   88.705609]  RSP <ffff88061e8fda28>
> [   88.705609] CR2: 0000000000000008
> [   88.705611] ---[ end trace 24b75fe520341c20 ]---
> [   88.705985] note: modprobe[3017] exited with preempt_count 6
> [   88.706135] Dead loop on virtual device mgmt, fix it urgently!
> [   88.706201] Dead loop on virtual device mgmt, fix it urgently!
> [  148.557967] INFO: rcu_preempt detected stalls on CPUs/tasks: {}
> (detected by 0, t=60002 jiffies)
> [  148.557967] INFO: Stall ended before state dump start
> [  328.112761] INFO: rcu_preempt detected stalls on CPUs/tasks: {}
> (detected by 2, t=240007 jiffies)
> [  328.112761] INFO: Stall ended before state dump start
> 
> 
> And when trying on another machine that has Intel network cards, it
> just completely freezes the machine ... nothing even gets printed on
> the screen or anywhere I can see.
> 
> Also note that this also doesn't work in 3.5.1 so it's not a new
> behavior. 3.2.x don't support netconsole over vlan at all so can't
> test on it.
> 
> Cheers,
> 
>   

Could be the infamous slave_dev_queue_mapping striking again.

Could you please try :

diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index 346b1eb..df731a0 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -335,8 +335,11 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
 	/* don't get messages out of order, and no recursion */
 	if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
 		struct netdev_queue *txq;
+		int queue_index = skb_get_queue_mapping(skb);
 
-		txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
+		if (queue_index >= dev->real_num_tx_queues)
+			queue_index = 0;
+		txq = netdev_get_tx_queue(dev, queue_index);
 
 		/* try until next clock tick */
 		for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi,

> Could be the infamous slave_dev_queue_mapping striking again.
>
> Could you please try :
>
> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
> index 346b1eb..df731a0 100644
> --- a/net/core/netpoll.c
> +++ b/net/core/netpoll.c
> @@ -335,8 +335,11 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>         /* don't get messages out of order, and no recursion */
>         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>                 struct netdev_queue *txq;
> +               int queue_index = skb_get_queue_mapping(skb);
>
> -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
> +               if (queue_index >= dev->real_num_tx_queues)
> +                       queue_index = 0;
> +               txq = netdev_get_tx_queue(dev, queue_index);
>
>                 /* try until next clock tick */
>                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;


Well, it doesn't solve the problem :(

It does have an effect though. Now even on the machine with the
broadcom card, it just freeze the machine ...
On the machine with intel card, it actually does get a couple of
netconsole packet out and then freeze as well.


FYI this is the disass of bnx2 module around the issue :

   0x00000000000065f9 <+433>:	mov    %rax,%rsi
   0x00000000000065fc <+436>:	mov    %rax,0x8(%rsp)
   0x0000000000006601 <+441>:	add    $0x98,%rdi
   0x0000000000006608 <+448>:	callq  0xac9 <dma_mapping_error>
   0x000000000000660d <+453>:	test   %eax,%eax
   0x000000000000660f <+455>:	mov    0x8(%rsp),%rdx
   0x0000000000006614 <+460>:	mov    0x10(%rsp),%r8d
   0x0000000000006619 <+465>:	mov    0x18(%rsp),%r9d
   0x000000000000661e <+470>:	jne    0x6966 <bnx2_start_xmit+1310>
   0x0000000000006624 <+476>:	movzbl %r15b,%eax
   0x0000000000006628 <+480>:	shl    $0x10,%r9d
   0x000000000000662c <+484>:	mov    %rdx,%rsi
   0x000000000000662f <+487>:	imul   $0x18,%rax,%rcx
   0x0000000000006633 <+491>:	shl    $0x4,%rax
   0x0000000000006637 <+495>:	shr    $0x20,%rsi
   0x000000000000663b <+499>:	add    0x350(%r12),%rcx
   0x0000000000006643 <+507>:	or     %r9d,%r8d
   0x0000000000006646 <+510>:	mov    %r9d,0x38(%rsp)
   0x000000000000664b <+515>:	movl   $0x0,0x24(%rsp)
   0x0000000000006653 <+523>:	mov    %rdx,0x8(%rcx)
   0x0000000000006657 <+527>:	mov    %rbx,(%rcx)
   0x000000000000665a <+530>:	add    0x348(%r12),%rax
   0x0000000000006662 <+538>:	mov    %edx,0x4(%rax)
   0x0000000000006665 <+541>:	mov    %r14d,%edx
   0x0000000000006668 <+544>:	mov    %esi,(%rax)
   0x000000000000666a <+546>:	or     $0x80,%dl
   0x000000000000666d <+549>:	mov    %r8d,0x8(%rax)
   0x0000000000006671 <+553>:	mov    %edx,0xc(%rax)


The issue it at this line :

 0x0000000000006653 <+523>:	mov    %rdx,0x8(%rcx)

RCX is NULL it seems.


Cheers,

    Sylvain Munaut

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Eric Dumazet]

On Wed, 2012-08-22 at 14:17 +0200, Sylvain Munaut wrote:
> Hi,
> 
> > Could be the infamous slave_dev_queue_mapping striking again.
> >
> > Could you please try :
> >
> > diff --git a/net/core/netpoll.c b/net/core/netpoll.c
> > index 346b1eb..df731a0 100644
> > --- a/net/core/netpoll.c
> > +++ b/net/core/netpoll.c
> > @@ -335,8 +335,11 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
> >         /* don't get messages out of order, and no recursion */
> >         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
> >                 struct netdev_queue *txq;
> > +               int queue_index = skb_get_queue_mapping(skb);
> >
> > -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
> > +               if (queue_index >= dev->real_num_tx_queues)
> > +                       queue_index = 0;
> > +               txq = netdev_get_tx_queue(dev, queue_index);
> >
> >                 /* try until next clock tick */
> >                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;
> 
> 
> Well, it doesn't solve the problem :(
> 
> It does have an effect though. Now even on the machine with the
> broadcom card, it just freeze the machine ...
> On the machine with intel card, it actually does get a couple of
> netconsole packet out and then freeze as well.
> 

my patch was incomplete, sorry :

diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index 346b1eb..ddc453b 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -335,8 +335,13 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
 	/* don't get messages out of order, and no recursion */
 	if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
 		struct netdev_queue *txq;
+		int queue_index = skb_get_queue_mapping(skb);
 
-		txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
+		if (queue_index >= dev->real_num_tx_queues) {
+			queue_index = 0;
+			skb_set_queue_mapping(skb, 0);
+		}
+		txq = netdev_get_tx_queue(dev, queue_index);
 
 		/* try until next clock tick */
 		for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi,

> my patch was incomplete, sorry :
>
> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
> index 346b1eb..ddc453b 100644
> --- a/net/core/netpoll.c
> +++ b/net/core/netpoll.c
> @@ -335,8 +335,13 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>         /* don't get messages out of order, and no recursion */
>         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>                 struct netdev_queue *txq;
> +               int queue_index = skb_get_queue_mapping(skb);
>
> -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
> +               if (queue_index >= dev->real_num_tx_queues) {
> +                       queue_index = 0;
> +                       skb_set_queue_mapping(skb, 0);
> +               }
> +               txq = netdev_get_tx_queue(dev, queue_index);
>
>                 /* try until next clock tick */
>                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;

Ok, I tried this.

The machine with the intel card still hard freeze (no output / no nothing ...)
The machine with the bnx2 don't crash anymore and no NULL deref, but
the modprobe still hangs and I get this every 180 sec or so :

[  775.956926] INFO: rcu_preempt self-detected stall on CPU { 14}
(t=600009 jiffies)
[  775.956927] Pid: 3154, comm: modprobe Not tainted
3.6.0-rc2-00092-g9040592-dirty #8
[  775.956928] Call Trace:
[  775.956931]  <IRQ>  [<ffffffff81094a58>] ? rcu_pending+0xee/0x490
[  775.956933]  [<ffffffff810571da>] ?
irqtime_account_process_tick.isra.73+0x134/0x23a
[  775.956934]  [<ffffffff81095406>] ? rcu_check_callbacks+0x79/0x85
[  775.956936]  [<ffffffff81041609>] ? update_process_times+0x30/0x62
[  775.956938]  [<ffffffff8106f204>] ? tick_sched_timer+0x75/0x9e
[  775.956939]  [<ffffffff8106f18f>] ? tick_nohz_handler+0xd0/0xd0
[  775.956941]  [<ffffffff810501cf>] ? __run_hrtimer.isra.26+0x75/0xcd
[  775.956943]  [<ffffffff81050873>] ? hrtimer_interrupt+0xe2/0x1f0
[  775.956948]  [<ffffffffa000941d>] ? bnx2_poll_work+0x2d3/0xa52 [bnx2]
[  775.956950]  [<ffffffff8100fa5c>] ? sched_clock+0x5/0x8
[  775.956952]  [<ffffffff81023d93>] ? smp_apic_timer_interrupt+0x6d/0x7f
[  775.956954]  [<ffffffff8133244a>] ? apic_timer_interrupt+0x6a/0x70
[  775.956956]  [<ffffffff8126d407>] ? __napi_complete+0x1c/0x23
[  775.956958]  [<ffffffff81074249>] ? do_raw_spin_lock+0x18/0x1b
[  775.956960]  [<ffffffff8126e596>] ? net_rx_action+0x7f/0x185
[  775.956962]  [<ffffffff8100f80e>] ? __cycles_2_ns+0x9/0x45
[  775.956964]  [<ffffffff8103caae>] ? __do_softirq+0x9c/0x14b
[  775.956966]  [<ffffffff81332b3c>] ? call_softirq+0x1c/0x30
[  775.956968]  <EOI>  [<ffffffff8100aea6>] ? do_softirq+0x3c/0x7a
[  775.956970]  [<ffffffff8103c9db>] ? _local_bh_enable_ip.isra.7+0x76/0xa3
[  775.956972]  [<ffffffff812804b7>] ? netpoll_poll_dev+0xfe/0x4bc
[  775.956974]  [<ffffffff81280b02>] ? netpoll_send_skb_on_dev+0x28d/0x33b
[  775.956978]  [<ffffffffa0ff2c4c>] ? bond_dev_queue_xmit+0x62/0x7f [bonding]
[  775.956982]  [<ffffffffa0ff7588>] ? bond_3ad_xmit_xor+0xe7/0x10c [bonding]
[  775.956984]  [<ffffffffa0ff2ffd>] ? bond_start_xmit+0x394/0x3ff [bonding]
[  775.956987]  [<ffffffff81280ac1>] ? netpoll_send_skb_on_dev+0x24c/0x33b
[  775.956990]  [<ffffffffa0079fd5>] ?
vlan_dev_hard_start_xmit+0xab/0xf6 [8021q]
[  775.956992]  [<ffffffff81280ac1>] ? netpoll_send_skb_on_dev+0x24c/0x33b
[  775.956996]  [<ffffffffa01998e8>] ? __br_deliver+0x93/0xbe [bridge]
[  775.956998]  [<ffffffffa019837d>] ? br_dev_xmit+0x14a/0x16b [bridge]
[  775.957001]  [<ffffffff81280ac1>] ? netpoll_send_skb_on_dev+0x24c/0x33b
[  775.957003]  [<ffffffff81280372>] ? find_skb.isra.24+0x31/0x78
[  775.957005]  [<ffffffff81280bdc>] ? netpoll_send_skb+0x2c/0x39
[  775.957007]  [<ffffffffa00c422a>] ? write_msg+0x98/0xf3 [netconsole]
[  775.957009]  [<ffffffff81037db2>] ?
call_console_drivers.constprop.17+0x6e/0x7d
[  775.957011]  [<ffffffff81038248>] ? console_unlock+0x2ab/0x351
[  775.957012]  [<ffffffff81039112>] ? register_console+0x273/0x303
[  775.957014]  [<ffffffffa0103182>] ? init_netconsole+0x182/0x210 [netconsole]
[  775.957016]  [<ffffffffa0103000>] ? 0xffffffffa0102fff
[  775.957018]  [<ffffffff81002085>] ? do_one_initcall+0x75/0x12c
[  775.957019]  [<ffffffff81077b35>] ? sys_init_module+0x80/0x1c5
[  775.957020]  [<ffffffff813319b9>] ? system_call_fastpath+0x16/0x1b


Cheers,

    Sylvain

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On Wed, 22 Aug 2012 at 14:29 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
> Hi,
>
>> my patch was incomplete, sorry :
>>
>> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
>> index 346b1eb..ddc453b 100644
>> --- a/net/core/netpoll.c
>> +++ b/net/core/netpoll.c
>> @@ -335,8 +335,13 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>>         /* don't get messages out of order, and no recursion */
>>         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>>                 struct netdev_queue *txq;
>> +               int queue_index = skb_get_queue_mapping(skb);
>>
>> -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
>> +               if (queue_index >= dev->real_num_tx_queues) {
>> +                       queue_index = 0;
>> +                       skb_set_queue_mapping(skb, 0);
>> +               }
>> +               txq = netdev_get_tx_queue(dev, queue_index);
>>
>>                 /* try until next clock tick */
>>                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;
>
> Ok, I tried this.
>
> The machine with the intel card still hard freeze (no output / no nothing ...)
> The machine with the bnx2 don't crash anymore and no NULL deref, but
> the modprobe still hangs and I get this every 180 sec or so :
>


Thanks for reporting this!

I will try to see if I can reproduce it on KVM guest tomorrow.
Need to go to sleep now.

To be honest, I never try to setup netconsole on such a complex NIC,
bridge on vlan tagged bonding.

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On Wed, 22 Aug 2012 at 14:29 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
> Hi,
>
>
> The machine with the intel card still hard freeze (no output / no nothing ...)
> The machine with the bnx2 don't crash anymore and no NULL deref, but
> the modprobe still hangs and I get this every 180 sec or so :

The NULL-deref can be reproduced easily, and Eric's patch could fix it.
So, Eric, can you resend your patch with your SOB?

I can't reproduce the hang as it is net driver specific, it is
probably related with my patch:

commit 6bdb7fe31046ac50b47e83c35cd6c6b6160a475d
Author: Amerigo Wang <amwang@redhat.com>
Date:   Fri Aug 10 01:24:50 2012 +0000

    netpoll: re-enable irq in poll_napi()

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On Thu, 23 Aug 2012 at 07:57 GMT, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> On Wed, 22 Aug 2012 at 14:29 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
>> Hi,
>>
>>
>> The machine with the intel card still hard freeze (no output / no nothing ...)
>> The machine with the bnx2 don't crash anymore and no NULL deref, but
>> the modprobe still hangs and I get this every 180 sec or so :
>
> The NULL-deref can be reproduced easily, and Eric's patch could fix it.
> So, Eric, can you resend your patch with your SOB?
>
> I can't reproduce the hang as it is net driver specific, it is
> probably related with my patch:
>
> commit 6bdb7fe31046ac50b47e83c35cd6c6b6160a475d
> Author: Amerigo Wang <amwang@redhat.com>
> Date:   Fri Aug 10 01:24:50 2012 +0000
>
>     netpoll: re-enable irq in poll_napi()
>

Could you test the following patch?

diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index ddc453b..ed4d1e4 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -166,11 +166,18 @@ static int poll_one_napi(struct netpoll_info *npinfo,
 static void poll_napi(struct net_device *dev)
 {
 	struct napi_struct *napi;
+	LIST_HEAD(napi_list);
 	int budget = 16;
 
 	WARN_ON_ONCE(!irqs_disabled());
 
-	list_for_each_entry(napi, &dev->napi_list, dev_list) {
+	/* After we enable the IRQ, new entries could be added
+	 * to this list, we need to save it before re-enable
+	 * IRQ.
+	 */
+	list_splice_tail(&dev->napi_list, &napi_list);
+
+	list_for_each_entry(napi, &napi_list, dev_list) {
 		local_irq_enable();
 		if (napi->poll_owner != smp_processor_id() &&
 		    spin_trylock(&napi->poll_lock)) {
@@ -187,6 +194,7 @@ static void poll_napi(struct net_device *dev)
 		}
 		local_irq_disable();
 	}
+	list_splice_tail(&napi_list, &dev->napi_list);
 }
 
 static void service_arp_queue(struct netpoll_info *npi)




However, it seems we should take rtnl lock to make sure dev->napi_list
is really safe, I am not sure if the following one makes sense.


diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index ddc453b..7770e2b 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -170,8 +170,9 @@ static void poll_napi(struct net_device *dev)
 
 	WARN_ON_ONCE(!irqs_disabled());
 
+	local_irq_enable();
+	rtnl_lock();
 	list_for_each_entry(napi, &dev->napi_list, dev_list) {
-		local_irq_enable();
 		if (napi->poll_owner != smp_processor_id() &&
 		    spin_trylock(&napi->poll_lock)) {
 			rcu_read_lock_bh();
@@ -180,13 +181,12 @@ static void poll_napi(struct net_device *dev)
 			rcu_read_unlock_bh();
 			spin_unlock(&napi->poll_lock);
 
-			if (!budget) {
-				local_irq_disable();
+			if (!budget)
 				break;
-			}
 		}
-		local_irq_disable();
 	}
+	rtnl_unlock();
+	local_irq_disable();
 }
 
 static void service_arp_queue(struct netpoll_info *npi)

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On Thu, 23 Aug 2012 at 08:31 GMT, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> On Thu, 23 Aug 2012 at 07:57 GMT, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>> On Wed, 22 Aug 2012 at 14:29 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
>>> Hi,
>>>
>>>
>>> The machine with the intel card still hard freeze (no output / no nothing ...)
>>> The machine with the bnx2 don't crash anymore and no NULL deref, but
>>> the modprobe still hangs and I get this every 180 sec or so :
>>
>> The NULL-deref can be reproduced easily, and Eric's patch could fix it.
>> So, Eric, can you resend your patch with your SOB?
>>
>> I can't reproduce the hang as it is net driver specific, it is
>> probably related with my patch:
>>
>> commit 6bdb7fe31046ac50b47e83c35cd6c6b6160a475d
>> Author: Amerigo Wang <amwang@redhat.com>
>> Date:   Fri Aug 10 01:24:50 2012 +0000
>>
>>     netpoll: re-enable irq in poll_napi()
>>
>
> Could you test the following patch?
>
> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
> index ddc453b..ed4d1e4 100644
> --- a/net/core/netpoll.c
> +++ b/net/core/netpoll.c
> @@ -166,11 +166,18 @@ static int poll_one_napi(struct netpoll_info *npinfo,
>  static void poll_napi(struct net_device *dev)
>  {
>  	struct napi_struct *napi;
> +	LIST_HEAD(napi_list);
>  	int budget = 16;
>  
>  	WARN_ON_ONCE(!irqs_disabled());
>  
> -	list_for_each_entry(napi, &dev->napi_list, dev_list) {
> +	/* After we enable the IRQ, new entries could be added
> +	 * to this list, we need to save it before re-enable
> +	 * IRQ.
> +	 */
> +	list_splice_tail(&dev->napi_list, &napi_list);
> +

This one should be list_splice_init()...


> +	list_for_each_entry(napi, &napi_list, dev_list) {
>  		local_irq_enable();
>  		if (napi->poll_owner != smp_processor_id() &&
>  		    spin_trylock(&napi->poll_lock)) {
> @@ -187,6 +194,7 @@ static void poll_napi(struct net_device *dev)
>  		}
>  		local_irq_disable();
>  	}
> +	list_splice_tail(&napi_list, &dev->napi_list);
>  }
>  
>  static void service_arp_queue(struct netpoll_info *npi)
>
>
>
>
> However, it seems we should take rtnl lock to make sure dev->napi_list
> is really safe, I am not sure if the following one makes sense.
>
>
> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
> index ddc453b..7770e2b 100644
> --- a/net/core/netpoll.c
> +++ b/net/core/netpoll.c
> @@ -170,8 +170,9 @@ static void poll_napi(struct net_device *dev)
>  
>  	WARN_ON_ONCE(!irqs_disabled());
>  
> +	local_irq_enable();
> +	rtnl_lock();
>  	list_for_each_entry(napi, &dev->napi_list, dev_list) {
> -		local_irq_enable();
>  		if (napi->poll_owner != smp_processor_id() &&
>  		    spin_trylock(&napi->poll_lock)) {
>  			rcu_read_lock_bh();
> @@ -180,13 +181,12 @@ static void poll_napi(struct net_device *dev)
>  			rcu_read_unlock_bh();
>  			spin_unlock(&napi->poll_lock);
>  
> -			if (!budget) {
> -				local_irq_disable();
> +			if (!budget)
>  				break;
> -			}
>  		}
> -		local_irq_disable();
>  	}
> +	rtnl_unlock();
> +	local_irq_disable();
>  }
>  
>  static void service_arp_queue(struct netpoll_info *npi)
>

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi,

>>
>> Could you test the following patch?
>>
>> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
>> index ddc453b..ed4d1e4 100644
>> --- a/net/core/netpoll.c
>> +++ b/net/core/netpoll.c
>> @@ -166,11 +166,18 @@ static int poll_one_napi(struct netpoll_info *npinfo,
>>  static void poll_napi(struct net_device *dev)
>>  {
>>       struct napi_struct *napi;
>> +     LIST_HEAD(napi_list);
>>       int budget = 16;
>>
>>       WARN_ON_ONCE(!irqs_disabled());
>>
>> -     list_for_each_entry(napi, &dev->napi_list, dev_list) {
>> +     /* After we enable the IRQ, new entries could be added
>> +      * to this list, we need to save it before re-enable
>> +      * IRQ.
>> +      */
>> +     list_splice_tail(&dev->napi_list, &napi_list);
>> +
>
> This one should be list_splice_init()...
>
>
>> +     list_for_each_entry(napi, &napi_list, dev_list) {
>>               local_irq_enable();
>>               if (napi->poll_owner != smp_processor_id() &&
>>                   spin_trylock(&napi->poll_lock)) {
>> @@ -187,6 +194,7 @@ static void poll_napi(struct net_device *dev)
>>               }
>>               local_irq_disable();
>>       }
>> +     list_splice_tail(&napi_list, &dev->napi_list);
>>  }
>>
>>  static void service_arp_queue(struct netpoll_info *npi)

I've just tested this patch on the intel machine and the behavior didn't change.
When I do the netconsole modprobe, it sends a couple of line, the
modprobe hangs and then a couple of second later the whole machine
hangs, with nothing printed on the screen or anything.

Cheers,

    Sylvain

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Lin Ming]

On Wed, Aug 22, 2012 at 10:29 PM, Sylvain Munaut
<s.munaut@whatever-company.com> wrote:
> Hi,
>
>> my patch was incomplete, sorry :
>>
>> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
>> index 346b1eb..ddc453b 100644
>> --- a/net/core/netpoll.c
>> +++ b/net/core/netpoll.c
>> @@ -335,8 +335,13 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>>         /* don't get messages out of order, and no recursion */
>>         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>>                 struct netdev_queue *txq;
>> +               int queue_index = skb_get_queue_mapping(skb);
>>
>> -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
>> +               if (queue_index >= dev->real_num_tx_queues) {
>> +                       queue_index = 0;
>> +                       skb_set_queue_mapping(skb, 0);
>> +               }
>> +               txq = netdev_get_tx_queue(dev, queue_index);
>>
>>                 /* try until next clock tick */
>>                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;
>
> Ok, I tried this.
>
> The machine with the intel card still hard freeze (no output / no nothing ...)

Did you enable hard lockup detector?

CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y

> The machine with the bnx2 don't crash anymore and no NULL deref, but
> the modprobe still hangs and I get this every 180 sec or so :
>

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On Fri, Aug 24, 2012 at 5:50 PM, Sylvain Munaut
<s.munaut@whatever-company.com> wrote:
>
> I've just tested this patch on the intel machine and the behavior didn't change.
> When I do the netconsole modprobe, it sends a couple of line, the
> modprobe hangs and then a couple of second later the whole machine
> hangs, with nothing printed on the screen or anything.

Hi, Sylvain

I just sent a new patch:
http://marc.info/?l=linux-netdev&m=134588049503667&w=2

Please help to test it.

Thanks a lot!

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi Eric,

AFAICT, the following patch was never merged but I just confirmed that
I do need it over 3.6-rc5 for things to work properly.

>> > diff --git a/net/core/netpoll.c b/net/core/netpoll.c
>> > index 346b1eb..df731a0 100644
>> > --- a/net/core/netpoll.c
>> > +++ b/net/core/netpoll.c
>> > @@ -335,8 +335,11 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>> >         /* don't get messages out of order, and no recursion */
>> >         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>> >                 struct netdev_queue *txq;
>> > +               int queue_index = skb_get_queue_mapping(skb);
>> >
>> > -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
>> > +               if (queue_index >= dev->real_num_tx_queues)
>> > +                       queue_index = 0;
>> > +               txq = netdev_get_tx_queue(dev, queue_index);
>> >
>> >                 /* try until next clock tick */
>> >                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;
>>
>>
>> Well, it doesn't solve the problem :(
>>
>> It does have an effect though. Now even on the machine with the
>> broadcom card, it just freeze the machine ...
>> On the machine with intel card, it actually does get a couple of
>> netconsole packet out and then freeze as well.
>>
>
> my patch was incomplete, sorry :
>
> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
> index 346b1eb..ddc453b 100644
> --- a/net/core/netpoll.c
> +++ b/net/core/netpoll.c
> @@ -335,8 +335,13 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>         /* don't get messages out of order, and no recursion */
>         if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>                 struct netdev_queue *txq;
> +               int queue_index = skb_get_queue_mapping(skb);
>
> -               txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
> +               if (queue_index >= dev->real_num_tx_queues) {
> +                       queue_index = 0;
> +                       skb_set_queue_mapping(skb, 0);
> +               }
> +               txq = netdev_get_tx_queue(dev, queue_index);
>
>                 /* try until next clock tick */
>                 for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;

Cheers,

    Sylvain

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On Wed, 12 Sep 2012 at 11:53 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
> Hi Eric,
>
> AFAICT, the following patch was never merged but I just confirmed that
> I do need it over 3.6-rc5 for things to work properly.
>

Yes, indeed.

Eric, please resend your patch?

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Eric Dumazet]

On Wed, 2012-09-12 at 12:49 +0000, Cong Wang wrote:
> On Wed, 12 Sep 2012 at 11:53 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
> > Hi Eric,
> >
> > AFAICT, the following patch was never merged but I just confirmed that
> > I do need it over 3.6-rc5 for things to work properly.
> >
> 
> Yes, indeed.
> 
> Eric, please resend your patch?
> 

Yes, but I have some worries of why it is needed.

Isnt it covering a bug elsewhere ?

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi,

> Yes, but I have some worries of why it is needed.
>
> Isnt it covering a bug elsewhere ?

That may very well be.

Of the few test servers I have running the same kernel, I just found
the one with netconsole active to be "stuck".

Not frozen, but all user process are hanged up and it's spitting
message about processes and CPU being "stuck". The trace is different
in each case depending on what the process was actually doing at the
time it got stuck.

No message sent to the netconsole with the root cause and nothing was
written in the logs ...

Cheers,

    Sylvain

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On 09/14/2012 01:35 AM, Sylvain Munaut wrote:
> Hi,
>
>> Yes, but I have some worries of why it is needed.
>>
>> Isnt it covering a bug elsewhere ?
>
> That may very well be.
>
> Of the few test servers I have running the same kernel, I just found
> the one with netconsole active to be "stuck".
>
> Not frozen, but all user process are hanged up and it's spitting
> message about processes and CPU being "stuck". The trace is different
> in each case depending on what the process was actually doing at the
> time it got stuck.
>
> No message sent to the netconsole with the root cause and nothing was
> written in the logs ...
>

Yeah, in this case, kdump is your friend. :)

Anyway, I think Eric is right, the bug may be in other place. I am 
wondering if the attached patch could help? It seems in netpoll tx path, 
we miss the chance of calling ->ndo_select_queue().

Please give it a try.

Thanks!

[-- Attachment #2: netpoll-txq.diff --]
[-- Type: text/x-patch, Size: 1645 bytes --]

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index ae3153c0..72661f6 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1403,6 +1403,9 @@ static inline void netdev_for_each_tx_queue(struct net_device *dev,
 		f(dev, &dev->_tx[i], arg);
 }
 
+extern struct netdev_queue *netdev_pick_tx(struct net_device *dev,
+					   struct sk_buff *skb);
+
 /*
  * Net namespace inlines
  */
diff --git a/net/core/dev.c b/net/core/dev.c
index b1e6d63..d76bf73 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2381,8 +2381,8 @@ static inline int get_xps_queue(struct net_device *dev, struct sk_buff *skb)
 #endif
 }
 
-static struct netdev_queue *dev_pick_tx(struct net_device *dev,
-					struct sk_buff *skb)
+struct netdev_queue *netdev_pick_tx(struct net_device *dev,
+				    struct sk_buff *skb)
 {
 	int queue_index;
 	const struct net_device_ops *ops = dev->netdev_ops;
@@ -2556,7 +2556,7 @@ int dev_queue_xmit(struct sk_buff *skb)
 
 	skb_update_prio(skb);
 
-	txq = dev_pick_tx(dev, skb);
+	txq = netdev_pick_tx(dev, skb);
 	q = rcu_dereference_bh(txq->qdisc);
 
 #ifdef CONFIG_NET_CLS_ACT
diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index dd67818..77a0388 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -328,7 +328,7 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
 	if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
 		struct netdev_queue *txq;
 
-		txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
+		txq = netdev_pick_tx(dev, skb);
 
 		/* try until next clock tick */
 		for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi,

> Anyway, I think Eric is right, the bug may be in other place. I am wondering
> if the attached patch could help? It seems in netpoll tx path, we miss the
> chance of calling ->ndo_select_queue().
>
> Please give it a try.

This fixes the NULL deref as well. I applied your patch over vanilla
3.6-rc5 (so without eric's patch) and I can load netconsole without it
crashing directly.

I will try to put a bit of load on the machine and see how it behaves,
if it solves the hung cpu tasks as well.

Cheers,

    Sylvain

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Sylvain Munaut]

Hi,

>> Anyway, I think Eric is right, the bug may be in other place. I am wondering
>> if the attached patch could help? It seems in netpoll tx path, we miss the
>> chance of calling ->ndo_select_queue().
>>
>> Please give it a try.
>
> This fixes the NULL deref as well. I applied your patch over vanilla
> 3.6-rc5 (so without eric's patch) and I can load netconsole without it
> crashing directly.
>
> I will try to put a bit of load on the machine and see how it behaves,
> if it solves the hung cpu tasks as well.

So far so good, the machine is still alive and well after 3 days with
netconsole enabled.


Cheers,

    Sylvain

Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )

[Cong Wang]

On 09/17/2012 06:57 PM, Sylvain Munaut wrote:
> Hi,
>
>>> Anyway, I think Eric is right, the bug may be in other place. I am wondering
>>> if the attached patch could help? It seems in netpoll tx path, we miss the
>>> chance of calling ->ndo_select_queue().
>>>
>>> Please give it a try.
>>
>> This fixes the NULL deref as well. I applied your patch over vanilla
>> 3.6-rc5 (so without eric's patch) and I can load netconsole without it
>> crashing directly.
>>
>> I will try to put a bit of load on the machine and see how it behaves,
>> if it solves the hung cpu tasks as well.
>
> So far so good, the machine is still alive and well after 3 days with
> netconsole enabled.
>

Great!

Thanks a lot for testing, Sylvain! I will post the patch tomorrow, as I 
need some sleep now. :)