From: Matheos Worku <Matheos.Worku@Sun.COM>
To: Jarek Poplawski <jarkao2@gmail.com>
Cc: netdev@vger.kernel.org
Subject: Re: 2.6.24 BUG: soft lockup - CPU#X
Date: Wed, 26 Mar 2008 13:26:00 -0700 [thread overview]
Message-ID: <47EAB158.3080806@sun.com> (raw)
In-Reply-To: <47EAAE9A.9050305@gmail.com>
Jarek Poplawski wrote:
> Matheos Worku wrote, On 03/26/2008 05:46 PM:
> ...
>
>
>> outside the driver as well. I have attached several lockup error
>> traces and corresponding profile data. Any clues?
>>
>
> Are network cards' irqs balanced? If so, could you reproduce this
> with affinity set?
>
> Regards,
> Jarek P.
>
Jarek,
Reproduced the lockup with irqbalance disabled and with single src of
interrupt (TX interrupt, UDP transmit). Lockup appears in different
location though.
Regards
matheos
irq of interest: 454 (TX interrupt)
454: 19249 93234 907186 2691 0
188 0 160 PCI-MSI-edge eth6
455: 22607 15083 5 13104 25569
161519 62514 25637 PCI-MSI-edge eth6
456: 22390 14921 5 24605 37438
110453 251315 66 PCI-MSI-edge eth6
457: 11109 26849 2 58895 251720
84 0 67420 PCI-MSI-edge eth6
458: 22348 15859 1 21978 27839
10231 0 267743 PCI-MSI-edge eth6
459: 19922 15331 2 59275 0
149788 12394 82549 PCI-MSI-edge eth6
460: 22928 19058 4 1268 49775
183189 160901 25150 PCI-MSI-edge eth6
461: 497 32134 1 31428 0
69182 68889 45407 PCI-MSI-edge eth6
462: 11932 23212 10 11355 120509
47588 1 118637 PCI-MSI-edge eth6
463: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth6
464: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth6
465: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth6
.......
454: 19249 126519 907186 2691 0
188 0 160 PCI-MSI-edge eth6
455: 22609 15083 5 13104 25569
161519 62514 25637 PCI-MSI-edge eth6
456: 22390 14923 5 24605 37438
110453 251315 66 PCI-MSI-edge eth6
457: 11109 26849 2 58895 251720
84 0 67420 PCI-MSI-edge eth6
458: 22348 15867 1 21978 27839
10231 0 267744 PCI-MSI-edge eth6
459: 19922 15331 2 59275 0
149788 12394 82549 PCI-MSI-edge eth6
460: 22928 19058 4 1268 49775
183189 160901 25150 PCI-MSI-edge eth6
461: 498 32134 1 31428 0
69182 68889 45407 PCI-MSI-edge eth6
462: 11932 23216 10 11355 120509
47588 1 118637 PCI-MSI-edge eth6
463: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth6
464: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth6
465: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth6
nsn57-110 login: BUG: soft lockup - CPU#2 stuck for 11s!
[uperf.x86_64:16606]
CPU 2:
Modules linked in: ixgbe oprofile niu nfs lockd nfs_acl autofs4 hidp
rfcomm l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm
rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp ib_cm ib_ipoib ib_sa
ib_uverbs ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp
parport joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core
dm_snapshot dm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 16606, comm: uperf.x86_64 Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff803ef525>] [<ffffffff803ef525>]
__copy_skb_header+0x10d/0x134
RSP: 0018:ffff8101ae14ba38 EFLAGS: 00000246
RAX: 0000000020000000 RBX: ffff8101d059a400 RCX: 000000000000000c
RDX: 0000000000000000 RSI: ffff8101d059a468 RDI: ffff8101f7db4868
RBP: ffff8101ffe50d80 R08: ffff8101f7db4800 R09: ffff8101d059a400
R10: 00000001b1c64660 R11: ffffffff80221995 R12: 0000000000000000
R13: 0000000100000000 R14: ffffffff802858e4 R15: ffff8101fec71900
FS: 0000000040800940(0063) GS:ffff8101fb072700(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000044005f48 CR3: 00000001d0513000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff803ef5f6>] __skb_clone+0x24/0xdc
[<ffffffff803f152e>] skb_realloc_headroom+0x30/0x63
[<ffffffff882edd40>] :niu:niu_start_xmit+0x114/0x5af
[<ffffffff80221995>] gart_map_single+0x0/0x70
[<ffffffff803f5e2b>] dev_hard_start_xmit+0x1d2/0x246
[<ffffffff80406fb8>] pfifo_fast_dequeue+0x3b/0x59
[<ffffffff80406dab>] __qdisc_run+0x77/0x174
[<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
[<ffffffff80417faf>] ip_push_pending_frames+0x32c/0x3a0
[<ffffffff80419676>] ip_generic_getfrag+0x0/0x8b
[<ffffffff8043359f>] udp_push_pending_frames+0x2ba/0x337
[<ffffffff80434794>] udp_sendmsg+0x4c8/0x606
[<ffffffff803eafbb>] sock_sendmsg+0xe2/0xff
[<ffffffff8029e1a1>] iput+0x42/0x7b
[<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
[<ffffffff80275d0c>] find_extend_vma+0x16/0x59
[<ffffffff8045e4d3>] _spin_lock_irqsave+0x9/0xe
[<ffffffff80311d88>] __up_read+0x13/0x8a
[<ffffffff803eba5c>] sys_sendto+0x128/0x151
[<ffffffff8045e3ed>] _spin_unlock_bh+0x9/0x15
[<ffffffff8020b7fc>] tracesys+0xdc/0xe1
BUG: soft lockup - CPU#2 stuck for 11s! [uperf.x86_64:16606]
CPU 2:
Modules linked in: ixgbe oprofile niu nfs lockd nfs_acl autofs4 hidp
rfcomm l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm
rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp ib_cm ib_ipoib ib_sa
ib_uverbs ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp
parport joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core
dm_snapshot dm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 16606, comm: uperf.x86_64 Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff803ef462>] [<ffffffff803ef462>]
__copy_skb_header+0x4a/0x134
RSP: 0018:ffff8101ae14ba38 EFLAGS: 00000202
RAX: ffff8101fa048300 RBX: ffff8103fb35c100 RCX: ffffffff803f0453
RDX: ffff8101fa1e5d00 RSI: ffff8103fb35c100 RDI: ffff8101fa1e5d00
RBP: 0000000000000020 R08: ffff8101fa1e5d00 R09: ffff8103fb35c100
R10: 00000001c6920e60 R11: ffffffff80221995 R12: ffff810100052cc0
R13: ffffffff805abb88 R14: ffff8101ff231b80 R15: 0000000000000000
FS: 0000000040800940(0063) GS:ffff8101fb072700(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000044005f48 CR3: 00000001d0513000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff803ef5f6>] __skb_clone+0x24/0xdc
[<ffffffff803f152e>] skb_realloc_headroom+0x30/0x63
[<ffffffff882edd40>] :niu:niu_start_xmit+0x114/0x5af
[<ffffffff80221995>] gart_map_single+0x0/0x70
[<ffffffff803f5e2b>] dev_hard_start_xmit+0x1d2/0x246
[<ffffffff80406daf>] __qdisc_run+0x7b/0x174
[<ffffffff80406dab>] __qdisc_run+0x77/0x174
[<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
[<ffffffff80417faf>] ip_push_pending_frames+0x32c/0x3a0
[<ffffffff80419676>] ip_generic_getfrag+0x0/0x8b
[<ffffffff8043359f>] udp_push_pending_frames+0x2ba/0x337
[<ffffffff80434794>] udp_sendmsg+0x4c8/0x606
[<ffffffff803eafbb>] sock_sendmsg+0xe2/0xff
[<ffffffff8029e1a1>] iput+0x42/0x7b
[<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
[<ffffffff80275d0c>] find_extend_vma+0x16/0x59
[<ffffffff8045e4d3>] _spin_lock_irqsave+0x9/0xe
[<ffffffff80311d88>] __up_read+0x13/0x8a
[<ffffffff803eba5c>] sys_sendto+0x128/0x151
[<ffffffff8045e3ed>] _spin_unlock_bh+0x9/0x15
[<ffffffff8020b7fc>] tracesys+0xdc/0xe1
BUG: soft lockup - CPU#2 stuck for 11s! [uperf.x86_64:16606]
CPU 2:
Modules linked in: ixgbe oprofile niu nfs lockd nfs_acl autofs4 hidp
rfcomm l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm
rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp ib_cm ib_ipoib ib_sa
ib_uverbs ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp
parport joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core
dm_snapshot dm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 16606, comm: uperf.x86_64 Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff803f065e>] [<ffffffff803f065e>]
pskb_expand_head+0x73/0x147
RSP: 0018:ffff8101ae14ba18 EFLAGS: 00000286
RAX: 0000000000000080 RBX: ffff8101c6476080 RCX: 000000000000059f
RDX: 0000000000000138 RSI: ffff8103f64ad841 RDI: ffff8101c64760c1
RBP: 0000000000000000 R08: ffff8101fb0722cb R09: 0000000000000002
R10: 0000000000000001 R11: 0000000000000002 R12: ffffffff8028725b
R13: ffff8101c6478000 R14: ffff8101ff191d80 R15: ffffffff805abb88
FS: 0000000040800940(0063) GS:ffff8101fb072700(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000044005f48 CR3: 00000001d0513000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff803f0630>] pskb_expand_head+0x45/0x147
[<ffffffff803f154b>] skb_realloc_headroom+0x4d/0x63
[<ffffffff882edd40>] :niu:niu_start_xmit+0x114/0x5af
[<ffffffff80221995>] gart_map_single+0x0/0x70
[<ffffffff803f5e2b>] dev_hard_start_xmit+0x1d2/0x246
[<ffffffff80406fb8>] pfifo_fast_dequeue+0x3b/0x59
[<ffffffff80406dab>] __qdisc_run+0x77/0x174
[<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
[<ffffffff80417faf>] ip_push_pending_frames+0x32c/0x3a0
[<ffffffff80419676>] ip_generic_getfrag+0x0/0x8b
[<ffffffff8043359f>] udp_push_pending_frames+0x2ba/0x337
[<ffffffff80434794>] udp_sendmsg+0x4c8/0x606
[<ffffffff803eafbb>] sock_sendmsg+0xe2/0xff
[<ffffffff8029e1a1>] iput+0x42/0x7b
[<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
[<ffffffff80275d0c>] find_extend_vma+0x16/0x59
[<ffffffff8045e4d3>] _spin_lock_irqsave+0x9/0xe
[<ffffffff80311d88>] __up_read+0x13/0x8a
[<ffffffff803eba5c>] sys_sendto+0x128/0x151
[<ffffffff8045e3ed>] _spin_unlock_bh+0x9/0x15
[<ffffffff8020b7fc>] tracesys+0xdc/0xe1
next prev parent reply other threads:[~2008-03-26 20:27 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-26 16:46 2.6.24 BUG: soft lockup - CPU#X Matheos Worku
2008-03-26 17:31 ` Rick Jones
2008-03-26 20:14 ` Jarek Poplawski
2008-03-26 20:26 ` Matheos Worku [this message]
2008-03-26 21:46 ` Jarek Poplawski
2008-03-26 21:53 ` Jarek Poplawski
2008-03-27 10:33 ` Jarek Poplawski
2008-03-27 23:18 ` Brandeburg, Jesse
2008-03-27 23:45 ` Matheos Worku
2008-03-28 0:02 ` David Miller
2008-03-28 0:19 ` Matheos Worku
2008-03-28 0:34 ` David Miller
2008-03-28 1:22 ` Herbert Xu
2008-03-28 1:38 ` David Miller
2008-03-28 10:29 ` Herbert Xu
2008-03-28 10:56 ` Ingo Molnar
2008-03-28 11:06 ` Herbert Xu
2008-03-28 11:29 ` Herbert Xu
2008-03-28 12:19 ` jamal
2008-03-28 13:26 ` Herbert Xu
2008-03-28 14:07 ` jamal
2008-03-28 14:12 ` Ingo Molnar
2008-03-28 23:25 ` David Miller
2008-03-28 14:09 ` Ingo Molnar
2008-03-28 1:58 ` Matheos Worku
2008-03-28 10:33 ` jamal
2008-03-28 17:00 ` Matheos Worku
2008-03-28 10:38 ` Herbert Xu
2008-03-28 13:38 ` Jarek Poplawski
2008-03-28 13:53 ` Herbert Xu
2008-03-28 14:39 ` Jarek Poplawski
2008-03-28 14:56 ` Herbert Xu
2008-03-28 15:29 ` Jarek Poplawski
2008-03-28 15:47 ` Jarek Poplawski
2008-03-29 1:06 ` Herbert Xu
2008-03-29 9:11 ` Jarek Poplawski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47EAB158.3080806@sun.com \
--to=matheos.worku@sun.com \
--cc=jarkao2@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.