All of lore.kernel.org
 help / color / mirror / Atom feed
From: Brian Foster <bfoster@redhat.com>
To: 符永涛 <yongtaofu@gmail.com>
Cc: "xfs@oss.sgi.com" <xfs@oss.sgi.com>
Subject: Re: question about xfs soft lockup stuck
Date: Thu, 27 Jun 2013 10:02:55 -0400	[thread overview]
Message-ID: <51CC460F.2060402@redhat.com> (raw)
In-Reply-To: <CADFMGu+eJ+skfYHZY8FVV8y2fqjEuRR4LAGPE2BMgYCoMu73wg@mail.gmail.com>

On 06/26/2013 11:20 PM, 符永涛 wrote:
> Dear Brian and xfs experts,
> 
> Recently we have encountered an issue related to xfs. The log reports xfs
> soft lockup stuck which is similar to this bug:
> http://bugs.centos.org/view.php?id=6287
> 
> I wan to ask a question. Is this issue already been fixed in kernel 358 or
> not(BZ#921961, BZ#896224)?
> 

Both of those bz's have metadata that shows the fixed-in release. The
latter is a backport that notes the fix being pulled back into
kernel-2.6.32-358.4.1.el6.

Brian

> 
> Thank you very much.
> 
> logs:
> Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#0 stuck for 67s!
> [xfslogd/0:2014]
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel: CPU 0
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel:
> Jun 26 17:57:06 cqdx kernel: Pid: 2014, comm: xfslogd/0 Not tainted
> 2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
> Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55c>]
> [<ffffffff814ec55c>] _spin_lock+0x1c/0x30
> Jun 26 17:57:06 cqdx kernel: RSP: 0000:ffff88101465bd50  EFLAGS: 00000293
> Jun 26 17:57:06 cqdx kernel: RAX: 0000000000005468 RBX: ffff88101465bd50
> RCX: ffffe8efaf600e88
> Jun 26 17:57:06 cqdx kernel: RDX: 0000000000005466 RSI: ffff882011755898
> RDI: ffff88201955b100
> Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: 0000000000000000
> R09: 00000000ffffffff
> Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000
> R12: ffff88101465bd70
> Jun 26 17:57:06 cqdx kernel: R13: ffffffff8100b9ce R14: 0000000000000400
> R15: 0000000000000094
> Jun 26 17:57:06 cqdx kernel: FS:  0000000000000000(0000)
> GS:ffff880028200000(0000) knlGS:0000000000000000
> Jun 26 17:57:06 cqdx kernel: CS:  0010 DS: 0018 ES: 0018 CR0:
> 000000008005003b
> Jun 26 17:57:06 cqdx kernel: CR2: 00007f9e001dc0a0 CR3: 000000201311e000
> CR4: 00000000000406f0
> Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Jun 26 17:57:06 cqdx kernel: Process xfslogd/0 (pid: 2014, threadinfo
> ffff88101465a000, task ffff881015b75500)
> Jun 26 17:57:06 cqdx kernel: Stack:
> Jun 26 17:57:06 cqdx kernel: ffff88101465bd80 ffffffffa02a2b3b
> 0000000000000001 ffff8820137c7300
> Jun 26 17:57:06 cqdx kernel: <d> ffff882016daee40 ffff882017e03800
> ffff88101465bda0 ffffffffa02a2adc
> Jun 26 17:57:06 cqdx kernel: <d> ffff882016d330a8 ffff882016daee40
> ffff88101465be10 ffffffffa02a2d51
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ?
> xfs_buf_iodone+0x2b/0x60 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ?
> xfs_buf_do_callbacks+0x3c/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ?
> xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ?
> xfs_buf_iodone_work+0x0/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ?
> xfs_buf_iodone_work+0x26/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ?
> worker_thread+0x170/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ?
> autoremove_wake_function+0x0/0x40
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
> Jun 26 17:57:06 cqdx kernel: Code: 81 2f 00 00 00 01 74 05 e8 72 8c d8 ff
> c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8
> 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40
> 00 55
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ?
> xfs_buf_iodone+0x2b/0x60 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ?
> xfs_buf_do_callbacks+0x3c/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ?
> xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ?
> xfs_buf_iodone_work+0x0/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ?
> xfs_buf_iodone_work+0x26/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ?
> worker_thread+0x170/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ?
> autoremove_wake_function+0x0/0x40
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
> Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#15 stuck for 67s!
> [glusterfsd:3245]
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel: CPU 15
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel:
> Jun 26 17:57:06 cqdx kernel: Pid: 3245, comm: glusterfsd Not tainted
> 2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
> Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55e>]
> [<ffffffff814ec55e>] _spin_lock+0x1e/0x30
> Jun 26 17:57:06 cqdx kernel: RSP: 0018:ffff880dceb5d528  EFLAGS: 00000297
> Jun 26 17:57:06 cqdx kernel: RAX: 00000000000087eb RBX: ffff880dceb5d528
> RCX: ffff8820137923e0
> Jun 26 17:57:06 cqdx kernel: RDX: 00000000000087ea RSI: 0000000000006e5c
> RDI: ffff882016d330a8
> Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: ffff880dceb5d674
> R09: ffff880dceb5d670
> Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000
> R12: 0000000000000000
> Jun 26 17:57:06 cqdx kernel: R13: 0000000000000cb5 R14: ffff8810788166e8
> R15: ffff882016ea9400
> Jun 26 17:57:06 cqdx kernel: FS:  00007f6cf3fff700(0000)
> GS:ffff8810788e0000(0000) knlGS:0000000000000000
> Jun 26 17:57:06 cqdx kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> Jun 26 17:57:06 cqdx kernel: CR2: 0000000000481046 CR3: 000000201311e000
> CR4: 00000000000406e0
> Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Jun 26 17:57:06 cqdx kernel: Process glusterfsd (pid: 3245, threadinfo
> ffff880dceb5c000, task ffff8810191ecaa0)
> Jun 26 17:57:06 cqdx kernel: Stack:
> Jun 26 17:57:06 cqdx kernel: ffff880dceb5d5d8 ffffffffa02c27a8
> ffff881078816680 ffff880dceb5d670
> Jun 26 17:57:06 cqdx kernel: <d> ffff880dceb5d674 ffff880dceb5d668
> 00006e5c0000000f 0000000000000001
> Jun 26 17:57:06 cqdx kernel: <d> 00000000000172a0 ffff882016d33080
> ffff880dceb5d588 0000000000000082
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ?
> xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ?
> try_to_wake_up+0x24c/0x3e0
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ?
> xlog_cil_push+0x261/0x3d0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ?
> xlog_cil_force_lsn+0x115/0x120 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ?
> xfs_bmap_add_extent+0xad/0x3c0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2308>] ?
> _xfs_log_force+0x68/0x240 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2718>] ? xfs_log_force+0x18/0x40
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d99d0>] ?
> xfs_buf_cond_lock+0x90/0xc0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dadb3>] ?
> _xfs_buf_find+0xe3/0x280 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dafbb>] ? xfs_buf_get+0x6b/0x1a0
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d16d8>] ?
> xfs_trans_get_buf+0xe8/0x180 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a4a67>] ?
> xfs_da_do_buf+0x4c7/0x770 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a5fda>] ?
> xfs_da_get_buf+0x2a/0x30 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ?
> xfs_dir2_data_init+0x4d/0xf0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ?
> xfs_dir2_data_init+0x4d/0xf0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a8c23>] ?
> xfs_dir2_sf_to_block+0xe3/0x5e0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b5ce6>] ? xfs_iunlock+0x86/0x100
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8118ecaa>] ? dput+0x9a/0x150
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b0c18>] ?
> xfs_dir2_sf_addname+0x458/0x5b0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d6687>] ?
> kmem_zone_alloc+0x77/0xf0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a81bc>] ?
> xfs_dir_createname+0x14c/0x1b0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d3390>] ? xfs_link+0x1f0/0x300
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81195560>] ?
> mntput_no_expire+0x30/0x110
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8118e7a7>] ? __d_lookup+0xa7/0x150
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8107721a>] ? capable+0x2a/0x60
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02e1120>] ? xfs_vn_link+0x40/0x70
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8118379c>] ? vfs_link+0x11c/0x190
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81182aaa>] ? lookup_hash+0x3a/0x50
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81186b71>] ? sys_linkat+0x141/0x160
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b83b>] ? vfs_stat+0x1b/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b864>] ? sys_newstat+0x24/0x50
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81186bae>] ? sys_link+0x1e/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100b072>] ?
> system_call_fastpath+0x16/0x1b
> Jun 26 17:57:06 cqdx kernel: Code: 00 00 00 01 74 05 e8 72 8c d8 ff c9 c3
> 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39
> c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55
> 48 89
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8106210b>] ?
> enqueue_task_fair+0xfb/0x100
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ?
> xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ?
> try_to_wake_up+0x24c/0x3e0
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ?
> xlog_cil_push+0x261/0x3d0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ?
> xlog_cil_force_lsn+0x115/0x120 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ?
> xfs_bmap_add_extent+0xad/0x3c0 [xfs]
> 
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

  reply	other threads:[~2013-06-27 13:58 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-06-27  3:20 question about xfs soft lockup stuck 符永涛
2013-06-27 14:02 ` Brian Foster [this message]
2013-06-27 14:29   ` Mark Tinguely

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=51CC460F.2060402@redhat.com \
    --to=bfoster@redhat.com \
    --cc=xfs@oss.sgi.com \
    --cc=yongtaofu@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.