From: Jens Axboe <axboe@kernel.dk>
To: Saritha Vinod <sarithavn@gmail.com>, fio@vger.kernel.org
Subject: Re: Kernel message "BUG:soft lockup" during fio runs
Date: Tue, 26 Nov 2013 11:54:55 -0700 [thread overview]
Message-ID: <5294EE7F.2080609@kernel.dk> (raw)
In-Reply-To: <CA+C06oQ5fOZ5ND-3Tzcyd-DnbG_XQGCoppW74NHWMbF_wKV0Tg@mail.gmail.com>
On 11/26/2013 11:14 AM, Saritha Vinod wrote:
> While running fio on RHEL 6.4, ppc_64, got the below error:
> BUG: soft lockup - CPU#0 stuck for 68s! [fio:49580]
>
> The system does not respond during this interval. Observed this
> occurring multiple times.
> Has anyone faced this before? Could anyone please help me with this?
>
> The dmesg output is pasted below.
> BUG: soft lockup - CPU#0 stuck for 68s! [fio:49580]
> Modules linked in: autofs4 sunrpc ipt_REJECT nf_conntrack_ipv4
> nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
> nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6
> dm_round_robin dm_multipath shpchp ses enclosure sg be2net ext4 jbd2
> mbcache sd_mod crc_t10dif ipr lpfc scsi_transport_fc scsi_tgt
> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> NIP: c0000000002cefa8 LR: c0000000002cf548 CTR: d00000000d3a1120
> REGS: c000001f1ffcb790 TRAP: 0901 Not tainted (2.6.32-358.el6.ppc64)
> MSR: 8000000000009032 <EE,ME,IR,DR> CR: 24002448 XER: 00000000
> TASK = c000001eb1a459b0[49580] 'fio' THREAD: c000001eb1d4c000 CPU: 0
> GPR00: c0000000002cf548 c000001f1ffcba10 c000000000f37e78 c000000f4cc64c60
> GPR04: 0000000000000000 0000000000001000 0000000000000000 0000000001327f78
> GPR08: c000001f02bd13c8 c000001effab9b00 c000001f1ffcbe90 ffffffffffffffff
> GPR12: 0000000024002442 c000000001002500
> NIP [c0000000002cefa8] .blk_update_request+0x38/0x5b0
> LR [c0000000002cf548] .blk_update_bidi_request+0x28/0xd0
> Call Trace:
> [c000001f1ffcba10] [0000000024002448] 0x24002448 (unreliable)
> [c000001f1ffcbae0] [c0000000002cf548] .blk_update_bidi_request+0x28/0xd0
> [c000001f1ffcbb70] [c0000000002d0a4c] .blk_end_bidi_request+0x2c/0x90
> [c000001f1ffcbc10] [c0000000003e2e48] .scsi_io_completion+0xc8/0x680
> [c000001f1ffcbce0] [c0000000003d7e68] .scsi_finish_command+0x128/0x190
> [c000001f1ffcbd80] [c0000000003e35e8] .scsi_softirq_done+0x1d8/0x210
> [c000001f1ffcbe20] [c0000000002d9b80] .blk_done_softirq+0xb0/0xe0
> [c000001f1ffcbeb0] [c00000000009c428] .__do_softirq+0x118/0x290
> [c000001f1ffcbf90] [c000000000032da8] .call_do_softirq+0x14/0x24
> [c000001eb1d4f810] [c00000000000e700] .do_softirq+0xf0/0x110
> [c000001eb1d4f8b0] [c00000000009c144] .irq_exit+0xb4/0xc0
> [c000001eb1d4f930] [c00000000000e964] .do_IRQ+0x144/0x230
> [c000001eb1d4f9e0] [c000000000004898] hardware_interrupt_entry+0x18/0x80
> --- Exception: 501 at .do_munmap+0x344/0x3d0
> LR = .do_munmap+0x318/0x3d0
> [c000001eb1d4fd90] [c000000000187d54] .SyS_munmap+0x54/0x90
> [c000001eb1d4fe30] [c000000000008564] syscall_exit+0x0/0x40
> Instruction dump:
> fba1ffe8 fbc1fff0 fbe1fff8 fae1ffb8 fb01ffc0 f8010010 fb21ffc8 fb41ffd0
> fb61ffd8 f821ff31 ebc2c0c8 7c7d1b78 <7c9c2378> 7cbf2b78 e8030060 38600000
> INFO: task fio:49522 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 00000080d6b76340 0 49522 49230 0x00008080
> Call Trace:
> [c000001eb30730b0] [c000001eb3073160] 0xc000001eb3073160 (unreliable)
> [c000001eb3073280] [c0000000000142d8] .__switch_to+0xf8/0x1d0
> [c000001eb3073310] [c0000000005ba5c8] .schedule+0x3f8/0xd30
> [c000001eb3073610] [c0000000005baf90] .io_schedule+0x90/0x110
> [c000001eb30736a0] [c000000000209100] .__blockdev_direct_IO_newtrunc+0xaa0/0xc70
> [c000001eb30737d0] [c00000000020932c] .__blockdev_direct_IO+0x5c/0x110
> [c000001eb30738a0] [c000000000206098] .blkdev_direct_IO+0x48/0x60
> [c000001eb3073940] [c00000000015002c] .generic_file_aio_read+0x72c/0x780
> [c000001eb3073a90] [c00000000020536c] .blkdev_aio_read+0x5c/0xf0
> [c000001eb3073b40] [c0000000001c2594] .do_sync_read+0xd4/0x160
> [c000001eb3073ce0] [c0000000001c36cc] .vfs_read+0xec/0x1f0
> [c000001eb3073d80] [c0000000001c38f8] .SyS_read+0x58/0xb0
> [c000001eb3073e30] [c000000000008564] syscall_exit+0x0/0x40
> INFO: task fio:49538 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 00000080d6b76340 0 49538 49230 0x00008080
You'll want to report this to Red Hat, it's not a fio issue. It might
still be my issue, however, as it could be a bug in the block stack in
Linux... But please report it to RH first, it might be something they
are already aware of.
--
Jens Axboe
prev parent reply other threads:[~2013-11-26 18:55 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-11-26 18:14 Kernel message "BUG:soft lockup" during fio runs Saritha Vinod
2013-11-26 18:54 ` Jens Axboe [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5294EE7F.2080609@kernel.dk \
--to=axboe@kernel.dk \
--cc=fio@vger.kernel.org \
--cc=sarithavn@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.