From: Shaohua Li <shli@kernel.org>
To: Coly Li <colyli@suse.de>
Cc: linux-raid@vger.kernel.org
Subject: Re: a hard lockup in md raid5 sequential write (v4.7-rc7)
Date: Tue, 19 Jul 2016 16:35:56 -0700 [thread overview]
Message-ID: <20160719233556.GC79792@kernel.org> (raw)
In-Reply-To: <cd2e1d88-683c-81f7-c1e7-674d04ccdda1@suse.de>
On Mon, Jul 18, 2016 at 04:55:04PM +0800, Coly Li wrote:
> Hi,
>
> These days I observe a hard lockup in md raid5. This issue can be easily
> reproduced in kernel v4.7-rc7 (up to commit:
> 47ef4ad2684d380dd6d596140fb79395115c3950) by this fio job file:
>
> [global]
> direct=1
> thread=1
> [job]
> filename=/dev/md0
> blocksize=8m
> rw=write
> name=raid5
> lockmem=1
> numjobs=40
> write_bw_log=example
> group_reporting=1
> norandommap=1
> log_avg_msec=0
> runtime=600.0
> iodepth=64
> write_lat_log=example
>
> Where md0 is a raid5 target assembled by 3 Memblaze (PBlaze3) PCIe SSDs.
> This test runs on a dual 10-core processors Dell T7910 machine.
>
> From the crash dump, dmesg of the panic by nmi watchdog timeout is,
>
> [ 2330.544036] NMI watchdog: Watchdog detected hard LOCKUP on cpu
> 18.dModules linked in: raid456 async_raid6_recov async_memcpy libcrc32c
> async_pq async_xor async_tx joydev st memdisk(O) memcon(O) af_packet
> iscsi_ibft iscsi_boot_sysfs msr snd_hda_codec_hdmi intel_rapl sb_edac
> raid1 edac_core x86_pkg_temp_thermal intel_powerclamp coretemp raid0
> md_mod snd_hda_codec_realtek snd_hda_codec_generic kvm_intel kvm
> snd_hda_intel irqbypass snd_hda_codec crct10dif_pclmul snd_hda_core
> crc32_pclmul ghash_clmulni_intel snd_hwdep dm_mod aesni_intel aes_x86_64
> snd_pcm mei_wdt e1000e igb iTCO_wdt lrw dcdbas iTCO_vendor_support
> snd_timer gf128mul mei_me dell_smm_hwmon glue_helper serio_raw
> ablk_helper cryptd snd lpc_ich pcspkr ptp i2c_i801 mei mptctl dca
> mfd_core pps_core soundcore mptbase shpchp fjes tpm_tis tpm btrfs xor
> raid6_pq hid_generic usbhid crc32c_intel nouveau video mxm_wmi
> i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt xhci_pci
> fb_sys_fops ehci_pci xhci_hcd ehci_hcd sr_mod ttm cd
> [ 2330.544036] CPU: 18 PID: 30308 Comm: kworker/u42:4 Tainted: G
> O 4.7.0-rc7-vanilla #1
> [ 2330.544036] Hardware name: Dell Inc. Precision Tower 7910/0215PR,
> BIOS A07 04/14/2015
> [ 2330.544036] Workqueue: raid5wq raid5_do_work [raid456]
> [ 2330.544036] 0000000000000000 ffff88103f405bb0 ffffffff813a6eea
> 0000000000000000
> [ 2330.544036] 0000000000000000 ffff88103f405bc8 ffffffff8113c3e8
> ffff8808dc7d8800
> [ 2330.544036] ffff88103f405c00 ffffffff81180f8c 0000000000000001
> ffff88103f40a440
> [ 2330.544036] Call Trace:
> [ 2330.544036] <NMI> [<ffffffff813a6eea>] dump_stack+0x63/0x89
> [ 2330.544036] [<ffffffff8113c3e8>] watchdog_overflow_callback+0xc8/0xf0
> [ 2330.544036] [<ffffffff81180f8c>] __perf_event_overflow+0x7c/0x1b0
> [ 2330.544036] [<ffffffff8118b644>] perf_event_overflow+0x14/0x20
> [ 2330.544036] [<ffffffff8100bf57>] intel_pmu_handle_irq+0x1c7/0x460
> [ 2330.544036] [<ffffffff810053ad>] perf_event_nmi_handler+0x2d/0x50
> [ 2330.544036] [<ffffffff810312e1>] nmi_handle+0x61/0x140
> [ 2330.544036] [<ffffffff81031888>] default_do_nmi+0x48/0x130
> [ 2330.544036] [<ffffffff81031a5b>] do_nmi+0xeb/0x160
> [ 2330.544036] [<ffffffff816e5c71>] end_repeat_nmi+0x1a/0x1e
> [ 2330.544036] [<ffffffff810cbcc7>] ?
> native_queued_spin_lock_slowpath+0x117/0x1a0
> [ 2330.544036] [<ffffffff810cbcc7>] ?
> native_queued_spin_lock_slowpath+0x117/0x1a0
> [ 2330.544036] [<ffffffff810cbcc7>] ?
> native_queued_spin_lock_slowpath+0x117/0x1a0
> [ 2330.544036] <<EOE>> [<ffffffff81193bbf>]
> queued_spin_lock_slowpath+0xb/0xf
> [ 2330.544036] [<ffffffff816e31ff>] _raw_spin_lock_irq+0x2f/0x40
> [ 2330.544036] [<ffffffffa084c5d8>]
> handle_active_stripes.isra.51+0x378/0x4f0 [raid456]
> [ 2330.544036] [<ffffffffa083f1a6>] ?
> raid5_wakeup_stripe_thread+0x96/0x1b0 [raid456]
> [ 2330.544036] [<ffffffffa084cf1d>] raid5_do_work+0x8d/0x120 [raid456]
> [ 2330.544036] [<ffffffff8109b5bb>] process_one_work+0x14b/0x450
> [ 2330.544036] [<ffffffff8109b9eb>] worker_thread+0x12b/0x490
> [ 2330.544036] [<ffffffff8109b8c0>] ? process_one_work+0x450/0x450
> [ 2330.544036] [<ffffffff810a1599>] kthread+0xc9/0xe0
> [ 2330.544036] [<ffffffff816e3a9f>] ret_from_fork+0x1f/0x40
> [ 2330.544036] [<ffffffff810a14d0>] ? kthread_create_on_node+0x180/0x180
> [ 2330.544036] Kernel panic - not syncing: Hard LOCKUP
> [ 2330.544036] CPU: 18 PID: 30308 Comm: kworker/u42:4 Tainted: G
> O 4.7.0-rc7-vanilla #1
> [ 2330.544036] Hardware name: Dell Inc. Precision Tower 7910/0215PR,
> BIOS A07 04/14/2015
> [ 2330.544036] Workqueue: raid5wq raid5_do_work [raid456]
> [ 2330.544036] 0000000000000000 ffff88103f405b28 ffffffff813a6eea
> ffffffff81a45241
> [ 2330.544036] 0000000000000000 ffff88103f405ba0 ffffffff81193642
> 0000000000000010
> [ 2330.544036] ffff88103f405bb0 ffff88103f405b50 0000000000000086
> ffffffff81a2a2e2
> [ 2330.544036] Call Trace:
> [ 2330.544036] <NMI> [<ffffffff813a6eea>] dump_stack+0x63/0x89
> [ 2330.544036] [<ffffffff81193642>] panic+0xd2/0x223
> [ 2330.544036] [<ffffffff810823af>] nmi_panic+0x3f/0x40
> [ 2330.544036] [<ffffffff8113c401>] watchdog_overflow_callback+0xe1/0xf0
> [ 2330.544036] [<ffffffff81180f8c>] __perf_event_overflow+0x7c/0x1b0
> [ 2330.544036] [<ffffffff8118b644>] perf_event_overflow+0x14/0x20
> [ 2330.544036] [<ffffffff8100bf57>] intel_pmu_handle_irq+0x1c7/0x460
> [ 2330.544036] [<ffffffff810053ad>] perf_event_nmi_handler+0x2d/0x50
> [ 2330.544036] [<ffffffff810312e1>] nmi_handle+0x61/0x140
> [ 2330.544036] [<ffffffff81031888>] default_do_nmi+0x48/0x130
> [ 2330.544036] [<ffffffff81031a5b>] do_nmi+0xeb/0x160
> [ 2330.544036] [<ffffffff816e5c71>] end_repeat_nmi+0x1a/0x1e
> [ 2330.544036] [<ffffffff810cbcc7>] ?
> native_queued_spin_lock_slowpath+0x117/0x1a0
> [ 2330.544036] [<ffffffff810cbcc7>] ?
> native_queued_spin_lock_slowpath+0x117/0x1a0
> [ 2330.544036] [<ffffffff810cbcc7>] ?
> native_queued_spin_lock_slowpath+0x117/0x1a0
> [ 2330.544036] <<EOE>> [<ffffffff81193bbf>]
> queued_spin_lock_slowpath+0xb/0xf
> [ 2330.544036] [<ffffffff816e31ff>] _raw_spin_lock_irq+0x2f/0x40
> [ 2330.544036] [<ffffffffa084c5d8>]
> handle_active_stripes.isra.51+0x378/0x4f0 [raid456]
> [ 2330.544036] [<ffffffffa083f1a6>] ?
> raid5_wakeup_stripe_thread+0x96/0x1b0 [raid456]
> [ 2330.544036] [<ffffffffa084cf1d>] raid5_do_work+0x8d/0x120 [raid456]
> [ 2330.544036] [<ffffffff8109b5bb>] process_one_work+0x14b/0x450
> [ 2330.544036] [<ffffffff8109b9eb>] worker_thread+0x12b/0x490
> [ 2330.544036] [<ffffffff8109b8c0>] ? process_one_work+0x450/0x450
> [ 2330.544036] [<ffffffff810a1599>] kthread+0xc9/0xe0
> [ 2330.544036] [<ffffffff816e3a9f>] ret_from_fork+0x1f/0x40
> [ 2330.544036] [<ffffffff810a14d0>] ? kthread_create_on_node+0x180/0x180
>
> The crash dump file is quite big (124MB), I need to find a method to
> share, if anyone of you wants it, please let me know.
>
> IMHO, this hard lockup seems related to bitmap allocation, because it
> can be easily reproduced on a new-created md raid5 target, with 40+
> processes doing big size (8MB+) writing.
Hi,
Sounds like a deadlock. Can you enable lockdep and run the test again and see
if lockdep gives any hint?
Thanks,
Shaohua
next prev parent reply other threads:[~2016-07-19 23:35 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-18 8:55 a hard lockup in md raid5 sequential write (v4.7-rc7) Coly Li
2016-07-19 23:35 ` Shaohua Li [this message]
2016-09-06 16:46 ` Coly Li
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160719233556.GC79792@kernel.org \
--to=shli@kernel.org \
--cc=colyli@suse.de \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.