From: Joe Landman <landman@scalableinformatics.com>
To: linux-bcache@vger.kernel.org
Subject: hang/stuck loop in bch_ptr_bad/bch_btree_iter_next_filter
Date: Tue, 31 May 2016 08:20:21 -0400 [thread overview]
Message-ID: <574D8185.5070105@scalableinformatics.com> (raw)
Hi folks:
3.8.20 kernel on a system with 256GB ram, Xeon E5-2680 v3 cpus (2x 6
cores). (Debian 7) OS booted from PXE into a ramdisk
df -h /
Filesystem Size Used Avail Use% Mounted on
tmpfs 8.0G 3.8G 4.3G 47% /
Swap set up on a file:
swapon -s
Filename Type Size Used Priority
/data/swap/swapfile file 33554428 0 0
Bcache set up atop the devices (SSD /dev/sdb and spinning disk RAID
/dev/sda)
df -h /data
Filesystem Size Used Avail Use% Mounted on
/dev/bcache0 12T 6.6T 5.3T 56% /data
(yes, swap is on top of that as well, which might be a/the problem)
This is in writeback mode.
What we are seeing is this (cpu stuck in bcache_gc, with a bad
pointer). I am wondering if the swap on the cache is a problem. Seems
to occur after significant IO loads and heavy computing tasks have been
running for a few hours. After restart (forced), the dirty data is
slowly dropping:
cat /sys/block/bcache0/bcache/dirty_data
97.9G
44 00 00 eb d8 0f 1f 00
[1374092.191983] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s!
[bcache_gc:2989]
[1374092.199907] Modules linked in: 8021q garp mrp stp llc bonding
rdma_ucm ib_ucm ib_uverbs ib_umad ib_ipoib mlx4_ib(O) mlx_compat(O)
af_packet ixgbe i40e igb cpufreq_ondemand cpufreq_powersave
cpufreq_stats cpufreq_userspace cpufreq_conservative ib_iser rdma_cm
iw_cm ib_cm ib_sa ib_mad ib_core ib_addr ipv6 nfsd dm_crypt joydev
hid_generic usbhid hid iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal
coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel
aesni_intel aes_x86_64 ablk_helper cryptd lrw gf128mul glue_helper
microcode pcspkr sb_edac edac_core snd_hda_codec_realtek
snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec
ehci_pci ehci_hcd snd_pcm i2c_i801 mei_me snd_timer lpc_ich usbcore snd
i2c_core shpchp mfd_core usb_common soundcore mei ioatdma tpm_tis tpm
ipmi_si rtc_cmos ipmi_msghandler evdev processor thermal_sys
acpi_power_meter button dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi configfs e1000e raid1 md_mod
sg ses enclosure sd_mod vxlan ip6_udp_tunnel dca udp_tunnel ptp ahci
libahci libata aacraid scsi_mod pps_core [last unloaded: cpuid]
[1374092.314940] CPU: 12 PID: 2989 Comm: bcache_gc Tainted: G W O L
3.18.20.scalable #1
[1374092.323475] Hardware name: Supermicro X10DRG-Q/X10DRG-Q, BIOS 1.0b
01/07/2015
[1374092.330876] task: ffff883fd0e34960 ti: ffff883fa3cb0000 task.ti:
ffff883fa3cb0000
[1374092.338621] RIP: 0010:[<ffffffff81634625>] [<ffffffff81634625>]
bch_extent_bad+0x135/0x1c0
[1374092.347297] RSP: 0000:ffff883fa3cb3ae8 EFLAGS: 00000206
[1374092.352859] RAX: 0000000000000007 RBX: ffffffff816344b5 RCX:
000000000000000b
[1374092.360258] RDX: ffff881fa3ff8000 RSI: 00000165a662b007 RDI:
0000000000000001
[1374092.367657] RBP: ffff883fa3cb3b08 R08: ffff881f9f040000 R09:
0000000000000000
[1374092.375056] R10: 000007ffffffffff R11: 0000000000000001 R12:
ffff881f9f040000
[1374092.382457] R13: ffff883d942c4dd8 R14: ffff883fa3cb3a58 R15:
ffff883fa3cb3cf0
[1374092.389857] FS: 0000000000000000(0000) GS:ffff883ffde00000(0000)
knlGS:0000000000000000
[1374092.398212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1374092.404207] CR2: 00007f61fba73ec0 CR3: 0000000001c14000 CR4:
00000000001407e0
[1374092.411605] Stack:
[1374092.413862] ffffffff00000001 ffff883d942c4dd8 ffff883fa3cb3b58
ffffffff8162b560
[1374092.421757] ffff883fa3cb3b18 ffffffff8162b56a ffff883fa3cb3b48
ffffffff8162b389
[1374092.429650] 00000000000009b7 ffff883bc4a679c8 ffff883fa3cb3cf0
ffff881fa93bc1c8
[1374092.437545] Call Trace:
[1374092.440236] [<ffffffff8162b560>] ? bch_ptr_invalid+0x10/0x10
[1374092.446231] [<ffffffff8162b56a>] bch_ptr_bad+0xa/0x10
[1374092.451616] [<ffffffff8162b389>] bch_btree_iter_next_filter+0x39/0x50
[1374092.458393] [<ffffffff8162b7d1>] btree_gc_count_keys+0x51/0x70
[1374092.464561] [<ffffffff816314af>] btree_gc_recurse+0x1bf/0x330
[1374092.470640] [<ffffffff8162cc23>] ? btree_gc_mark_node+0x63/0x240
[1374092.476985] [<ffffffff8109a071>] ? down_write_nested+0x91/0xb0
[1374092.483152] [<ffffffff81631752>] ? bch_btree_gc+0x132/0x5d0
[1374092.489060] [<ffffffff81631abd>] bch_btree_gc+0x49d/0x5d0
[1374092.494792] [<ffffffff81093c80>] ? __init_waitqueue_head+0x60/0x60
[1374092.501309] [<ffffffff81631c28>] bch_gc_thread+0x38/0x140
[1374092.507043] [<ffffffff81631bf0>] ? bch_btree_gc+0x5d0/0x5d0
[1374092.512950] [<ffffffff81073244>] kthread+0xe4/0x100
[1374092.518163] [<ffffffff81073160>] ? __init_kthread_worker+0x70/0x70
[1374092.524680] [<ffffffff8178f898>] ret_from_fork+0x58/0x90
[1374092.530327] [<ffffffff81073160>] ? __init_kthread_worker+0x70/0x70
[1374092.536839] Code: 0f 00 00 49 8b 94 c0 d0 0c 00 00 48 89 f0 48 c1
e8 08 4c 21 d0 48 d3 e8 4c 8b a2 08 0b 00 00 48 8d 04 40 49 8d 04 84 0f
b6 40 06 <29> f0 3c 80 77 85 0f b6 d0 83 fa 60 0f 86 71 ff ff ff 41 0f b6
Thanks in advance for any guidance/advice
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
e: landman@scalableinformatics.com
w: http://scalableinformatics.com
t: @scalableinfo
p: +1 734 786 8423 x121
c: +1 734 612 4615
next reply other threads:[~2016-05-31 12:20 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-05-31 12:20 Joe Landman [this message]
2016-05-31 12:28 ` hang/stuck loop in bch_ptr_bad/bch_btree_iter_next_filter Vojtech Pavlik
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=574D8185.5070105@scalableinformatics.com \
--to=landman@scalableinformatics.com \
--cc=linux-bcache@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.