From: Johannes Rohr <jorohr@gmail.com>
To: linux-btrfs@vger.kernel.org
Subject: system hangs when running btrfs balance
Date: Mon, 10 Aug 2020 11:22:53 +0200 [thread overview]
Message-ID: <c684e9ac-f28b-7056-0a46-6c6450ac4c1f@gmail.com> (raw)
Dear devs,
since I upgraded our system from Ubuntu 18.04 LTS to 20.04 LTS, the file
system completely freezes when I run a btrfs balance on it. The only way
to get a usable system for the time being is with the mount option
"skip_balance".
The server has a raid1 with 4 SSDs with 500 GB each.
Here is a backtrace of what I am seeing:
[Sun Aug 9 12:21:35 2020] ------------[ cut here ]------------
[Sun Aug 9 12:21:35 2020] kernel BUG at fs/btrfs/
relocation.c:2626!
[Sun Aug 9 12:21:35 2020] invalid opcode: 0000 [#1] SMP PTI
[Sun Aug 9 12:21:35 2020] CPU: 1 PID: 4537 Comm: btrfs-balance Tainted: G O 5.4.47 #1
[Sun Aug 9 12:21:35 2020] Hardware name: FUJITSU D3401-H1/D3401-H1,
BIOS V5.0.0.11 R1.14.0 for D3401-H1x 06/09/2016
[Sun Aug 9 12:21:35 2020] RIP: 0010:select_reloc_root+0x5b/0x19f [btrfs]
[Sun Aug 9 12:21:35 2020] Code: c0 c7 44 24 04 00 00 00 00 e8 8b 9d 17
e1 48 89 df 4c 89 f6 48 8d 54 24 04 e8 9c e6 ff ff 48 8b 58 60 48 89 c5
48 85 db 75 02 <0f> 0b 48 8b 43 20 a8 02 75 02 0f 0b 48 83 bb df
01 00 00 f8 75 45
[Sun Aug 9 12:21:35 2020] RSP: 0018:ffff8887e0b0bb20 EFLAGS: 00010246
[Sun Aug 9 12:21:35 2020] RAX: ffff8887dfab5280 RBX: 0000000000000000 RCX: 0000000000000000
[Sun Aug 9 12:21:35 2020] RDX: ffff8887e0b0bb24 RSI: ffff8887e0b0bc10 RDI: ffff8887dfab52c0
[Sun Aug 9 12:21:35 2020] RBP: ffff8887dfab5280 R08: ffff8887dfab52c0 R09: ffffffffa0491e7e
[Sun Aug 9 12:21:35 2020] R10: ffff8887f4ba7e70 R11: ffff8888090ed158 R12: ffff8887dfab5280
[Sun Aug 9 12:21:35 2020] R13: ffff8887fd330800 R14: ffff8887e0b0bc10 R15: ffff8887e7fa66e8
[Sun Aug 9 12:21:35 2020] FS: 0000000000000000(0000) GS:ffff88880e240000(0000) knlGS:0000000000000000
[Sun Aug 9 12:21:35 2020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sun Aug 9 12:21:35 2020] CR2: 000055b4d5b7cfe0 CR3: 000000000200a004 CR4: 00000000003606e0
[Sun Aug 9 12:21:35 2020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[Sun Aug 9 12:21:35 2020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[Sun Aug 9 12:21:35 2020] Call Trace:
[Sun Aug 9 12:21:35 2020] do_relocation+0xb6/0x4c2 [btrfs]
[Sun Aug 9 12:21:35 2020] ? calcu_metadata_size.isra.36.constprop.42+0x9e/0xc4 [btrfs]
[Sun Aug 9 12:21:35 2020] ? do_raw_spin_lock+0x2f/0x5a
[Sun Aug 9 12:21:35 2020] ? btrfs_block_rsv_refill+0x4b/0x8b [btrfs]
[Sun Aug 9 12:21:35 2020] relocate_tree_blocks+0x301/0x427 [btrfs]
[Sun Aug 9 12:21:35 2020] ? tree_insert+0x49/0x4e [btrfs]
[Sun Aug 9 12:21:35 2020] ? add_tree_block.isra.38+0x11e/0x144 [btrfs]
[Sun Aug 9 12:21:35 2020] relocate_block_group+0x279/0x49e [btrfs]
[Sun Aug 9 12:21:35 2020] btrfs_relocate_block_group+0x15e/0x23d [btrfs]
[Sun Aug 9 12:21:35 2020] btrfs_relocate_chunk+0x25/0x8c [btrfs]
[Sun Aug 9 12:21:35 2020] btrfs_balance+0xaf0/0xd45 [btrfs]
[Sun Aug 9 12:21:35 2020] ? btrfs_balance+0xd45/0xd45 [btrfs]
[Sun Aug 9 12:21:35 2020] balance_kthread+0x32/0x46 [btrfs]
[Sun Aug 9 12:21:35 2020] kthread+0xf5/0xfa
[Sun Aug 9 12:21:35 2020] ? kthread_associate_blkcg+0x86/0x86
[Sun Aug 9 12:21:35 2020] ret_from_fork+0x3a/0x50
[Sun Aug 9 12:21:35 2020] Modules linked in: btrfs xor zstd_decompress
zstd_compress lzo_compress lzo_decompress zlib_deflate raid6_pq
libcrc32c sd_mod ipmi_devintf ipmi_msghandler sg x86_pkg_temp_thermal
intel_powerclamp kvm_intel kvm irqbypass crc32_pclmul crc32c_intel
iTCO_wdt ghash_clmulni_intel aesni_intel crypto_simd psmouse ahci cryptd
libahci i2c_i801 serio_raw glue_helper intel_pch_thermal evdev video
thermal acpi_pad button fan jc42 ftsteutates nct6775 hwmon_vid coretemp
ip_tables x_tables autofs4 e1000e
[Sun Aug 9 12:21:36 2020] ---[ end trace 442b443de6cecc6e ]---
[Sun Aug 9 12:21:36 2020] RIP: 0010:select_reloc_root+0x5b/0x19f [btrfs]
[Sun Aug 9 12:21:36 2020] Code: c0 c7 44 24 04 00 00 00 00 e8 8b 9d 17
e1 48 89 df 4c 89 f6 48 8d 54 24 04 e8 9c e6 ff ff 48 8b 58 60 48 89 c5
48 85 db 75 02 <0f> 0b 48 8b 43 20 a8 02 75 02 0f 0b 48 83 bb df
01 00 00 f8 75 45
There has been a related bug report at kernel.org for a year,
https://bugzilla.kernel.org/show_bug.cgi?id=203405 and I have found
similar reports here and there, some pertaining to quite old kernel
versions, but we have only been hit with kernel 5.4. After this first
occurred, I had no better luck though, with older kernels (4 something
from Debian buster, also 4 something from Ubuntu 18.04).
Apart from fixing the underlying issue, would there be any wordaround
for it? Currently the balance for the fs is in suspended status. Since
there is quite a few people who depend on this server, I can't just play
around with it at random. That's why I am asking for advice here...
Thanks so much for any suggestions you might have!
Johannes
next reply other threads:[~2020-08-10 9:22 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-10 9:22 Johannes Rohr [this message]
2020-08-10 9:59 ` system hangs when running btrfs balance Qu Wenruo
2020-08-10 20:23 ` Johannes Rohr
2020-08-11 16:43 ` Johannes Rohr
2020-08-11 21:30 ` Lukas Tribus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c684e9ac-f28b-7056-0a46-6c6450ac4c1f@gmail.com \
--to=jorohr@gmail.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox