btrfs_remove_chunk call trace?

btrfs_remove_chunk call trace?

[Rich Rauenzahn]

Is this something to be concerned about?

I'm running the latest mainline kernel on CentOS 7.

[ 1338.882288] ------------[ cut here ]------------
[ 1338.883058] WARNING: CPU: 2 PID: 790 at fs/btrfs/ctree.h:1559
btrfs_update_device+0x1c5/0x1d0 [btrfs]
[ 1338.883809] Modules linked in: xt_nat veth ipt_MASQUERADE
nf_nat_masquerade_ipv4 xt_addrtype overlay loop nf_conntrack_ftp
nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_comment xt_recent
xt_multiport xt_conntrack iptable_filter xt_REDIRECT nf_nat_redirect
iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nct6775
nf_nat nf_conntrack hwmon_vid jc42 vfat fat dm_mirror dm_region_hash
dm_log dm_mod dax xfs libcrc32c x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_hdmi
snd_hda_codec_generic irqbypass crct10dif_pclmul crc32_pclmul
snd_hda_intel ghash_clmulni_intel pcbc snd_hda_codec aesni_intel
snd_hda_core iTCO_wdt snd_hwdep crypto_simd glue_helper cryptd
iTCO_vendor_support snd_seq mei_wdt snd_seq_device intel_cstate
cdc_acm snd_pcm intel_rapl_perf
[ 1338.888639]  input_leds snd_timer lpc_ich i2c_i801 pcspkr
intel_pch_thermal snd mfd_core sg mei_me soundcore mei acpi_pad shpchp
ie31200_edac nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
btrfs xor raid6_pq sd_mod crc32c_intel ahci e1000e libahci
firewire_ohci igb i915 dca firewire_core ptp i2c_algo_bit crc_itu_t
libata pps_core drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops drm video
[ 1338.891412] CPU: 2 PID: 790 Comm: btrfs-cleaner Tainted: G        W
      4.13.1-1.el7.elrepo.x86_64 #1
[ 1338.892171] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.0a 05/09/2014
[ 1338.892884] task: ffff880407cec5c0 task.stack: ffffc90002624000
[ 1338.893613] RIP: 0010:btrfs_update_device+0x1c5/0x1d0 [btrfs]
[ 1338.894299] RSP: 0018:ffffc90002627d00 EFLAGS: 00010206
[ 1338.894956] RAX: 0000000000000fff RBX: ffff880407cd9930 RCX: 000001d1c1011e00
[ 1338.895647] RDX: ffff880000000000 RSI: ffff880336e80f9e RDI: ffff88028395bd88
[ 1338.896304] RBP: ffffc90002627d48 R08: 0000000000003fc2 R09: ffffc90002627cb8
[ 1338.896934] R10: 0000000000001000 R11: 0000000000000003 R12: ffff880405f68c00
[ 1338.897618] R13: 0000000000000000 R14: ffff88028395bd88 R15: 0000000000003f9e
[ 1338.898251] FS:  0000000000000000(0000) GS:ffff88041fa80000(0000)
knlGS:0000000000000000
[ 1338.898867] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1338.899522] CR2: 00007ff82f2cb000 CR3: 0000000001c09000 CR4: 00000000001406e0
[ 1338.900157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1338.900772] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1338.901402] Call Trace:
[ 1338.902017]  btrfs_remove_chunk+0x2fb/0x8b0 [btrfs]
[ 1338.902673]  btrfs_delete_unused_bgs+0x363/0x440 [btrfs]
[ 1338.903304]  cleaner_kthread+0x150/0x180 [btrfs]
[ 1338.903908]  kthread+0x109/0x140
[ 1338.904593]  ? btree_invalidatepage+0xa0/0xa0 [btrfs]
[ 1338.905207]  ? kthread_park+0x60/0x60
[ 1338.905803]  ret_from_fork+0x25/0x30
[ 1338.906416] Code: 10 00 00 00 4c 89 fe e8 8a 30 ff ff 4c 89 f7 e8
32 f6 fc ff e9 d3 fe ff ff b8 f4 ff ff ff e9 d4 fe ff ff 0f 1f 00 e8
cb 3e be e0 <0f> ff eb af 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 31 d2
be 02

Re: btrfs_remove_chunk call trace?

[Rich Rauenzahn]

...and can it be related to the Samsung 840 SSD's not supporting NCQ
Trim?  (Although I can't tell which device this trace is from -- it
could be a mechanical Western Digital.)

On Sun, Sep 10, 2017 at 10:16 PM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
> Is this something to be concerned about?
>
> I'm running the latest mainline kernel on CentOS 7.
>
> [ 1338.882288] ------------[ cut here ]------------
> [ 1338.883058] WARNING: CPU: 2 PID: 790 at fs/btrfs/ctree.h:1559
> btrfs_update_device+0x1c5/0x1d0 [btrfs]
> [ 1338.883809] Modules linked in: xt_nat veth ipt_MASQUERADE
> nf_nat_masquerade_ipv4 xt_addrtype overlay loop nf_conntrack_ftp
> nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_comment xt_recent
> xt_multiport xt_conntrack iptable_filter xt_REDIRECT nf_nat_redirect
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nct6775
> nf_nat nf_conntrack hwmon_vid jc42 vfat fat dm_mirror dm_region_hash
> dm_log dm_mod dax xfs libcrc32c x86_pkg_temp_thermal intel_powerclamp
> coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda_codec_hdmi
> snd_hda_codec_generic irqbypass crct10dif_pclmul crc32_pclmul
> snd_hda_intel ghash_clmulni_intel pcbc snd_hda_codec aesni_intel
> snd_hda_core iTCO_wdt snd_hwdep crypto_simd glue_helper cryptd
> iTCO_vendor_support snd_seq mei_wdt snd_seq_device intel_cstate
> cdc_acm snd_pcm intel_rapl_perf
> [ 1338.888639]  input_leds snd_timer lpc_ich i2c_i801 pcspkr
> intel_pch_thermal snd mfd_core sg mei_me soundcore mei acpi_pad shpchp
> ie31200_edac nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
> btrfs xor raid6_pq sd_mod crc32c_intel ahci e1000e libahci
> firewire_ohci igb i915 dca firewire_core ptp i2c_algo_bit crc_itu_t
> libata pps_core drm_kms_helper syscopyarea sysfillrect sysimgblt
> fb_sys_fops drm video
> [ 1338.891412] CPU: 2 PID: 790 Comm: btrfs-cleaner Tainted: G        W
>       4.13.1-1.el7.elrepo.x86_64 #1
> [ 1338.892171] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.0a 05/09/2014
> [ 1338.892884] task: ffff880407cec5c0 task.stack: ffffc90002624000
> [ 1338.893613] RIP: 0010:btrfs_update_device+0x1c5/0x1d0 [btrfs]
> [ 1338.894299] RSP: 0018:ffffc90002627d00 EFLAGS: 00010206
> [ 1338.894956] RAX: 0000000000000fff RBX: ffff880407cd9930 RCX: 000001d1c1011e00
> [ 1338.895647] RDX: ffff880000000000 RSI: ffff880336e80f9e RDI: ffff88028395bd88
> [ 1338.896304] RBP: ffffc90002627d48 R08: 0000000000003fc2 R09: ffffc90002627cb8
> [ 1338.896934] R10: 0000000000001000 R11: 0000000000000003 R12: ffff880405f68c00
> [ 1338.897618] R13: 0000000000000000 R14: ffff88028395bd88 R15: 0000000000003f9e
> [ 1338.898251] FS:  0000000000000000(0000) GS:ffff88041fa80000(0000)
> knlGS:0000000000000000
> [ 1338.898867] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 1338.899522] CR2: 00007ff82f2cb000 CR3: 0000000001c09000 CR4: 00000000001406e0
> [ 1338.900157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 1338.900772] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 1338.901402] Call Trace:
> [ 1338.902017]  btrfs_remove_chunk+0x2fb/0x8b0 [btrfs]
> [ 1338.902673]  btrfs_delete_unused_bgs+0x363/0x440 [btrfs]
> [ 1338.903304]  cleaner_kthread+0x150/0x180 [btrfs]
> [ 1338.903908]  kthread+0x109/0x140
> [ 1338.904593]  ? btree_invalidatepage+0xa0/0xa0 [btrfs]
> [ 1338.905207]  ? kthread_park+0x60/0x60
> [ 1338.905803]  ret_from_fork+0x25/0x30
> [ 1338.906416] Code: 10 00 00 00 4c 89 fe e8 8a 30 ff ff 4c 89 f7 e8
> 32 f6 fc ff e9 d3 fe ff ff b8 f4 ff ff ff e9 d4 fe ff ff 0f 1f 00 e8
> cb 3e be e0 <0f> ff eb af 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 31 d2
> be 02

Re: btrfs_remove_chunk call trace?

[Duncan]

Rich Rauenzahn posted on Sun, 10 Sep 2017 22:45:50 -0700 as excerpted:

> ...and can it be related to the Samsung 840 SSD's not supporting NCQ
> Trim?  (Although I can't tell which device this trace is from -- it
> could be a mechanical Western Digital.)
> 
> On Sun, Sep 10, 2017 at 10:16 PM, Rich Rauenzahn <rrauenza@gmail.com>
> wrote:
>> Is this something to be concerned about?
>>
>> I'm running the latest mainline kernel on CentOS 7.
>>
>> [ 1338.891412] CPU: 2 PID: 790 Comm: btrfs-cleaner
>> Tainted: G        W     4.13.1-1.el7.elrepo.x86_64 #1

As a Samsung ssd (tho 850) owner myself, who looked into mounting with 
the discard mount option on my new ssds here...

Samsung ssds lack of queued-trim shouldn't be a problem on anything close 
to a current kernel, obviously including the 4.13.1 you're running above, 
because the kernel block layer has blacklisted queued-trim on all Samsung 
SSDs for at least several kernel cycles (not sure when it went in, but 
there was a time in the 3.x kernel era when queued-trim on samsung ssds 
was causing problems, thus the blacklisting).

So the only way that could be a problem would be if that blacklisting is 
somehow being bypassed, and that indeed would be a *BIG* problem, well 
beyond btrfs.


That said, enabling the discard mount option isn't recommended in 
general, because unless queued-trim /is/ supported it lowers 
performance.  Additionally, there have been btrfs data corruption level 
bugs related to trim handling in the past, altho AFAIK all such known 
bugs are now fixed.  The option is there for those who know their device 
supports queued-trim and want to use it, but it isn't recommended 
otherwise, and because of that it's less well tested than the more 
mainstream options, which means there's at least some additional risk in 
enabling it, even if there's no known issues on current kernels other 
than the performance issue, if your device doesn't support queued-trim.

The recommended alternative is periodic/scheduled use of the fstrim 
command.  Most reasonably current distros will have a scheduled cron or 
systemd-timer job for that, tho you may or may not need to enable it.  
Weekly should be fine for most users, tho for ssds nearing capacity, 
daily may be useful.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs_remove_chunk call trace?

[Rich Rauenzahn]

On 9/11/2017 1:35 PM, Duncan wrote:
> Rich Rauenzahn posted on Sun, 10 Sep 2017 22:45:50 -0700 as excerpted:
>
>> ...and can it be related to the Samsung 840 SSD's not supporting NCQ
>> Trim?  (Although I can't tell which device this trace is from -- it
>> could be a mechanical Western Digital.)
>>
>> On Sun, Sep 10, 2017 at 10:16 PM, Rich Rauenzahn <rrauenza@gmail.com>
>> wrote:
>>> Is this something to be concerned about?
>>>
>>> I'm running the latest mainline kernel on CentOS 7.
>>>
>>> [ 1338.891412] CPU: 2 PID: 790 Comm: btrfs-cleaner
>>> Tainted: G        W     4.13.1-1.el7.elrepo.x86_64 #1
> As a Samsung ssd (tho 850) owner myself, who looked into mounting with
> the discard mount option on my new ssds here...
>
> Samsung ssds lack of queued-trim shouldn't be a problem on anything close
> to a current kernel, obviously including the 4.13.1 you're running above,
> because the kernel block layer has blacklisted queued-trim on all Samsung
> SSDs for at least several kernel cycles (not sure when it went in, but
> there was a time in the 3.x kernel era when queued-trim on samsung ssds
> was causing problems, thus the blacklisting).

Looking at the source in git.kernel.org, yes, it looks like it was 
disabled quite some time ago for "Samsung SSD 8*".  And I'm not mounting 
with discard ... and I've added a weekly fstrim --all.  (I didn't see 
one in CentOS7.)

So what is causing these traces?  ....they look ... troubling.

Sep 11 13:03:01 tendo kernel: ------------[ cut here ]------------
Sep 11 13:03:01 tendo kernel: WARNING: CPU: 0 PID: 519 at 
fs/btrfs/ctree.h:1559 btrfs_update_device+0x1c5/0x1d0 [btrfs]
Sep 11 13:03:01 tendo kernel: Modules linked in: xt_nat ipt_MASQUERADE 
nf_nat_masquerade_ipv4 nf_conntrack_ftp xt_conntrack iptable_filter 
xt_REDIRECT nf_nat_redirect iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_nat_ipv4 nf_nat nf_conntrack ip_tables veth binfmt_misc xt_addrtype 
overlay loop nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_comment 
xt_recent xt_multiport nct6775 hwmon_vid jc42 vfat fat dm_mirror 
dm_region_hash dm_log dm_mod dax xfs libcrc32c x86_pkg_temp_thermal 
intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul 
crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd 
glue_helper cryptd iTCO_wdt iTCO_vendor_support mei_wdt intel_cstate 
snd_hda_codec_hdmi input_leds intel_rapl_perf sg cdc_acm shpchp 
intel_pch_thermal snd_hda_codec_realtek i2c_i801
  ie31200_edac snd_hda_codec_genericSep 11 13:03:01 tendo kernel: 
snd_hda_intel snd_hda_codec mei_me snd_hda_core snd_seq lpc_ich pcspkr 
snd_seq_device mfd_core mei snd_hwdep snd_pcm snd_timer acpi_pad snd 
soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc btrfs xor raid6_pq 
sd_mod crc32c_intel i915 firewire_ohci ahci e1000e libahci igb 
firewire_core crc_itu_t dca drm_kms_helper i2c_algo_bit syscopyarea ptp 
libata sysfillrect sysimgblt pps_core fb_sys_fops drm video [last 
unloaded: nf_conntrack]
Sep 11 13:03:01 tendo kernel: CPU: 0 PID: 519 Comm: systemd-journal 
Tainted: G        W       4.13.1-1.el7.elrepo.x86_64 #1
Sep 11 13:03:01 tendo kernel: Hardware name: Supermicro X10SAE/X10SAE, 
BIOS 2.0a 05/09/2014
Sep 11 13:03:01 tendo kernel: task: ffff880407f95d00 task.stack: 
ffffc900022b8000
Sep 11 13:03:01 tendo kernel: RIP: 0010:btrfs_update_device+0x1c5/0x1d0 
[btrfs]
Sep 11 13:03:01 tendo kernel: RSP: 0018:ffffc900022bbb00 EFLAGS: 00010206
Sep 11 13:03:01 tendo kernel: RAX: 0000000000000fff RBX: 
ffff880404c392a0 RCX: 0000001bc6c71e00
Sep 11 13:03:01 tendo kernel: RDX: ffff880000000000 RSI: 
ffff880244ed0f3c RDI: ffff880236623068
Sep 11 13:03:01 tendo kernel: RBP: ffffc900022bbb48 R08: 
0000000000003f60 R09: ffffc900022bbab8
Sep 11 13:03:01 tendo kernel: R10: 0000000000001000 R11: 
0000000000000003 R12: ffff880405fc7800
Sep 11 13:03:01 tendo kernel: R13: 0000000000000000 R14: 
ffff880236623068 R15: 0000000000003f3c
Sep 11 13:03:01 tendo kernel: FS:  00007fafb608e880(0000) 
GS:ffff88041fa00000(0000) knlGS:0000000000000000
Sep 11 13:03:01 tendo kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
Sep 11 13:03:01 tendo kernel: CR2: 00007fafb60ad000 CR3: 
000000040493b000 CR4: 00000000001406f0
Sep 11 13:03:01 tendo kernel: DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Sep 11 13:03:01 tendo kernel: DR3: 0000000000000000 DR6: 
00000000fffe0ff0 DR7: 0000000000000400
Sep 11 13:03:01 tendo kernel: Call Trace:
Sep 11 13:03:01 tendo kernel: btrfs_finish_chunk_alloc+0x126/0x4e0 [btrfs]
Sep 11 13:03:01 tendo kernel: ? btrfs_insert_item+0x80/0xf0 [btrfs]
Sep 11 13:03:01 tendo kernel: 
btrfs_create_pending_block_groups+0x13f/0x260 [btrfs]
Sep 11 13:03:01 tendo kernel: __btrfs_end_transaction+0x93/0x2f0 [btrfs]
Sep 11 13:03:01 tendo kernel: btrfs_end_transaction+0x10/0x20 [btrfs]
Sep 11 13:03:01 tendo kernel: __btrfs_prealloc_file_range+0x378/0x4a0 
[btrfs]
Sep 11 13:03:01 tendo kernel: btrfs_prealloc_file_range+0x23/0x30 [btrfs]
Sep 11 13:03:01 tendo kernel: btrfs_fallocate+0x554/0x7a0 [btrfs]
Sep 11 13:03:01 tendo kernel: vfs_fallocate+0x15b/0x290
Sep 11 13:03:01 tendo kernel: SyS_fallocate+0x44/0x70
Sep 11 13:03:01 tendo kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5
Sep 11 13:03:01 tendo kernel: RIP: 0033:0x7fafb4cab119
Sep 11 13:03:01 tendo kernel: RSP: 002b:00007ffcb99c5d00 EFLAGS: 
00000246 ORIG_RAX: 000000000000011d
Sep 11 13:03:01 tendo kernel: RAX: ffffffffffffffda RBX: 
0000558751d9f3d0 RCX: 00007fafb4cab119
Sep 11 13:03:01 tendo kernel: RDX: 0000000005000000 RSI: 
0000000000000000 RDI: 0000000000000013
Sep 11 13:03:01 tendo kernel: RBP: 0000000000000000 R08: 
0000000000000013 R09: 0000000005000000
Sep 11 13:03:01 tendo kernel: R10: 0000000000800000 R11: 
0000000000000246 R12: 0000000000000000
Sep 11 13:03:01 tendo kernel: R13: 00007ffcb99c58a0 R14: 
00007ffcb99c58ae R15: 00007ffcb99c58a2
Sep 11 13:03:01 tendo kernel: Code: 10 00 00 00 4c 89 fe e8 8a 30 ff ff 
4c 89 f7 e8 32 f6 fc ff e9 d3 fe ff ff b8 f4 ff ff ff e9 d4 fe ff ff 0f 
1f 00 e8 cb 5e be e0 <0f> ff eb af 0f 1f 80 00 00 00 00 0f 1f 44 00 00 
55 31 d2 be 02
Sep 11 13:03:01 tendo kernel: ---[ end trace 96a2d96b3058d885 ]---
Sep 11 13:03:02 tendo sh: abrt-dump-oops: Found oopses: 1
Sep 11 13:03:02 tendo sh: abrt-dump-oops: Creating problem directories
Sep 11 13:03:02 tendo sh: abrt-dump-oops: Not going to make dump 
directories world readable because PrivateReports is on
Sep 11 13:03:02 tendo abrt-server: Looking for kernel package