Re: fatal database corruption with btrfs "out of space" with ~50 GB left

[Nikolay Borisov <nborisov@suse.com>]

On 15.02.2018 09:17, Tomasz Chmielewski wrote:
> On 2018-02-15 16:02, Tomasz Chmielewski wrote:
>> On 2018-02-15 13:32, Qu Wenruo wrote:
>>
>>> Is there any kernel message like kernel warning or backtrace?
>>
>> I see there was this one:
>>
>> Feb 13 13:53:32 lxd01 kernel: [9351710.878404] ------------[ cut here
>> ]------------
>> Feb 13 13:53:32 lxd01 kernel: [9351710.878430] WARNING: CPU: 9 PID:
>> 7780 at /home/kernel/COD/linux/fs/btrfs/tree-log.c:3361
>> log_dir_items+0x54b/0x560 [btrfs]
> (...)
>> Feb 13 13:53:32 lxd01 kernel: [9351710.878707] ---[ end trace
>> 81aeb3fb0c68ce00 ]---
>>
>>
>> BTW we've updated to the latest 4.15 kernel after that.
> 
> Also, we were just running a balance there and it printed this in dmesg.
> 
> Not sure if it's something serious or not. The filesystem still runs
> correctly.
> 
> [60082.349447] WARNING: CPU: 2 PID: 780 at
> /home/kernel/COD/linux/fs/btrfs/extent-tree.c:124
> btrfs_put_block_group+0x4f/0x60 [btrfs]
> [60082.349449] Modules linked in: xt_nat xt_REDIRECT nf_nat_redirect
> tcp_diag inet_diag sunrpc xt_NFLOG cfg80211 xt_conntrack nfnetlink_log
> nfnetlink ipt_REJECT nf_reject_ipv4 binfmt_misc veth ebtable_filter
> ebtables ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat
> nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment nf_log_ipv4
> nf_log_common xt_LOG ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_vs nf_conntrack
> ip6table_filter ip6_tables iptable_filter xt_CHECKSUM xt_tcpudp
> iptable_mangle ip_tables x_tables intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel btrfs bridge pcbc
> zstd_compress stp llc aesni_intel aes_x86_64 crypto_simd glue_helper
> input_leds
> [60082.349471]  cryptd intel_cstate intel_rapl_perf serio_raw shpchp
> lpc_ich acpi_pad mac_hid autofs4 raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1
> raid0 multipath linear ttm drm_kms_helper syscopyarea sysfillrect
> sysimgblt fb_sys_fops igb drm dca ahci ptp libahci pps_core i2c_algo_bit
> wmi
> [60082.349488] CPU: 2 PID: 780 Comm: btrfs-cleaner Tainted: G        W  
>      4.15.3-041503-generic #201802120730
> [60082.349489] Hardware name: ASUSTeK COMPUTER INC. Z10PA-U8
> Series/Z10PA-U8 Series, BIOS 0601 06/26/2015
> [60082.349497] RIP: 0010:btrfs_put_block_group+0x4f/0x60 [btrfs]
> [60082.349497] RSP: 0018:ffffc1468d3afe48 EFLAGS: 00010206
> [60082.349498] RAX: 0000000000000000 RBX: ffff9d072f740000 RCX:
> 0000000000000000
> [60082.349499] RDX: 0000000000000001 RSI: 0000000000000246 RDI:
> ffff9d072bcb8c00
> [60082.349499] RBP: ffffc1468d3afe50 R08: ffff9d072bcbfc00 R09:
> 0000000180200019
> [60082.349500] R10: ffffc1468d3afe38 R11: 0000000000000100 R12:
> ffff9d072b87ce00
> [60082.349500] R13: ffff9d072bcb8c00 R14: ffff9d072b87ced0 R15:
> ffff9d072bcb8d20
> [60082.349501] FS:  0000000000000000(0000) GS:ffff9d073f280000(0000)
> knlGS:0000000000000000
> [60082.349502] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [60082.349502] CR2: 00007f9221375000 CR3: 00000019a660a005 CR4:
> 00000000001606e0
> [60082.349503] Call Trace:
> [60082.349513]  btrfs_delete_unused_bgs+0x243/0x3c0 [btrfs]
> [60082.349521]  cleaner_kthread+0x159/0x170 [btrfs]
> [60082.349524]  kthread+0x121/0x140
> [60082.349531]  ? __btree_submit_bio_start+0x20/0x20 [btrfs]
> [60082.349533]  ? kthread_create_worker_on_cpu+0x70/0x70
> [60082.349535]  ret_from_fork+0x35/0x40
> [60082.349536] Code: e8 01 00 00 48 85 c0 75 1a 48 89 fb 48 8b bf d8 00
> 00 00 e8 14 3a 66 d7 48 89 df e8 0c 3a 66 d7 5b 5d c3 0f ff eb e2 0f ff
> eb cb <0f> ff eb ce 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
> [60082.349554] ---[ end trace 9492ee1b902c858d ]---
> 

If you have the vmlinux for that particular kernel can you run the 
following command and provide the output: 

./scripts/faddr2line <path-to-vmlinux> btrfs_delete_unused_bgs+0x243/0x3c0

./scripts/faddr2line is in the kernel source directory. This 
warn on indicates we are freeing an unused blockgroup (one which 
shouldn't have any space reserved in it) yet we trigger a warning 
due to having reserved space for that block group. Quite strange

> 
> Tomasz Chmielewski
> https://lxadm.com
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>