btrfs scrub resulting in readonly filesystem

btrfs scrub resulting in readonly filesystem

[Lai Wei-Hwa]

Hi There, 

Found a host with readonly filesystem. This is the output from dmesg which occurred right after a cron job was executed to scrub. 



[Sep14 20:02] ------------[ cut here ]------------ 
[ +0.000042] WARNING: CPU: 18 PID: 28882 at /build/linux-TSOhpZ/linux-4.4.0/fs/btrfs/extent-tree.c:10046 btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]() 
[ +0.000002] BTRFS: Transaction aborted (error -27) 
[ +0.000002] Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common xt_LOG xt_limit xt_tcpudp xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid unix_diag veth bridge stp llc bonding binfmt_misc zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_devintf ipmi_ssif ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul dcdbas glue_helper input_leds ablk_helper ipmi_si gpio_ich serio_raw i7core_edac joydev 8250_fintek acpi_power_meter mac_hid ipmi_msghandler shpchp edac_core cryptd lpc_ich ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 
[ +0.000047] autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas uas hid_generic usbhid usb_storage wmi psmouse hid pata_acpi megaraid_sas bnx2 fjes 
[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu 
[ +0.000002] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013 
[ +0.000001] 0000000000000286 1faf5c66f8e16e90 ffff8802adbcfa18 ffffffff8140c9a1 
[ +0.000003] ffff8802adbcfa60 ffffffffc024d588 ffff8802adbcfa50 ffffffff810864d2 
[ +0.000002] ffff880697e29720 ffff880c11881da0 ffff880c0f6e6800 ffff880697e29640 
[ +0.000002] Call Trace: 
[ +0.000008] [<ffffffff8140c9a1>] dump_stack+0x63/0x82 
[ +0.000007] [<ffffffff810864d2>] warn_slowpath_common+0x82/0xc0 
[ +0.000002] [<ffffffff8108656c>] warn_slowpath_fmt+0x5c/0x80 
[ +0.000014] [<ffffffffc01f31c4>] ? btrfs_finish_chunk_alloc+0x204/0x5a0 [btrfs] 
[ +0.000011] [<ffffffffc01b1d24>] btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs] 
[ +0.000012] [<ffffffffc01c7ed3>] __btrfs_end_transaction+0x93/0x340 [btrfs] 
[ +0.000013] [<ffffffffc01c8190>] btrfs_end_transaction+0x10/0x20 [btrfs] 
[ +0.000010] [<ffffffffc01b5a4d>] btrfs_inc_block_group_ro+0xed/0x1b0 [btrfs] 
[ +0.000014] [<ffffffffc02253bf>] scrub_enumerate_chunks+0x21f/0x580 [btrfs] 
[ +0.000004] [<ffffffff810cb700>] ? wake_atomic_t_function+0x60/0x60 
[ +0.000013] [<ffffffffc0226d0c>] btrfs_scrub_dev+0x1bc/0x530 [btrfs] 
[ +0.000004] [<ffffffff8123f306>] ? __mnt_want_write+0x56/0x60 
[ +0.000013] [<ffffffffc0202408>] btrfs_ioctl+0x1ac8/0x28c0 [btrfs] 
[ +0.000003] [<ffffffff8119a3b9>] ? unlock_page+0x69/0x70 
[ +0.000002] [<ffffffff8119a654>] ? filemap_map_pages+0x224/0x230 
[ +0.000004] [<ffffffff811cdb77>] ? handle_mm_fault+0x10f7/0x1b80 
[ +0.000002] [<ffffffff811fb77b>] ? kmem_cache_alloc_node+0xbb/0x210 
[ +0.000003] [<ffffffff813e13e3>] ? create_task_io_context+0x23/0x100 
[ +0.000003] [<ffffffff812318ef>] do_vfs_ioctl+0x2af/0x4b0 
[ +0.000002] [<ffffffff813e1510>] ? get_task_io_context+0x50/0x90 
[ +0.000003] [<ffffffff813f0936>] ? set_task_ioprio+0x86/0xa0 
[ +0.000002] [<ffffffff81231b69>] SyS_ioctl+0x79/0x90 
[ +0.000004] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb 
[ +0.000002] ---[ end trace 13fce4e84d9b6aed ]--- 
[ +0.000003] BTRFS: error (device sda1) in btrfs_create_pending_block_groups:10046: errno=-27 unknown 
[ +0.003942] BTRFS info (device sda1): forced readonly 
[ +0.000193] pending csums is 28672 
[ +0.002295] BTRFS warning (device sda1): failed setting block group ro, ret=-30 

[ +7.197271] BTRFS warning (device sda1): failed setting block group ro, ret=-30 



What's the cause here? 


Best Regards, 

Lai Wei-Hwa

Thanks! 
Lai

Re: btrfs scrub resulting in readonly filesystem

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 5251 bytes --]



On 2019/9/17 上午4:32, Lai Wei-Hwa wrote:
> Hi There, 
> 
> Found a host with readonly filesystem. This is the output from dmesg which occurred right after a cron job was executed to scrub. 
> 
> 
> 
> [Sep14 20:02] ------------[ cut here ]------------ 
> [ +0.000042] WARNING: CPU: 18 PID: 28882 at /build/linux-TSOhpZ/linux-4.4.0/fs/btrfs/extent-tree.c:10046 btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]() 
> [ +0.000002] BTRFS: Transaction aborted (error -27) 
> [ +0.000002] Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common xt_LOG xt_limit xt_tcpudp xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid unix_diag veth bridge stp llc bonding binfmt_misc zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_devintf ipmi_ssif ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul dcdbas glue_helper input_leds ablk_helper ipmi_si gpio_ich serio_raw i7core_edac joydev 8250_fintek acpi_power_meter mac_hid ipmi_msghandler shpchp edac_core cryptd lpc_ich ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 
> [ +0.000047] autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas uas hid_generic usbhid usb_storage wmi psmouse hid pata_acpi megaraid_sas bnx2 fjes 
> [ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu 

Although your old kernel is not causing the problem of this case, it's
still recommended to upgrade to a newer kernel for btrfs usage.

> [ +0.000002] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013 
> [ +0.000001] 0000000000000286 1faf5c66f8e16e90 ffff8802adbcfa18 ffffffff8140c9a1 
> [ +0.000003] ffff8802adbcfa60 ffffffffc024d588 ffff8802adbcfa50 ffffffff810864d2 
> [ +0.000002] ffff880697e29720 ffff880c11881da0 ffff880c0f6e6800 ffff880697e29640 
> [ +0.000002] Call Trace: 
> [ +0.000008] [<ffffffff8140c9a1>] dump_stack+0x63/0x82 
> [ +0.000007] [<ffffffff810864d2>] warn_slowpath_common+0x82/0xc0 
> [ +0.000002] [<ffffffff8108656c>] warn_slowpath_fmt+0x5c/0x80 
> [ +0.000014] [<ffffffffc01f31c4>] ? btrfs_finish_chunk_alloc+0x204/0x5a0 [btrfs] 
> [ +0.000011] [<ffffffffc01b1d24>] btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs] 
> [ +0.000012] [<ffffffffc01c7ed3>] __btrfs_end_transaction+0x93/0x340 [btrfs] 
> [ +0.000013] [<ffffffffc01c8190>] btrfs_end_transaction+0x10/0x20 [btrfs] 
> [ +0.000010] [<ffffffffc01b5a4d>] btrfs_inc_block_group_ro+0xed/0x1b0 [btrfs] 
> [ +0.000014] [<ffffffffc02253bf>] scrub_enumerate_chunks+0x21f/0x580 [btrfs] 
> [ +0.000004] [<ffffffff810cb700>] ? wake_atomic_t_function+0x60/0x60 
> [ +0.000013] [<ffffffffc0226d0c>] btrfs_scrub_dev+0x1bc/0x530 [btrfs] 
> [ +0.000004] [<ffffffff8123f306>] ? __mnt_want_write+0x56/0x60 
> [ +0.000013] [<ffffffffc0202408>] btrfs_ioctl+0x1ac8/0x28c0 [btrfs] 
> [ +0.000003] [<ffffffff8119a3b9>] ? unlock_page+0x69/0x70 
> [ +0.000002] [<ffffffff8119a654>] ? filemap_map_pages+0x224/0x230 
> [ +0.000004] [<ffffffff811cdb77>] ? handle_mm_fault+0x10f7/0x1b80 
> [ +0.000002] [<ffffffff811fb77b>] ? kmem_cache_alloc_node+0xbb/0x210 
> [ +0.000003] [<ffffffff813e13e3>] ? create_task_io_context+0x23/0x100 
> [ +0.000003] [<ffffffff812318ef>] do_vfs_ioctl+0x2af/0x4b0 
> [ +0.000002] [<ffffffff813e1510>] ? get_task_io_context+0x50/0x90 
> [ +0.000003] [<ffffffff813f0936>] ? set_task_ioprio+0x86/0xa0 
> [ +0.000002] [<ffffffff81231b69>] SyS_ioctl+0x79/0x90 
> [ +0.000004] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb 
> [ +0.000002] ---[ end trace 13fce4e84d9b6aed ]--- 
> [ +0.000003] BTRFS: error (device sda1) in btrfs_create_pending_block_groups:10046: errno=-27 unknown

-27 is -EFBIG.

There are only several users of that errno, and according to your
backtrace, it's from btrfs_finish_chunk_alloc() which will try to add
your system chunk to super block syschunk array.

While the syschunk array in super block has a fixed size, if super block
can no longer contain the new chunk, it returns -EFBIG and we abort
transaction, causing the problem.

To verify my analyse, please provide the following dump:
# btrfs ins dump-super -fFa /dev/sda1
# btrfs ins dump-tree -t 3 /dev/sda1

The latter dump needs to be done on an RO fs, or unmounted fs.

I don't have a good workaround right now, but it looks like you have too
many system chunks.
Normally this means you have too many data/metadata chunks, but I'd like
to see the dumps to make the final call on what's causing the problem.

Thanks,
Qu

> [ +0.003942] BTRFS info (device sda1): forced readonly 
> [ +0.000193] pending csums is 28672 
> [ +0.002295] BTRFS warning (device sda1): failed setting block group ro, ret=-30 
> 
> [ +7.197271] BTRFS warning (device sda1): failed setting block group ro, ret=-30 
> 
> 
> 
> What's the cause here? 
> 
> 
> Best Regards, 
> 
> Lai Wei-Hwa
> 
> Thanks! 
> Lai
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: btrfs scrub resulting in readonly filesystem

[Patrik Lundquist]

On Tue, 17 Sep 2019 at 10:21, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> On 2019/9/17 上午4:32, Lai Wei-Hwa wrote:
> > [ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu
>
> Although your old kernel is not causing the problem of this case, it's
> still recommended to upgrade to a newer kernel for btrfs usage.

https://wiki.ubuntu.com/Kernel/LTSEnablementStack