20.11.2020 12:24, Kristupas Savickas пишет: > Hello, > > I tried to convert my (nearly full) file system to raid5. The fs originally contained 3 8TB disks. > After adding the forth disk I ran: > # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft -sconvert=raid5,soft -b /mnt/ > Using raid5 for metadata is not recommended with current state of raid5 support in btrfs. See also https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/ > However, it looks like the conversion failed and the fs was left in RO state, with only part of the data being converted to raid5: > > # btrfs fi show /dev/sdc > Label: 'array' uuid: 6e95de0a-4b51-4aab-b935-469626c83036 > Total devices 4 FS bytes used 21.52TiB > devid 1 size 7.28TiB used 7.28TiB path /dev/sdc > devid 2 size 7.28TiB used 7.28TiB path /dev/sdd > devid 3 size 7.28TiB used 7.28TiB path /dev/sda > devid 4 size 7.28TiB used 7.28TiB path /dev/sdb > > # mount /dev/sdc /mnt/ > # mount | grep /mnt > /dev/sdc on /mnt type btrfs (ro,relatime,space_cache,subvolid=5,subvol=/) > > # btrfs fi df /mnt > Data, single: total=14.22TiB, used=14.06TiB > Data, RAID5: total=7.44TiB, used=7.43TiB > System, RAID1: total=32.00MiB, used=2.47MiB > System, RAID5: total=32.00MiB, used=128.00KiB > Metadata, RAID1: total=13.00GiB, used=11.67GiB > Metadata, RAID5: total=12.00GiB, used=11.90GiB > GlobalReserve, single: total=512.00MiB, used=0.00B > WARNING: Multiple block group profiles detected, see 'man btrfs(5)'. > WARNING: Data: single, raid5 > WARNING: Metadata: raid1, raid5 > WARNING: System: raid1, raid5 > "btrfs filesystem usage -T" fives better overview, but it appears there are is not enough space to allocate new raid5 chunks. > # dmesg > [65134.312783] BTRFS info (device sdc): disk space caching is enabled > [65134.312784] BTRFS info (device sdc): has skinny extents > [65136.207839] BTRFS info (device sdc): bdev /dev/sdd errs: wr 0, rd 4, flush 0, corrupt 0, gen 0 > [65172.993492] BTRFS alert (device sdc): btrfs RAID5/6 is EXPERIMENTAL and has known data-loss bugs > [65204.813036] BTRFS info (device sdc): checking UUID tree > [65211.733565] ------------[ cut here ]------------ > [65211.733567] BTRFS: Transaction aborted (error -28) > [65211.733583] BTRFS: error (device sdc) in __btrfs_free_extent:3069: errno=-28 No space left > [65211.733629] WARNING: CPU: 2 PID: 19980 at fs/btrfs/extent-tree.c:3069 __btrfs_free_extent.isra.0+0x57e/0x8f0 [btrfs] > [65211.735427] BTRFS info (device sdc): forced readonly > [65211.735427] Modules linked in: xt_recent fuse ufs > [65211.735430] BTRFS: error (device sdc) in btrfs_run_delayed_refs:2173: errno=-28 No space left > [65211.735431] qnx4 hfsplus hfs cdrom minix msdos jfs xfs dm_mod uas usb_storage xt_nat veth xt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_chain_nat nf_nat br_netfilter bridge stp llc overlay hid_logitech_hidpp joydev hid_logitech_dj hid_generic usbhid hid amdgpu edac_mce_amd kvm_amd kvm irqbypass ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ghash_clmulni_intel nls_ascii nls_cp437 ppdev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg ip6t_rt snd_hda_codec vfat fat gpu_sched snd_hda_core ttm snd_hwdep ipt_REJECT nf_reject_ipv4 efi_pstore aesni_intel xt_comment wmi_bmof libaes crypto_simd cryptd snd_pcm xt_multiport glue_helper rapl drm_kms_helper ccp snd_timer sp5100_tco efivars pcspkr k10temp watchdog nft_limit snd cec rng_core soundcore i2c_algo_bit sg parport_pc parport evdev xt_limit xt_addrtype xt_tcpudp acpi_cpufreq button xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nft_counter nf_tables > [65211.735467] nfnetlink drm efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor zstd_compress raid6_pq libcrc32c crc32c_generic sd_mod nvme nvme_core crc32_pclmul t10_pi ahci libahci r8169 realtek mdio_devres crc_t10dif libphy i2c_piix4 crct10dif_generic libata crc32c_intel xhci_pci xhci_hcd usbcore scsi_mod crct10dif_pclmul crct10dif_common usb_common wmi video gpio_amdpt gpio_generic > [65211.737278] CPU: 2 PID: 19980 Comm: btrfs-transacti Tainted: G W X 5.9.0-1-amd64 #1 Debian 5.9.1-1 > [65211.737280] BTRFS info (device sdc): balance: resume -dconvert=raid5,soft -mconvert=raid5,soft -sconvert=raid5,soft > [65211.737281] Hardware name: Gigabyte Technology Co., Ltd. AB350M-D3H/AB350M-D3H-CF, BIOS F51c 07/02/2020 > [65211.737300] RIP: 0010:__btrfs_free_extent.isra.0+0x57e/0x8f0 [btrfs] > [65211.737302] Code: 24 0c ba 5b 0c 00 00 48 c7 c6 40 01 5d c0 4c 89 f7 e8 12 c2 0a 00 e9 b1 fe ff ff 44 89 e6 48 c7 c7 80 a5 5d c0 e8 98 c5 16 d1 <0f> 0b 44 89 e1 ba fd 0b 00 00 48 c7 c6 40 01 5d c0 4c 89 f7 e8 e5 > [65211.737303] RSP: 0018:ffffa398c192bc50 EFLAGS: 00010282 > [65211.737305] RAX: 0000000000000000 RBX: 000026d0f54f8000 RCX: ffff969590898ac8 > [65211.737305] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff969590898ac0 > [65211.737306] RBP: 0000259405cbd000 R08: 0000000000000995 R09: 0000000000000004 > [65211.737307] R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ffffffe4 > [65211.737308] R13: ffff96955d4f0310 R14: ffff9692ab751d00 R15: 0000000000000005 > [65211.737309] FS: 0000000000000000(0000) GS:ffff969590880000(0000) knlGS:0000000000000000 > [65211.737310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [65211.737311] CR2: 000055ff8d180cb0 CR3: 00000003b621a000 CR4: 00000000003506e0 > [65211.737312] Call Trace: > [65211.737333] ? __btrfs_run_delayed_refs+0xfd3/0x1060 [btrfs] > [65211.737351] __btrfs_run_delayed_refs+0x27a/0x1060 [btrfs] > [65211.737371] btrfs_run_delayed_refs+0x73/0x200 [btrfs] > [65211.737391] btrfs_commit_transaction+0x57/0xa30 [btrfs] > [65211.737411] ? start_transaction+0xd2/0x540 [btrfs] > [65211.737415] ? try_to_wake_up+0x130/0x5e0 > [65211.737434] transaction_kthread+0x14c/0x170 [btrfs] > [65211.737453] ? btrfs_cleanup_transaction.isra.0+0x5a0/0x5a0 [btrfs] > [65211.737455] kthread+0x11b/0x140 > [65211.737457] ? __kthread_bind_mask+0x60/0x60 > [65211.737460] ret_from_fork+0x22/0x30 > [65211.737462] ---[ end trace 2f4a1b25242944de ]--- > [65211.737463] BTRFS: error (device sdc) in __btrfs_free_extent:3069: errno=-28 No space left > [65211.738722] BTRFS: error (device sdc) in btrfs_run_delayed_refs:2173: errno=-28 No space left > [65211.748466] BTRFS info (device sdc): balance: ended with status: -30 > > Looking at the dmesg output, I see: > [65211.737280] BTRFS info (device sdc): balance: resume -dconvert=raid5,soft -mconvert=raid5,soft -sconvert=raid5,soft > > This indicates that the balance is ran again after remounting. Looking at the last line it says that: > [65211.748466] BTRFS info (device sdc): balance: ended with status: -30 > > I suppose this is what causes the fs to be mounted read only. > > Trying to remount the system as RW results in: > [68268.865313] BTRFS info (device sdc): disk space caching is enabled > [68268.865316] BTRFS error (device sdc): Remounting read-write after error is not allowed > > My question is how can I recover from a situation like this? > I could definitely reduce the file system usage if I could mount it as RW, but > the balance is ran immediately after mounting and fails, which results in a RO mount. > Would it be possible to cancel the balance, so it doesn't run on mounting? > Mount with skip_balance mount option. > Additional information: > # uname -a > Linux s 5.9.0-1-amd64 #1 SMP Debian 5.9.1-1 (2020-10-17) x86_64 GNU/Linux > > # btrfs --version > btrfs-progs v5.9 >