Re: Out of space loop: skip_balance not working

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2023/6/23 06:18, Stefan N wrote:
> Hi Qu,
>
> I got one new line this time, but it doesn't seem to match your commit
> ERROR: zoned: unable to stat /dev/loop/13

Please provide the dmesg of that attempt, as all the extra debug info is
inside dmesg.

With that info provided, we can determine what to do next.

Thanks,
Qu

>
> I tried it on the USB flash drives too and didn't get any extra line
>
> In context
> $ sudo mount -o skip_balance -t btrfs /dev/sde /mnt/data ; sudo btrfs
> dev add -K -f /dev/loop12 /dev/loop/13 /dev/loop14 /dev/loop15
> /mnt/data ; sudo btrfs fi sync /mnt/data
> ERROR: error adding device '/dev/loop12': Input/output error
> ERROR: zoned: unable to stat /dev/loop/13
> ERROR: checking status of /dev/loop/13: No such file or directory
> ERROR: error adding device '/dev/loop14': Read-only file system
> ERROR: error adding device '/dev/loop15': Read-only file system
> ERROR: Could not sync filesystem: Read-only file system
> $
>
> $ sudo mount -o skip_balance -t btrfs /dev/sde /mnt/data ; sudo btrfs
> dev add -f /dev/sdl /dev/sdm /dev/sdn /dev/sdo /mnt/data ; sudo btrfs
> fi sync /mnt/data
> ERROR: error adding device '/dev/sdl': Input/output error
> ERROR: error adding device '/dev/sdm': Read-only file system
> ERROR: error adding device '/dev/sdn': Read-only file system
> ERROR: error adding device '/dev/sdo': Read-only file system
> ERROR: Could not sync filesystem: Read-only file system
> $
>
> On Thu, 22 Jun 2023 at 18:48, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2023/6/22 16:33, Stefan N wrote:
>>> Hi Qu,
>>>
>>> Many thanks for the detailed instructions and your patience. I got it
>>> working combined with
>>> https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel on the main system
>>> OS instead, tagged +btrfix
>>> $ uname -vr
>>> 6.2.0-23-generic #23+btrfix SMP PREEMPT_DYNAMIC Thu Jun 22
>>>
>>> However, I've not had luck with the commands suggested, and would
>>> appreciate any further ideas.
>>>
>>> Outputs follow below, with /mnt/data as the btrfs mount point that
>>> currently contains 8x disks sd[a-j] with an additional 4x 64gb USB
>>> flash drives being added sd[l-o]
>>> $ sudo mount -o skip_balance -t btrfs /dev/sde /mnt/data ; sudo btrfs
>>> dev add -f /dev/sdl /dev/sdm /dev/sdn /dev/sdo /mnt/data ; sudo btrfs
>>> fi sync /mnt/data
>>> ERROR: error adding device '/dev/sdl': Input/output error
>>> ERROR: error adding device '/dev/sdm': Read-only file system
>>> ERROR: error adding device '/dev/sdn': Read-only file system
>>> ERROR: error adding device '/dev/sdo': Read-only file system
>>> ERROR: Could not sync filesystem: Read-only file system
>>> $
>>>
>>> The same occurs if I try to add 4x 100mb loop devices (on a ssd so
>>> they're super quick to zero);
>>> $ sudo mount -o skip_balance -t btrfs /dev/sde /mnt/data ; sudo btrfs
>>> dev add -K -f /dev/loop16 /dev/loop17 /dev/loop18 /dev/loop19
>>> /mnt/data ; sudo btrfs fi sync /mnt/data
>>> ERROR: error adding device '/dev/loop16': Input/output error
>>
>> This is the interesting part, this means we're erroring out due to -EIO
>> (not -ENOSPC) during the first device add.
>>
>> And by somehow, after the first device add, we already got the trans abort.
>>
>> Would you please try the following branch?
>>
>> https://github.com/adam900710/linux/tree/dev_add_no_commit
>>
>> It has not only the patch to skip the commit, but also extra debug
>> output for the situation.
>>
>> Thanks,
>> Qu
>>
>>> ERROR: error adding device '/dev/loop17': Read-only file system
>>> ERROR: error adding device '/dev/loop18': Read-only file system
>>> ERROR: error adding device '/dev/loop19': Read-only file system
>>> ERROR: Could not sync filesystem: Read-only file system
>>> $
>>>
>>> I confirmed before both these kernel builds that the replaced line was
>>> btrfs_end_transaction rather than btrfs_commit_transaction (anyone
>>> else following, I needed to remove the -n in the patch command
>>> earlier)
>>> $ grep -A3 -ri btrfs_sysfs_update_sprout */fs/btrfs/volumes.c*
>>> linux-6.2.0-dist/fs/btrfs/volumes.c:
>>> btrfs_sysfs_update_sprout_fsid(fs_devices);
>>> linux-6.2.0-dist/fs/btrfs/volumes.c-    }
>>> linux-6.2.0-dist/fs/btrfs/volumes.c-
>>> linux-6.2.0-dist/fs/btrfs/volumes.c-    ret = btrfs_commit_transaction(trans);
>>> --
>>> linux-6.2.0-v2/fs/btrfs/volumes.c:
>>> btrfs_sysfs_update_sprout_fsid(fs_devices);
>>> linux-6.2.0-v2/fs/btrfs/volumes.c-      }
>>> linux-6.2.0-v2/fs/btrfs/volumes.c-
>>> linux-6.2.0-v2/fs/btrfs/volumes.c-      ret = btrfs_end_transaction(trans);
>>> --
>>> linux-6.2.0-v3/fs/btrfs/volumes.c:
>>> btrfs_sysfs_update_sprout_fsid(fs_devices);
>>> linux-6.2.0-v3/fs/btrfs/volumes.c-      }
>>> linux-6.2.0-v3/fs/btrfs/volumes.c-
>>> linux-6.2.0-v3/fs/btrfs/volumes.c-      ret = btrfs_end_transaction(trans);
>>> $
>>>
>>> $ btrfs fi usage /mnt/data
>>> Overall:
>>>       Device size:                  87.31TiB
>>>       Device allocated:             87.31TiB
>>>       Device unallocated:            1.94GiB
>>>       Device missing:                  0.00B
>>>       Device slack:                    0.00B
>>>       Used:                         87.08TiB
>>>       Free (estimated):            173.29GiB      (min: 172.33GiB)
>>>       Free (statfs, df):           171.84GiB
>>>       Data ratio:                       1.34
>>>       Metadata ratio:                   4.00
>>>       Global reserve:              512.00MiB      (used: 371.25MiB)
>>>       Multiple profiles:                  no
>>>
>>> Data,RAID6: Size:64.76TiB, Used:64.59TiB (99.74%)
>>>      /dev/sdc       10.90TiB
>>>      /dev/sdf       10.90TiB
>>>      /dev/sda       10.86TiB
>>>      /dev/sdg       10.87TiB
>>>      /dev/sdh       10.86TiB
>>>      /dev/sdd       10.87TiB
>>>      /dev/sde       10.88TiB
>>>      /dev/sdb       10.88TiB
>>>
>>> Metadata,RAID1C4: Size:77.79GiB, Used:77.11GiB (99.12%)
>>>      /dev/sdc       15.33GiB
>>>      /dev/sdf       18.41GiB
>>>      /dev/sda       49.63GiB
>>>      /dev/sdg       49.50GiB
>>>      /dev/sdh       51.52GiB
>>>      /dev/sdd       48.70GiB
>>>      /dev/sde       39.09GiB
>>>      /dev/sdb       39.01GiB
>>>
>>> System,RAID1C4: Size:37.00MiB, Used:5.11MiB (13.81%)
>>>      /dev/sdc        1.00MiB
>>>      /dev/sda       37.00MiB
>>>      /dev/sdg       37.00MiB
>>>      /dev/sdh       36.00MiB
>>>      /dev/sdd       37.00MiB
>>>
>>> Unallocated:
>>>      /dev/sdc        1.00MiB
>>>      /dev/sdf        1.00MiB
>>>      /dev/sda        1.27GiB
>>>      /dev/sdg        1.00MiB
>>>      /dev/sdh        1.00MiB
>>>      /dev/sdd      687.00MiB
>>>      /dev/sde        1.00MiB
>>>      /dev/sdb        1.00MiB
>>> $
>>>
>>>
>>> This first attempt generated the following syslog output:
>>> kernel: [  868.435387] BTRFS info (device sde): using crc32c
>>> (crc32c-intel) checksum algorithm
>>> kernel: [  868.435407] BTRFS info (device sde): disk space caching is enabled
>>> kernel: [  874.477712] BTRFS info (device sde): bdev /dev/sdg errs: wr
>>> 0, rd 0, flush 0, corrupt 845, gen 0
>>> kernel: [  874.477727] BTRFS info (device sde): bdev /dev/sdc errs: wr
>>> 41089, rd 1556, flush 0, corrupt 0, gen 0
>>> kernel: [  874.477735] BTRFS info (device sde): bdev /dev/sdj errs: wr
>>> 3, rd 7, flush 0, corrupt 0, gen 0
>>> kernel: [  874.477740] BTRFS info (device sde): bdev /dev/sdf errs: wr
>>> 41, rd 0, flush 0, corrupt 0, gen 0
>>> kernel: [ 1082.645551] BTRFS info (device sde): balance: resume skipped
>>> kernel: [ 1082.645564] BTRFS info (device sde): checking UUID tree
>>> kernel: [ 1082.645551] BTRFS info (device sde): balance: resume skipped
>>> kernel: [ 1082.645564] BTRFS info (device sde): checking UUID tree
>>> kernel: [ 1267.280506] BTRFS: Transaction aborted (error -28)
>>> kernel: [ 1267.280553] BTRFS: error (device sde: state A) in
>>> do_free_extent_accounting:2847: errno=-28 No space left
>>> kernel: [ 1267.280604] BTRFS info (device sde: state EA): forced readonly
>>> kernel: [ 1267.280610] BTRFS error (device sde: state EA): failed to
>>> run delayed ref for logical 102255404044288 num_bytes 294912 type 184
>>> action 2 ref_mod 1: -28
>>> kernel: [ 1267.280584] WARNING: CPU: 3 PID: 14519 at
>>> fs/btrfs/extent-tree.c:2847 do_free_extent_accounting+0x21a/0x220
>>> [btrfs]
>>> kernel: [ 1267.280666] BTRFS: error (device sde: state EA) in
>>> btrfs_run_delayed_refs:2151: errno=-28 No space left
>>> kernel: [ 1267.280695] BTRFS warning (device sde: state EA):
>>> btrfs_uuid_scan_kthread failed -5
>>> kernel: [ 1267.280794] Modules linked in: xt_nat xt_tcpudp veth
>>> xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink
>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo
>>> xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc
>>> ipmi_devintf ipmi_msghandler overlay iwlwifi_compat(O) binfmt_misc
>>> nls_iso8859_1 intel_rapl_msr intel_rapl_common edac_mce_amd
>>> snd_hda_codec_realtek kvm_amd snd_hda_codec_generic ledtrig_audio kvm
>>> snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi
>>> snd_hda_codec irqbypass snd_hda_core snd_hwdep rapl snd_pcm snd_timer
>>> wmi_bmof k10temp snd ccp soundcore input_leds mac_hid dm_multipath
>>> scsi_dh_rdac scsi_dh_emc scsi_dh_alua bonding tls efi_pstore msr nfsd
>>> auth_rpcgss nfs_acl lockd grace sunrpc dmi_sysfs ip_tables x_tables
>>> autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov
>>> async_memcpy async_pq async_xor async_txxor raid6_pq libcrc32c raid1
>>> raid0 multipath linear hid_generic usbhid hid amdgpu uas usb_storage
>>> kernel: [ 1267.280994] CPU: 3 PID: 14519 Comm: btrfs-transacti
>>> Tainted: G        W  O       6.2.0-23-generic #23+btrfix
>>> kernel: [ 1267.281005] RIP: 0010:do_free_extent_accounting+0x21a/0x220 [btrfs]
>>> kernel: [ 1267.281181]  __btrfs_free_extent+0x6bc/0xf50 [btrfs]
>>> kernel: [ 1267.281310]  run_delayed_data_ref+0x8b/0x180 [btrfs]
>>> kernel: [ 1267.281444]  btrfs_run_delayed_refs_for_head+0x196/0x520 [btrfs]
>>> kernel: [ 1267.281570]  __btrfs_run_delayed_refs+0xe6/0x1d0 [btrfs]
>>> kernel: [ 1267.281694]  btrfs_run_delayed_refs+0x6d/0x1f0 [btrfs]
>>> kernel: [ 1267.281818]  btrfs_start_dirty_block_groups+0x36b/0x530 [btrfs]
>>> kernel: [ 1267.281976]  btrfs_commit_transaction+0xb3/0xbc0 [btrfs]
>>> kernel: [ 1267.282110]  ? start_transaction+0xc8/0x600 [btrfs]
>>> kernel: [ 1267.282244]  transaction_kthread+0x14b/0x1c0 [btrfs]
>>> kernel: [ 1267.282375]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
>>> kernel: [ 1267.282548] BTRFS info (device sde: state EA): dumping space info:
>>> kernel: [ 1267.282552] BTRFS info (device sde: state EA): space_info
>>> DATA has 160777674752 free, is not full
>>> kernel: [ 1267.282558] BTRFS info (device sde: state EA): space_info
>>> total=71201958395904, used=71018191273984, pinned=22985908224,
>>> reserved=0, may_use=0, readonly=3538944 zone_unusable=0
>>> kernel: [ 1267.282566] BTRFS info (device sde: state EA): space_info
>>> METADATA has -124944384 free, is full
>>> kernel: [ 1267.282571] BTRFS info (device sde: state EA): space_info
>>> total=83530612736, used=82791497728, pinned=242745344,
>>> reserved=496369664, may_use=124944384, readonly=0 zone_unusable=0
>>> kernel: [ 1267.282577] BTRFS info (device sde: state EA): space_info
>>> SYSTEM has 33439744 free, is not full
>>> kernel: [ 1267.282582] BTRFS info (device sde: state EA): space_info
>>> total=38797312, used=5357568, pinned=0, reserved=0, may_use=0,
>>> readonly=0 zone_unusable=0
>>> kernel: [ 1267.282588] BTRFS info (device sde: state EA):
>>> global_block_rsv: size 536870912 reserved 124944384
>>> kernel: [ 1267.282592] BTRFS info (device sde: state EA):
>>> trans_block_rsv: size 0 reserved 0
>>> kernel: [ 1267.282595] BTRFS info (device sde: state EA):
>>> chunk_block_rsv: size 0 reserved 0
>>> kernel: [ 1267.282599] BTRFS info (device sde: state EA):
>>> delayed_block_rsv: size 0 reserved 0
>>> kernel: [ 1267.282602] BTRFS info (device sde: state EA):
>>> delayed_refs_rsv: size 251322957824 reserved 0
>>> kernel: [ 1267.282608] BTRFS: error (device sde: state EA) in
>>> do_free_extent_accounting:2847: errno=-28 No space left
>>> kernel: [ 1267.282653] BTRFS error (device sde: state EA): failed to
>>> run delayed ref for logical 102255401897984 num_bytes 126976 type 184
>>> action 2 ref_mod 1: -28
>>> kernel: [ 1267.282708] BTRFS: error (device sde: state EA) in
>>> btrfs_run_delayed_refs:2151: errno=-28 No space left
>>>
>>> A couple of kernel recompiles later, the second attempt on the SSD
>>> generated similar:
>>> kernel: [ 1472.203470] BTRFS info (device sdc): using crc32c
>>> (crc32c-intel) checksum algorithm
>>> kernel: [ 1472.203491] BTRFS info (device sdc): disk space caching is enabled
>>> kernel: [ 1478.155004] BTRFS info (device sdc): bdev /dev/sdf errs: wr
>>> 0, rd 0, flush 0, corrupt 845, gen 0
>>> kernel: [ 1478.155022] BTRFS info (device sdc): bdev /dev/sda errs: wr
>>> 41089, rd 1556, flush 0, corrupt 0, gen 0
>>> kernel: [ 1478.155034] BTRFS info (device sdc): bdev /dev/sdh errs: wr
>>> 3, rd 7, flush 0, corrupt 0, gen 0
>>> kernel: [ 1478.155041] BTRFS info (device sdc): bdev /dev/sdd errs: wr
>>> 41, rd 0, flush 0, corrupt 0, gen 0
>>> kernel: [ 1696.662526] BTRFS info (device sdc): balance: resume skipped
>>> kernel: [ 1696.662537] BTRFS info (device sdc): checking UUID tree
>>> kernel: [ 1919.452464] BTRFS: Transaction aborted (error -28)
>>> kernel: [ 1919.452534] WARNING: CPU: 1 PID: 161 at
>>> fs/btrfs/extent-tree.c:2847 do_free_extent_accounting+0x21a/0x220
>>> [btrfs]
>>> kernel: [ 1919.452655] Modules linked in: xt_nat xt_tcpudp veth
>>> xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink
>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo
>>> xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc
>>> ipmi_devintf ipmi_msghandler overlay iwlwifi_compat(O) binfmt_misc
>>> nls_iso8859_1 snd_hda_codec_realtek snd_hda_codec_generic
>>> ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg
>>> snd_intel_sdw_acpi snd_hda_codec intel_rapl_msr snd_hda_core
>>> intel_rapl_common edac_mce_amd snd_hwdep kvm_amd snd_pcm snd_timer kvm
>>> irqbypass rapl wmi_bmof snd k10temp soundcore ccp input_leds mac_hid
>>> dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua bonding tls nfsd
>>> msr auth_rpcgss efi_pstore nfs_acl lockd grace sunrpc dmi_sysfs
>>> ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456
>>> async_raid6_recov async_memcpy async_pq async_xor async_tx xor
>>> raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid
>>> amdgpu uas hid iommu_v2
>>> kernel: [ 1919.452839] Workqueue: events_unbound
>>> btrfs_async_reclaim_metadata_space [btrfs]
>>> kernel: [ 1919.452985] RIP: 0010:do_free_extent_accounting+0x21a/0x220 [btrfs]
>>> kernel: [ 1919.453141]  __btrfs_free_extent+0x6bc/0xf50 [btrfs]
>>> kernel: [ 1919.453256]  run_delayed_data_ref+0x8b/0x180 [btrfs]
>>> kernel: [ 1919.453368]  btrfs_run_delayed_refs_for_head+0x196/0x520 [btrfs]
>>> kernel: [ 1919.453480]  __btrfs_run_delayed_refs+0xe6/0x1d0 [btrfs]
>>> kernel: [ 1919.453592]  btrfs_run_delayed_refs+0x6d/0x1f0 [btrfs]
>>> kernel: [ 1919.453703]  flush_space+0x23c/0x2c0 [btrfs]
>>> kernel: [ 1919.453845]  btrfs_async_reclaim_metadata_space+0x19b/0x2b0 [btrfs]
>>> kernel: [ 1919.454034] BTRFS info (device sdc: state A): dumping space info:
>>> kernel: [ 1919.454038] BTRFS info (device sdc: state A): space_info
>>> DATA has 160778723328 free, is not full
>>> kernel: [ 1919.454043] BTRFS info (device sdc: state A): space_info
>>> total=71201958395904, used=71017442181120, pinned=23733952512,
>>> reserved=0, may_use=0, readonly=3538944 zone_unusable=0
>>> kernel: [ 1919.454050] BTRFS info (device sdc: state A): space_info
>>> METADATA has -147570688 free, is full
>>> kernel: [ 1919.454054] BTRFS info (device sdc: state A): space_info
>>> total=83530612736, used=82792185856, pinned=238059520,
>>> reserved=500367360, may_use=147570688, readonly=0 zone_unusable=0
>>> kernel: [ 1919.454060] BTRFS info (device sdc: state A): space_info
>>> SYSTEM has 33439744 free, is not full
>>> kernel: [ 1919.454064] BTRFS info (device sdc: state A): space_info
>>> total=38797312, used=5357568, pinned=0, reserved=0, may_use=0,
>>> readonly=0 zone_unusable=0
>>> kernel: [ 1919.454070] BTRFS info (device sdc: state A):
>>> global_block_rsv: size 536870912 reserved 147570688
>>> kernel: [ 1919.454074] BTRFS info (device sdc: state A):
>>> trans_block_rsv: size 0 reserved 0
>>> kernel: [ 1919.454077] BTRFS info (device sdc: state A):
>>> chunk_block_rsv: size 0 reserved 0
>>> kernel: [ 1919.454080] BTRFS info (device sdc: state A):
>>> delayed_block_rsv: size 0 reserved 0
>>> kernel: [ 1919.454083] BTRFS info (device sdc: state A):
>>> delayed_refs_rsv: size 254292787200 reserved 0
>>> kernel: [ 1919.454086] BTRFS: error (device sdc: state A) in
>>> do_free_extent_accounting:2847: errno=-28 No space left
>>> kernel: [ 1919.454123] BTRFS info (device sdc: state EA): forced readonly
>>> kernel: [ 1919.454127] BTRFS error (device sdc: state EA): failed to
>>> run delayed ref for logical 102538713931776 num_bytes 245760 type 184
>>> action 2 ref_mod 1: -28
>>> kernel: [ 1919.454176] BTRFS: error (device sdc: state EA) in
>>> btrfs_run_delayed_refs:2151: errno=-28 No space left
>>> kernel: [ 1919.454249] BTRFS warning (device sdc: state EA):
>>> btrfs_uuid_scan_kthread failed -5
>>> kernel: [ 1919.472381] BTRFS: error (device sdc: state EA) in
>>> __btrfs_free_extent:3077: errno=-28 No space left
>>> kernel: [ 1919.472417] BTRFS error (device sdc: state EA): failed to
>>> run delayed ref for logical 102538732191744 num_bytes 245760 type 184
>>> action 2 ref_mod 1: -28
>>> kernel: [ 1919.472442] BTRFS: error (device sdc: state EA) in
>>> btrfs_run_delayed_refs:2151: errno=-28 No space left
>>>
>>>
>>> On Sat, 17 Jun 2023 at 15:00, Qu Wenruo <wqu@suse.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2023/6/17 13:11, Stefan N wrote:
>>>>> Hi Qu,
>>>>>
>>>>> I believe I've got this environment ready, with the 6.2.0 kernel as
>>>>> before using the Ubuntu kernel, but can switch to vanilla if required.
>>>>>
>>>>> I've not done anything kernel modifications for a solid decade, so
>>>>> would be keen for a bit of guidance.
>>>>
>>>> Sure no problem.
>>>>
>>>> Please fetch the kernel source tar ball (6.2.x) first, decompress, then
>>>> apply the attached one-line patch by:
>>>>
>>>> $ tar czf linux*.tar.xz
>>>> $ cd linux*
>>>> $ patch -np1 -i <the patch file>
>>>>
>>>> Then use your running system kernel config if possible:
>>>>
>>>> $ cp /proc/config.gz .
>>>> $ gunzip config.gz
>>>> $ mv config .config
>>>> $ make olddefconfig
>>>>
>>>> Then you can start your kernel compiling, and considering you're using
>>>> your distro's default, it would include tons of drivers, thus would be
>>>> very slow. (Replace the number to something more suitable to your
>>>> system, using all CPU cores can be very hot)
>>>>
>>>> $ make -j12
>>>>
>>>> Finally you need to install the modules/kernel.
>>>>
>>>> Unfortunately this is distro specific, but if you're using Ubuntu, it
>>>> may be much easier:
>>>>
>>>> $ make bindeb-pkg
>>>>
>>>> Then install the generated dpkg I guess? I have never tried kernel
>>>> building using deb/rpm, but only manual installation, which is also
>>>> distro dependent in the initramfs generation part.
>>>>
>>>> # cp arch/x86/boot/bzImage /boot/vmlinuz-custom
>>>> # make modules_install
>>>> # mkinitcpio -k /boot/vmlinuz-custom -g /boot/initramfs-custom.img
>>>>
>>>>
>>>> The last step is to update your bootloader to add the new kernel, which
>>>> is not only distro dependent but also bootloader dependent.
>>>>
>>>> In my case, I go with systemd-boot with manually crafted entries.
>>>> But if you go Ubuntu I believe just installing the kernel dpkg would
>>>> have everything handled?
>>>>
>>>> Finally you can try reboot into the newer kernel, and try device add
>>>> (need to add 4 disks), then sync and see if things work as expected.
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>> I will recover a 1tb SSD and partition it into 4 in a USB enclosure,
>>>>> but failing this will use 4x loop devices.
>>>>>
>>>>> On Tue, 13 Jun 2023 at 11:28, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>> In your particular case, since you're running RAID1C4 you need to add 4
>>>>>> devices in one transaction.
>>>>>>
>>>>>> I can easily craft a patch to avoid commit transaction, but still you'll
>>>>>> need to add at least 4 disks, and then sync to see if things would work.
>>>>>>
>>>>>> Furthermore this means you need a liveCD with full kernel compiling
>>>>>> environment.
>>>>>>
>>>>>> If you want to go this path, I can send you the patch when you've
>>>>>> prepared the needed environment.