Re: RAID mount fails after upgrading to kernel 6.2.0

[Eric Levy <contact@ericlevy.name>]

On Fri, Jul 21 2023 at 12:49:13 PM +0800, Anand Jain 
<anand.jain@oracle.com> wrote:
> 
> 
> On 21/7/23 04:10, Eric Levy wrote:
>> 
>> 
>> On Thu, Jul 20 2023 at 01:48:24 PM +0800, Anand Jain 
>> \x7f<anand.jain@oracle.com> wrote:
>>> 
>>> 
>>> On 20/07/2023 10:50, Eric Levy wrote:
>>>> 
>>>> 
>>>> On Thu, Jul 20 2023 at 10:26:57 AM +0800, Anand Jain 
>>>> \x7f\x7f\x7f\x7f<anand.jain@oracle.com> wrote:
>>>>> On 20/07/2023 09:13, Eric Levy wrote:
>>>>>> I recently performed a routine update on a Linux Mint system, 
>>>>>> \x7f\x7f\x7f\x7f\x7fversion \x7f\x7f\x7f\x7f21.2 (Victoria). The update moved the kernel 
>>>>>> from \x7f\x7f\x7f\x7f\x7f5.19.0 to 6.2.0. \x7f\x7f\x7fThe \x7fsystem includes a non-root 
>>>>>> mount that is \x7f\x7f\x7f\x7f\x7fBtrfs with RAID, which \x7f\x7f\x7fno \x7flonger mounts. 
>>>>>> Error reporting is \x7f\x7f\x7f\x7f\x7frather limited and opaque.
>>>>>> 
>>>>>> I am assuming the file system is healthy from the standpoint of 
>>>>>> the \x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7fold \x7fkernel, but I may need help understanding how 
>>>>>> to make it \x7f\x7f\x7f\x7f\x7fviable \x7f\x7f\x7ffor the \x7fnew one.
>>>>>> 
>>>>>> Mounting from the command line prints the following:
>>>>>> 
>>>>>> mount: /mnt: wrong fs type, bad option, bad superblock on 
>>>>>> /dev/sdg, \x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7fmissing codepage or helper program, or other 
>>>>>> error.
>>>>>> 
>>>>>> The following is extracted from the boot sequence recorded in 
>>>>>> the \x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7fkernel \x7fring:
>>>>>> 
>>>>>> kernel: BTRFS error: device /dev/sdd belongs to fsid 
>>>>>> \x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7fc6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is 
>>>>>> already \x7f\x7f\x7f\x7f\x7fmounted
>>>>>> kernel: BTRFS error: device /dev/sdf belongs to fsid 
>>>>>> \x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7fc6f83d24-1ac3-4417-bdd9-6249c899604d, and the fs is 
>>>>>> already \x7f\x7f\x7f\x7f\x7fmounted
>>>>>> kernel: BTRFS info (device sde): using crc32c (crc32c-intel) 
>>>>>> \x7f\x7f\x7f\x7f\x7fchecksum \x7f\x7f\x7f\x7falgorithm
>>>>>> kernel: BTRFS info (device sde): turning on async discard
>>>>>> kernel: BTRFS info (device sde): disk space caching is enabled
>>>>>> kernel: BTRFS error (device sde): devid 7 uuid 
>>>>>> \x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f2f62547b-067f-433c-bec1-b90e0c8cb75e is missing
>>>>>> kernel: BTRFS error (device sde): failed to read the system 
>>>>>> array: -2
>>>>>> kernel: BTRFS error (device sde): open_ctree failed
>>>>>> mount[969]: mount: /mnt: wrong fs type, bad option, bad 
>>>>>> superblock \x7f\x7f\x7f\x7f\x7fon \x7f\x7f\x7f\x7f/dev/sde, missing codepage or helper 
>>>>>> program, or other error.
>>>>>> systemd[1]: mnt.mount: Mount process exited, code=exited, 
>>>>>> \x7f\x7f\x7f\x7f\x7fstatus=32/n/a
>>>>> 
>>>>> 
>>>>> Looks like the fsid is already mounted. Could you please help 
>>>>> check?
>>>>> 
>>>>>     cat /proc/self/mounts | grep btrfs
>>>>> 
>>>>> You could try a fresh scan and mount.
>>>>> 
>>>>>     umount  ..
>>>>>     btrfs device scan
>>>>>     mount ...
>>>>> 
>>>>> If this doesn't help. Can you share the output of:
>>>>> 
>>>>>     btrfs filesystem dump-super /dev/sd[a-g]  <-- basically all 
>>>>> devices
>>>>> 
>>>>> Thanks.
>>>> 
>>>> 
>>>> The unmount command followed by rescan does enable a successful 
>>>> \x7f\x7f\x7fmount, \x7fbut the suggestion that the volume was mounted already 
>>>> had \x7f\x7f\x7fnot been \x7fvalidated by the dump of the mount table. Based 
>>>> on the \x7f\x7f\x7fmount table, the \x7fvolume appeared as unmounted even 
>>>> before the command.
>>>> 
>>>> Do you have any suggestions for how to resolve why the volume 
>>>> would \x7f\x7f\x7fbe \x7fregistered as having been mounted?
>>> 
>>>  As mentioned, dump-super might help.
>> 
>> 
>> After further investigation, I believe the issue is not particularly 
>> \x7frelated to the kernel or the filesystem.
>> 
>> I believe that systemd is attempting to mount a volume before all of 
>> the \x7fdevices are attached through the iSCSI login process.
>> 
>> The issue may be outside the scope of Btrfs, but I certainly would 
>> \x7fappreciate any suggestions.
>> 
>> How can systemd be forced to wait, before attempting to mount, until 
>> all \x7funits in the volume, identified by an UUID, have been 
>> successfully \x7fattached?
> 
>  Thanks for narrowing it down. Please check if you have the btrfs 
> kernel commit:
> 
>    50d281fc434c btrfs: scan device in non-exclusive mode
> 
> This commit addresses a bug that was caused by interference from
> systemd during the device scan/mount process.

I don't know how to check for a specific commit.

I am running Linux Mint, which is based on regular updates for Ubuntu 
LTS.

The current kernel is Ubuntu mainline 6.2.0-25-generic for x86_64.