Re: [opensuse] raid device mounting problem in Leap 42.2

Re: [opensuse] raid device mounting problem in Leap 42.2

[Wols Lists]

On 13/07/17 23:46, Istvan Gabor wrote:
> On Wed, 12 Jul 2017 17:05:17 +0300, Andrei Borzenkov wrote:
>> On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists <antlists@youngman.org.uk>
>> wrote:
>>>
>>> But I suspect that going back into 12.2, and shrinking the filesystem
>>> (MAKE SURE you shrink the md device, not the sd device)
>>
>> Quite the opposite. Attempting to shrink filesystem that is already
>> effectively corrupted is potentially dangerous. So the right thing
>> here is to stop md, backup superblocks (in case they will be
> 
> I am puzzled. Do I need to backup the raid superblock or the device
> superblocks or all? How do I backup the superblock? I googled but only
> find how to recover from superblock backup.
> 
The best bet is to ask the linux-raid list what to do.

For the raid list, this is the array I mentioned where the filesystem,
and the partitions the raid was on, were the same size.

So we have a v1.0 mirror where the filesystem on the mirror uses up ALL
the space on the md device, not just the free space that it's supposed
to use, with all the issues that brings like filling up the filesystem
will overwrite the superblock.

So what's the best way to recover? Will just shrinking the filesystem
bring everything back the way it should be, or are we better just
mounting the original filesystem from one disk as if it weren't a
mirror, and then recreating the mirror on the other disk, copying the
data, and then adding the first disk back in to put everything back the
way it should have been?

Cheers,
Wol

Re: [opensuse] raid device mounting problem in Leap 42.2

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

On Fri, Jul 14 2017, Wols Lists wrote:

> On 13/07/17 23:46, Istvan Gabor wrote:
>> On Wed, 12 Jul 2017 17:05:17 +0300, Andrei Borzenkov wrote:
>>> On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists <antlists@youngman.org.uk>
>>> wrote:
>>>>
>>>> But I suspect that going back into 12.2, and shrinking the filesystem
>>>> (MAKE SURE you shrink the md device, not the sd device)
>>>
>>> Quite the opposite. Attempting to shrink filesystem that is already
>>> effectively corrupted is potentially dangerous. So the right thing
>>> here is to stop md, backup superblocks (in case they will be
>> 
>> I am puzzled. Do I need to backup the raid superblock or the device
>> superblocks or all?

You cannot have too many backups.

>>                      How do I backup the superblock? I googled but only

mdadm --dump=/some/directory list of devices.

>> find how to recover from superblock backup.
>> 
> The best bet is to ask the linux-raid list what to do.
>
> For the raid list, this is the array I mentioned where the filesystem,
> and the partitions the raid was on, were the same size.

To the byte?  I really like to see the output of commands that report
sizes, rather than have someone claim "were the same size".  It isn't
that I don't trust you.  I just don't trust humans in general (myself included).

NeilBrown

>
> So we have a v1.0 mirror where the filesystem on the mirror uses up ALL
> the space on the md device, not just the free space that it's supposed
> to use, with all the issues that brings like filling up the filesystem
> will overwrite the superblock.
>
> So what's the best way to recover? Will just shrinking the filesystem
> bring everything back the way it should be, or are we better just
> mounting the original filesystem from one disk as if it weren't a
> mirror, and then recreating the mirror on the other disk, copying the
> data, and then adding the first disk back in to put everything back the
> way it should have been?
>
> Cheers,
> Wol
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [opensuse] raid device mounting problem in Leap 42.2

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 5192 bytes --]

On Thu, Jul 13 2017, David C. Rankin wrote:

> On 07/13/2017 08:40 PM, NeilBrown wrote:
>
>> 
>> To the byte?  I really like to see the output of commands that report
>> sizes, rather than have someone claim "were the same size".  It isn't
>> that I don't trust you.  I just don't trust humans in general (myself included).
>> 
>> NeilBrown
>> 
>
> Yep,
>
>   Earlier in the post we have:

Thanks.  (I'm not subscribed to opensuse@opensuse.org, so I couldn't see
any of this before.  Also, people on that list might not see my reply.
Feel free to forward this email to that list.
I'm Ccing to linux-raid as that is the list that I first heard about
this on).

>
> <quote>
>
> It is assembled from /dev/sdb9 and /dev/sdc9.
>
> # sfdisk -l /dev/sdb
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Device     Boot     Start       End   Sectors   Size Id Type
> /dev/sdb9       298005813 360916289  62910477    30G fd Linux raid autodetect
>
> # sfdisk -l /dev/sdc
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Device     Boot     Start       End   Sectors   Size Id Type
> /dev/sdc9       298005813 360916289  62910477    30G fd Linux raid autodetect

Device is 62910477 sectors.  So 29.998053 GiB or 32.210164 GB


>
> </quote>
>
> and the following at the beginning of the thread
>
> <quote>
>
>
> I have several mdraid RAID1 (mirror) devices I used without
> problem in openSUSE 12.2. In openSUSE Leap 42.2 I can't
> mount some of the same raid devices.
>
> In openSUSE 12.2 I can mount the raid device:
>
> cat /proc/mdstat
>
> md9 : active raid1 sdc9[1] sdb9[0]
>       31455164 blocks super 1.0 [2/2] [UU]

Array is 31455164 KiB, so 29.997982 GiB or 32.210087 GB.

>
> # mount /dev/md9 /mnt -o ro
> #
>
> # df -h
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/md9         30G   28G  364M  99% /mnt
>

"-h" means to use powers of 1024, so this "30G" compares with
the array size of "29.997982 GiB".

I don't know exactly how 'df' formats numbers (and reading the code
makes my head spin), but I wouldn't be surprised if it chose to print
29.997 as 30.

Can we see the output of "df" without the "-h" ('h' stands for 'human'
and we know I don't trust those :-).

>
>
> In openSUSE Leap 42.2 I can't mount the same raid device:
>
> cat /proc/mdstat
>
> md9 : active raid1 sdc9[1] sdb9[0]
>       31455164 blocks super 1.0 [2/2] [UU]
>
>
> # mount /dev/md9 /mnt -o ro
> mount: wrong fs type, bad option, bad superblock on /dev/md9,
>        missing codepage or helper program, or other error
>
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.

Has there been any report of the output of this "dmesg | tail" command?

>
>
> Why is this and how can I fix it?
>
> </quote>
>
> Glad to see we have a direct pipeline to the master here :)

So I went hunting in the archives (I shouldn't have to..) and found:

> The filesystem size (according to the superblock) is 7863809 blocks
> The physical size of the device is 7863791 blocks

This would be 4K blocks, so
  filesystem: 7863809 blocks, 31455236K, 62910472 sectors
  device:     7863791 blocks, 31455164K, 62910328 sectors

So the sd[bc]9 is 5 sectors larger than the filesystem.

Hmm. opensuse 12.2... That had a 3.4.6 kernel?

3.4 (and up to 4.3) had separate ext3 and ext4 implementations.
In 4.3, ext3 was discarded and ext4 used to manage ext3 filesystems.
So small changes in behaviour for corner cases are not impossible.

ext4 has a test :
	/* check blocks count against device size */
	blocks_count = sb->s_bdev->bd_inode->i_size >> sb->s_blocksize_bits;
	if (blocks_count && ext4_blocks_count(es) > blocks_count) {
		ext4_msg(sb, KERN_WARNING, "bad geometry: block count %llu "
		       "exceeds size of device (%llu blocks)",
		       ext4_blocks_count(es), blocks_count);
		goto failed_mount;
	}

which is presumably failing.  If "dmesg | tail" had been run, we
probably would have seen that.
This test has been present since 2.6.30.
I don't think ext3 ever got this test (until it became ext4).

So there is your answer.
The filesystem was created on the member device, instead of on the
array.
The kernel being used didn't check for this inconsistency.
fsck does, but presumably was never run.
The new kernel does check - as it should.

I would suggest:
1/ stop the array (mdadm -S /dev/md9)
2/ run fsck --force /dev/sdb9
3/ run resize2fs /dev/sdb9 29G  (or something like that)
4/ "mdadm -C /dev/md9 -l1 -e 1.0 -n2 /dev/sdb9 missing"
5/ "fsck /dev/md9" - if there are errors, stop here.
6/ "resize2fs /dev/md9"  (which will make use of all available space)
 only continue if you are really sure /dev/md9 looks good
7/ mdadm --zero /dev/sdc9
8/ mdadm /dev/md9 --add /dev/sdc9
9/ wait for resync to complete.

NeilBrown

>
>
> -- 
> David C. Rankin, J.D.,P.E.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]