Reading the same block via partition and non-partitioned device gives different content

Reading the same block via partition and non-partitioned device gives different content

[Andrei Borzenkov]

[Please Cc me on reply; thank you]

QEMU KVM virtual machine with openSUSE Tumbleweed (kernel
4.3.3-3-default); MD RAID1 with 1.2 metadata on /dev/vdb1 and /dev/vdc1.

If I do

mdadm /dev/mdX --fail /dev/vdb1
mdadm /dev/mdX --add /dev/vdd1

and wait for synchronization to finish and then look directly on on-disk
suportblock, I see different content whether I read from /dev/vdd or
from /dev/vdd1.

/dev/vdd1 claims new disk is still spare (i.e. it is apparently
immediately after adding it).

The *same* superblock when read from /dev/vdd (of course with
appropriate offset) correctly marks /dev/vdd1 as RAID member. The same
content also seen when looking from host.

I hit this when debugging problem with grub2 that scans devices for
Linux MD (it is using the same code both at boot and run time). It
skipped replacement disk because it believed disk was spare.

Is it expected behavior? Note that if we now replace /dev/vdc1 with
something else we have "wrong" superblock on both partitions so grub
fails to find array completely. Fortunately this is transient state, but
it also means it is impossible to reconfigure grub until reboot.
Alternatively shutting down and restarting array cleats it as well.

Any trick we can use to force content to be the same in this state?

-andrei

Re: Reading the same block via partition and non-partitioned device gives different content

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On Tue, Jan 05 2016, Andrei Borzenkov wrote:

> [Please Cc me on reply; thank you]
>
> QEMU KVM virtual machine with openSUSE Tumbleweed (kernel
> 4.3.3-3-default); MD RAID1 with 1.2 metadata on /dev/vdb1 and /dev/vdc1.
>
> If I do
>
> mdadm /dev/mdX --fail /dev/vdb1
> mdadm /dev/mdX --add /dev/vdd1
>
> and wait for synchronization to finish and then look directly on on-disk
> suportblock, I see different content whether I read from /dev/vdd or
> from /dev/vdd1.
>
> /dev/vdd1 claims new disk is still spare (i.e. it is apparently
> immediately after adding it).
>
> The *same* superblock when read from /dev/vdd (of course with
> appropriate offset) correctly marks /dev/vdd1 as RAID member. The same
> content also seen when looking from host.
>
> I hit this when debugging problem with grub2 that scans devices for
> Linux MD (it is using the same code both at boot and run time). It
> skipped replacement disk because it believed disk was spare.
>
> Is it expected behavior? Note that if we now replace /dev/vdc1 with
> something else we have "wrong" superblock on both partitions so grub
> fails to find array completely. Fortunately this is transient state, but
> it also means it is impossible to reconfigure grub until reboot.
> Alternatively shutting down and restarting array cleats it as well.
>
> Any trick we can use to force content to be the same in this state?

blockdev --flushbufs /dev/vdd*

or use O_DIRECT to access the devices.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]