Re: status of bugzilla #99171 - mdraid broken for O_DIRECT

[Roland <devzero@web.de>]

hello,

some late reply for this...

 > Am 10.10.24 um 10:34 schrieb Hannes Reinecke:
 > Which I really would love
 > to see reproduced, especially with recent kernels, as there is a lot of
 > vagueness around it (add part of the disk on the raid as swap? How?
 > In the host? On the guest?).

here is a reproducer everybody should be able to follow/reproduce.

1. install proxmox pve9 on a system with two empty disks for mdraid. 
build mdraid and format with ext4 .

2. add that ext4 mountpoint as a datastore type "dir" for file/vm 
storage in proxmox.

3. install a debian13 in a normal/default (cache=none, i.e. O_DIRECT = 
on)  linux VM. the virtual disk should be backed by that mdraid/ext4 
datastore created above.

4. inside the vm as an ordinary user get break-raid-odirect.c from 
https://forum.proxmox.com/threads/mdraid-o_direct.156036/post-713543 , 
compile that and let that run for a while. then terminate with ctrl-c.

5. on the pve host, check if your raid did not throw any error or has 
mismatch_count >0 ( cat /sys/block/md127/md/mismatch_cnt ) in the meantime.

6. on the pve host start raid check with  "echo check > 
/sys/block/md127/md/sync_action"

6. let that check run and wait until it finishes (/proc/mdstat)

7. check for inconsistencies  via "cat /sys/block/md127/md/mismatch_cnt" 
again

i am getting:

cat /sys/block/md127/md/mismatch_cnt
1048832

so , we see that even with recent kernel (pve9 kernel is 6.14 based on 
ubuntu kernel),  we can break mdraid from non-root user inside a qemu VM 
on top ext4 on top of mdraid.

roland




Am 10.10.24 um 10:34 schrieb Hannes Reinecke:
> On 10/10/24 09:29, Roland wrote:
>> thank you for clearing things up.
>>
>>  >Which means that the test case is actually invalid; you either would
>> need drop O_DIRECT or modify the buffer
>>  >after write() to arrive with a valid example.
>>
>> ok, but what about running virtual machines in O_DIRECT mode on top of
>> mdraid then ?
>>
>> https://forum.proxmox.com/threads/zfs-on-debian-or-mdadm-softraid- 
>> stability-and-reliability-of-zfs.116871/post-505697
>>
>
> The example quoted is this:
> > Take a virtual machine, give it a disk - put the image on a software
> > raid and tell qemu to disable caching (iow. use O_DIRECT, because the
> > guest already does caching anyway).
> > Run linux in the VM, add part of the/a disk on the raid as swap, and
> > cause the guest to start swapping a lot.
>
> And then ending up with data corruption on MD. Which I really would love
> to see reproduced, especially with recent kernels, as there is a lot of
> vagueness around it (add part of the disk on the raid as swap? How?
> In the host? On the guest?).
>
> Hint: we (SUSE) have a bugzilla.suse.com. And if someone would be 
> reproducing that with, say, OpenSUSE Tumbleweed and open a bugzilla
> someone on this list would be more than happy to have a look and do
> a proper debugging here. There are a lot of things which have changed
> since 2017 (Stable pages? Anyone?), so it might be that the cited issue
> simply is not reproducible anymore.
>
> Cheers,
>
> Hannes