Re: kvm + raid1 showstopper bug

[Pete Ashdown <pashdown@xmission.com>]

On 2/18/12 6:25 AM, Stefan Hajnoczi wrote:
>> In my case, it is drbd+RAID10, but the bug still applies.  It isn't
>> whenever checkarray runs, but whenever checkarray decides to do a resync,
>> it will block all IO somewhere before the end of the resync.  Then yes, it
>> isn't long before the guests start to fail due to their inability to
>> read/write.
> I have not attempted to reproduce this yet but have taken a look at
> drviers/md/raid10.c resync code.  md resync uses a similar mechanism
> for RAID1 and RAID10.  While a block is being synced the entire device
> will force regular I/O requests to wait.  There are tunables which let
> you rate-limit resyncing, I think this can solve your problem.
> Perhaps the resync is too aggressive and is impacting regular I/O so
> much that the guest is warning about it.  See Documentation/md.txt for
> sync_speed_max and other sysfs attributes.

Is sync_speed_max independent of dev.raid.speed_limit_max?  Because I
tried that to no avail.

> The bug report suggests qemu-kvm itself is operating fine because the
> guest is still executing and VNC/monitor are alive.  After a while the
> guest warns about the stuck I/O.
>
> Networking may become unresponsive if there is disk I/O required, e.g.
> ssh daemon reading keys for a user.  Your best bet at testing that
> theory is using ICMP ping because that shouldn't involve disk I/O.
>
> It would be interesting to start resync and then run the following on
> the host: time dd if=/dev/zero of=/path/to/device/tmpfile oflag=sync
> bs=4k count=1.  You don't even need qemu-kvm for this test.  I suspect
> this single 4 KB write to the file system will take many
> seconds/minutes.  It would show that the problem is in the host -
> there is too little time for regular I/O which causes guest operating
> systems and applications to freak out.
>
> Another approach to testing is running a guest without RAID resync
> underneath.  Use dm-delay to insert an artificial delay on I/O
> requests (try 130 seconds).  My guess is the guest operating system
> will react in the same way because its I/O requests take an extremely
> long time.
>
I will try these other tests when I get a chance.