Re: kvm + raid1 showstopper bug

Stefan Hajnoczi
2012-02-18T13:25:41Z

On Fri, Feb 17, 2012 at 3:31 PM, Pete Ashdown <pashdown@xmission.com> wrote:
> On 02/17/2012 04:30 AM, Stefan Hajnoczi wrote:
>> On Fri, Feb 17, 2012 at 4:57 AM, Pete Ashdown <pashdown@xmission.com> wrote:
>>> I've been waiting for some response from the Ubuntu team regarding a bug on
>>> launchpad, but it appears that it isn't being taken seriously:
>>>
>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785
>> This looks interesting.  Let me try to summarize, please point out if
>> I get something wrong:
>>
>> You have software RAID1 on the host, your disk images live on this
>> device.  Whenever checkarray runs on the host you find that VMs become
>> unresponsive.  Guests print warnings that a task is blocked for more
>> than 120 seconds.  Guests become unresponsive on the network.
> In my case, it is drbd+RAID10, but the bug still applies.  It isn't
> whenever checkarray runs, but whenever checkarray decides to do a resync,
> it will block all IO somewhere before the end of the resync.  Then yes, it
> isn't long before the guests start to fail due to their inability to
> read/write.

I have not attempted to reproduce this yet but have taken a look at
drviers/md/raid10.c resync code. md resync uses a similar mechanism
for RAID1 and RAID10. While a block is being synced the entire device
will force regular I/O requests to wait. There are tunables which let
you rate-limit resyncing, I think this can solve your problem.
Perhaps the resync is too aggressive and is impacting regular I/O so
much that the guest is warning about it. See Documentation/md.txt for
sync_speed_max and other sysfs attributes.

The bug report suggests qemu-kvm itself is operating fine because the
guest is still executing and VNC/monitor are alive. After a while the
guest warns about the stuck I/O.

Networking may become unresponsive if there is disk I/O required, e.g.
ssh daemon reading keys for a user. Your best bet at testing that
theory is using ICMP ping because that shouldn't involve disk I/O.

It would be interesting to start resync and then run the following on
the host: time dd if=/dev/zero of=/path/to/device/tmpfile oflag=sync
bs=4k count=1. You don't even need qemu-kvm for this test. I suspect
this single 4 KB write to the file system will take many
seconds/minutes. It would show that the problem is in the host -
there is too little time for regular I/O which causes guest operating
systems and applications to freak out.

Another approach to testing is running a guest without RAID resync
underneath. Use dm-delay to insert an artificial delay on I/O
requests (try 130 seconds). My guess is the guest operating system
will react in the same way because its I/O requests take an extremely
long time.

This may be dependent on hardware. I have used RAID1 to host disk
images with Xen and KVM and never noticed an issue.

Stefan