kvm + raid1 showstopper bug

kvm + raid1 showstopper bug

[Pete Ashdown]

I've been waiting for some response from the Ubuntu team regarding a bug on
launchpad, but it appears that it isn't being taken seriously:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785

We are running Ubuntu Natty 11.04 with KVM with our storage on RAID10 + drbd.
While we were running the resync command, we could expect an IO block during
every resync.  I eventually disabled resync.  Yesterday we had a kernel panic,
which I am presuming may trace back to KVM and its interactions with RAID10.
I run KVM on some other boxes with some built-in Dell hardware RAIDs.  They
have been stable for well over a year on Ubuntu 10.04.

Here are the 11.04 current versions if that helps.

kvm  	     84+dfsg-0ubuntu16+0.14.0+noroms+0ubuntu4.4
kvm-pxe      5.4.4-7ubuntu2
qemu-kvm     0.14.0+noroms-0ubuntu4.4
drbd8-utils  8.3.9-1ubuntu1

What I'd like to know is if this bug between KVM & RAID1 is known and whether
it has been addressed in newer versions of the kernel and/or KVM.  I'm on the
verge of buying a hardware RAID card to resolve this and I'd rather not have
to deal with that.  Software RAID is so much nicer to manage.

Re: kvm + raid1 showstopper bug

[Stefan Hajnoczi]

On Fri, Feb 17, 2012 at 4:57 AM, Pete Ashdown <pashdown@xmission.com> wrote:
> I've been waiting for some response from the Ubuntu team regarding a bug on
> launchpad, but it appears that it isn't being taken seriously:
>
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785

This looks interesting.  Let me try to summarize, please point out if
I get something wrong:

You have software RAID1 on the host, your disk images live on this
device.  Whenever checkarray runs on the host you find that VMs become
unresponsive.  Guests print warnings that a task is blocked for more
than 120 seconds.  Guests become unresponsive on the network.

The fact that the QEMU monitor and VNC still work mean that QEMU is
not probably still running the VM.  I think the guest kernel is upset,
perhaps QEMU needs to do something to help these I/Os along.

Please post your qemu-kvm command-line or libvirt domain XML.

What's the easiest way to reproduce this?

Stefan

Re: kvm + raid1 showstopper bug

[Pete Ashdown]

Thank you for your response Stefan.

On 02/17/2012 04:30 AM, Stefan Hajnoczi wrote:
> On Fri, Feb 17, 2012 at 4:57 AM, Pete Ashdown <pashdown@xmission.com> wrote:
>> I've been waiting for some response from the Ubuntu team regarding a bug on
>> launchpad, but it appears that it isn't being taken seriously:
>>
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785
> This looks interesting.  Let me try to summarize, please point out if
> I get something wrong:
>
> You have software RAID1 on the host, your disk images live on this
> device.  Whenever checkarray runs on the host you find that VMs become
> unresponsive.  Guests print warnings that a task is blocked for more
> than 120 seconds.  Guests become unresponsive on the network.
In my case, it is drbd+RAID10, but the bug still applies.  It isn't
whenever checkarray runs, but whenever checkarray decides to do a resync,
it will block all IO somewhere before the end of the resync.  Then yes, it
isn't long before the guests start to fail due to their inability to
read/write.

> The fact that the QEMU monitor and VNC still work mean that QEMU is
> not probably still running the VM.  I think the guest kernel is upset,
> perhaps QEMU needs to do something to help these I/Os along.

Note that *ALL* IO is blocked, even on the host kernel.  It has trouble
rebooting at that point too.  I have to power cycle it.

> Please post your qemu-kvm command-line or libvirt domain XML.
>

<domain type='kvm'>
  <name>guestname</name>
  <uuid>c4cb4999-0713-dffa-32f8-1bb7278b3f5c</uuid>
  <memory>8388608</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vm/guestname'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </disk>
    <interface type='bridge'>
      <mac address='52:54:00:ca:7b:70'/>
      <source bridge='br30'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
    </memballoon>
  </devices>
</domain>


> What's the easiest way to reproduce this?

Based on the Launchpad bug, Ubuntu 11.04 with default packages +
RAID1/RAID10 I think would have the issue eventually.  I don't think it has
to be a particularly IO intensive system.

Re: kvm + raid1 showstopper bug

[Stefan Hajnoczi]

On Fri, Feb 17, 2012 at 3:31 PM, Pete Ashdown <pashdown@xmission.com> wrote:
> On 02/17/2012 04:30 AM, Stefan Hajnoczi wrote:
>> On Fri, Feb 17, 2012 at 4:57 AM, Pete Ashdown <pashdown@xmission.com> wrote:
>>> I've been waiting for some response from the Ubuntu team regarding a bug on
>>> launchpad, but it appears that it isn't being taken seriously:
>>>
>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785
>> This looks interesting.  Let me try to summarize, please point out if
>> I get something wrong:
>>
>> You have software RAID1 on the host, your disk images live on this
>> device.  Whenever checkarray runs on the host you find that VMs become
>> unresponsive.  Guests print warnings that a task is blocked for more
>> than 120 seconds.  Guests become unresponsive on the network.
> In my case, it is drbd+RAID10, but the bug still applies.  It isn't
> whenever checkarray runs, but whenever checkarray decides to do a resync,
> it will block all IO somewhere before the end of the resync.  Then yes, it
> isn't long before the guests start to fail due to their inability to
> read/write.

I have not attempted to reproduce this yet but have taken a look at
drviers/md/raid10.c resync code.  md resync uses a similar mechanism
for RAID1 and RAID10.  While a block is being synced the entire device
will force regular I/O requests to wait.  There are tunables which let
you rate-limit resyncing, I think this can solve your problem.
Perhaps the resync is too aggressive and is impacting regular I/O so
much that the guest is warning about it.  See Documentation/md.txt for
sync_speed_max and other sysfs attributes.

The bug report suggests qemu-kvm itself is operating fine because the
guest is still executing and VNC/monitor are alive.  After a while the
guest warns about the stuck I/O.

Networking may become unresponsive if there is disk I/O required, e.g.
ssh daemon reading keys for a user.  Your best bet at testing that
theory is using ICMP ping because that shouldn't involve disk I/O.

It would be interesting to start resync and then run the following on
the host: time dd if=/dev/zero of=/path/to/device/tmpfile oflag=sync
bs=4k count=1.  You don't even need qemu-kvm for this test.  I suspect
this single 4 KB write to the file system will take many
seconds/minutes.  It would show that the problem is in the host -
there is too little time for regular I/O which causes guest operating
systems and applications to freak out.

Another approach to testing is running a guest without RAID resync
underneath.  Use dm-delay to insert an artificial delay on I/O
requests (try 130 seconds).  My guess is the guest operating system
will react in the same way because its I/O requests take an extremely
long time.

This may be dependent on hardware.  I have used RAID1 to host disk
images with Xen and KVM and never noticed an issue.

Stefan

Re: kvm + raid1 showstopper bug

[Pete Ashdown]

On 2/18/12 6:25 AM, Stefan Hajnoczi wrote:
>> In my case, it is drbd+RAID10, but the bug still applies.  It isn't
>> whenever checkarray runs, but whenever checkarray decides to do a resync,
>> it will block all IO somewhere before the end of the resync.  Then yes, it
>> isn't long before the guests start to fail due to their inability to
>> read/write.
> I have not attempted to reproduce this yet but have taken a look at
> drviers/md/raid10.c resync code.  md resync uses a similar mechanism
> for RAID1 and RAID10.  While a block is being synced the entire device
> will force regular I/O requests to wait.  There are tunables which let
> you rate-limit resyncing, I think this can solve your problem.
> Perhaps the resync is too aggressive and is impacting regular I/O so
> much that the guest is warning about it.  See Documentation/md.txt for
> sync_speed_max and other sysfs attributes.

Is sync_speed_max independent of dev.raid.speed_limit_max?  Because I
tried that to no avail.

> The bug report suggests qemu-kvm itself is operating fine because the
> guest is still executing and VNC/monitor are alive.  After a while the
> guest warns about the stuck I/O.
>
> Networking may become unresponsive if there is disk I/O required, e.g.
> ssh daemon reading keys for a user.  Your best bet at testing that
> theory is using ICMP ping because that shouldn't involve disk I/O.
>
> It would be interesting to start resync and then run the following on
> the host: time dd if=/dev/zero of=/path/to/device/tmpfile oflag=sync
> bs=4k count=1.  You don't even need qemu-kvm for this test.  I suspect
> this single 4 KB write to the file system will take many
> seconds/minutes.  It would show that the problem is in the host -
> there is too little time for regular I/O which causes guest operating
> systems and applications to freak out.
>
> Another approach to testing is running a guest without RAID resync
> underneath.  Use dm-delay to insert an artificial delay on I/O
> requests (try 130 seconds).  My guess is the guest operating system
> will react in the same way because its I/O requests take an extremely
> long time.
>
I will try these other tests when I get a chance.

Re: kvm + raid1 showstopper bug

[Jes Sorensen]

On 02/18/12 14:25, Stefan Hajnoczi wrote:
> On Fri, Feb 17, 2012 at 3:31 PM, Pete Ashdown <pashdown@xmission.com> wrote:
>> > On 02/17/2012 04:30 AM, Stefan Hajnoczi wrote:
>>> >> On Fri, Feb 17, 2012 at 4:57 AM, Pete Ashdown <pashdown@xmission.com> wrote:
>>>> >>> I've been waiting for some response from the Ubuntu team regarding a bug on
>>>> >>> launchpad, but it appears that it isn't being taken seriously:
>>>> >>>
>>>> >>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785
>>> >> This looks interesting.  Let me try to summarize, please point out if
>>> >> I get something wrong:
>>> >>
>>> >> You have software RAID1 on the host, your disk images live on this
>>> >> device.  Whenever checkarray runs on the host you find that VMs become
>>> >> unresponsive.  Guests print warnings that a task is blocked for more
>>> >> than 120 seconds.  Guests become unresponsive on the network.
>> > In my case, it is drbd+RAID10, but the bug still applies.  It isn't
>> > whenever checkarray runs, but whenever checkarray decides to do a resync,
>> > it will block all IO somewhere before the end of the resync.  Then yes, it
>> > isn't long before the guests start to fail due to their inability to
>> > read/write.
> I have not attempted to reproduce this yet but have taken a look at
> drviers/md/raid10.c resync code.  md resync uses a similar mechanism
> for RAID1 and RAID10.  While a block is being synced the entire device
> will force regular I/O requests to wait.  There are tunables which let
> you rate-limit resyncing, I think this can solve your problem.
> Perhaps the resync is too aggressive and is impacting regular I/O so
> much that the guest is warning about it.  See Documentation/md.txt for
> sync_speed_max and other sysfs attributes.
> 
> The bug report suggests qemu-kvm itself is operating fine because the
> guest is still executing and VNC/monitor are alive.  After a while the
> guest warns about the stuck I/O.

It could be a bug in the raid1/raid10 code, which is triggered by the
way qemu keeps file on it (O_DIRECT or something). However there have
been a *lot* of fixes to the raid code since 2.6.38 (which is the kernel
version I saw referenced in the launchpad link). Please try and
reproduce it with a more uptodate kernel.

I have no idea what drbd8 is and what relation it has to the kernel or
kvm here.

Cheers,
Jes