From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pete Ashdown Subject: Re: kvm + raid1 showstopper bug Date: Sun, 19 Feb 2012 11:17:43 -0700 Message-ID: <4F413CC7.1020904@xmission.com> References: <20120217045733.GC31397@xmission.com> <4F3E72B5.6030502@xmission.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org, Aaron Toponce To: Stefan Hajnoczi Return-path: Received: from out02.mta.xmission.com ([166.70.13.232]:45890 "EHLO out02.mta.xmission.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754483Ab2BSSRl (ORCPT ); Sun, 19 Feb 2012 13:17:41 -0500 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On 2/18/12 6:25 AM, Stefan Hajnoczi wrote: >> In my case, it is drbd+RAID10, but the bug still applies. It isn't >> whenever checkarray runs, but whenever checkarray decides to do a resync, >> it will block all IO somewhere before the end of the resync. Then yes, it >> isn't long before the guests start to fail due to their inability to >> read/write. > I have not attempted to reproduce this yet but have taken a look at > drviers/md/raid10.c resync code. md resync uses a similar mechanism > for RAID1 and RAID10. While a block is being synced the entire device > will force regular I/O requests to wait. There are tunables which let > you rate-limit resyncing, I think this can solve your problem. > Perhaps the resync is too aggressive and is impacting regular I/O so > much that the guest is warning about it. See Documentation/md.txt for > sync_speed_max and other sysfs attributes. Is sync_speed_max independent of dev.raid.speed_limit_max? Because I tried that to no avail. > The bug report suggests qemu-kvm itself is operating fine because the > guest is still executing and VNC/monitor are alive. After a while the > guest warns about the stuck I/O. > > Networking may become unresponsive if there is disk I/O required, e.g. > ssh daemon reading keys for a user. Your best bet at testing that > theory is using ICMP ping because that shouldn't involve disk I/O. > > It would be interesting to start resync and then run the following on > the host: time dd if=/dev/zero of=/path/to/device/tmpfile oflag=sync > bs=4k count=1. You don't even need qemu-kvm for this test. I suspect > this single 4 KB write to the file system will take many > seconds/minutes. It would show that the problem is in the host - > there is too little time for regular I/O which causes guest operating > systems and applications to freak out. > > Another approach to testing is running a guest without RAID resync > underneath. Use dm-delay to insert an artificial delay on I/O > requests (try 130 seconds). My guess is the guest operating system > will react in the same way because its I/O requests take an extremely > long time. > I will try these other tests when I get a chance.