From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Tosatti Subject: Re: Avoiding I/O bottlenecks between VM's Date: Fri, 19 Sep 2008 16:53:25 -0300 Message-ID: <20080919195325.GA15908@dmt.cnet> References: <200809191126.09889.alberto@byu.edu> <20080919184146.GA12928@dmt.cnet> <200809191253.30228.alberto@byu.edu> <90eb1dc70809191214k70f2377cr55bea42be9cbe02e@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Alberto =?iso-8859-1?Q?Trevi=F1o?= , "kvm@vger.kernel.org" To: Javier Guerra Return-path: Received: from mx1.redhat.com ([66.187.233.31]:48244 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752997AbYISTyc (ORCPT ); Fri, 19 Sep 2008 15:54:32 -0400 Content-Disposition: inline In-Reply-To: <90eb1dc70809191214k70f2377cr55bea42be9cbe02e@mail.gmail.com> Sender: kvm-owner@vger.kernel.org List-ID: On Fri, Sep 19, 2008 at 02:14:32PM -0500, Javier Guerra wrote: > On Fri, Sep 19, 2008 at 1:53 PM, Alberto Trevi=F1o = wrote: > > On Friday 19 September 2008 12:41:46 pm you wrote: > >> Are you using filesystem backed storage for the guest images or di= rect > >> block device storage? I assume there's heavy write activity on the > >> guests when these hangs happen? > > > > Yes, they happen when one VM is doing heavy writes. I'm actually u= sing a > > whole stack of things: > > > > OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-= encrypted > > partition. Fun debugging that, heh? Heh. Lots of variables there. > a not-so-wild guess might be the inter-node locking needed by any > cluster FS. you'd do much better using just CLVM or EVMS-Ha >=20 > if it's a single box, it would be interesting to compare with ext3 >=20 > > So, any ideas on how to solve the bottleneck? Isn't the CFQ schedu= ler > > supposed to grant every processes the same amount of I/O? =20 Yes, but if the filesystem on top is at fault, the IO scheduler can't help (this is the case with ext3 ordered mode and fsync latency, which could last for hundreds of seconds last time I checked). > > Is there a way to > > change something in proc to avoid this situation? >=20 > i don't think CFQ can do much to alleviate the heavy lock-dependency > of a cluster FS Perhaps isolate the problem by having the guest images directly on partitions first (or ext3 with writeback data mode).