Re: Avoiding I/O bottlenecks between VM's

[Marcelo Tosatti <mtosatti@redhat.com>]

On Fri, Sep 19, 2008 at 02:14:32PM -0500, Javier Guerra wrote:
> On Fri, Sep 19, 2008 at 1:53 PM, Alberto Treviño <alberto@byu.edu> wrote:
> > On Friday 19 September 2008 12:41:46 pm you wrote:
> >> Are you using filesystem backed storage for the guest images or direct
> >> block device storage? I assume there's heavy write activity on the
> >> guests when these hangs happen?
> >
> > Yes, they happen when one VM is doing heavy writes.  I'm actually using a
> > whole stack of things:
> >
> > OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted
> > partition.  Fun debugging that, heh?

Heh. Lots of variables there.

> a not-so-wild guess might be the inter-node locking needed by any
> cluster FS.  you'd do much better using just CLVM or EVMS-Ha
> 
> if it's a single box, it would be interesting to compare with ext3
> 
> > So, any ideas on how to solve the bottleneck?  Isn't the CFQ scheduler
> > supposed to grant every processes the same amount of I/O?  

Yes, but if the filesystem on top is at fault, the IO scheduler can't
help (this is the case with ext3 ordered mode and fsync latency, which
could last for hundreds of seconds last time I checked).

> > Is there a way to
> > change something in proc to avoid this situation?
> 
> i don't think CFQ can do much to alleviate the heavy lock-dependency
> of a cluster FS

Perhaps isolate the problem by having the guest images directly on
partitions first (or ext3 with writeback data mode).