Avoiding I/O bottlenecks between VM's

Avoiding I/O bottlenecks between VM's

[Alberto Treviño]

I am using KVM 69 and kernel 2.6.25.17 to host several VM's in a large 
server.  So far, everything has been great.  Except I'm adding a Windows 
Server VM that will run a SQL Server database.  A few times I've noticed 
that I/O becomes a bottleneck for the VM and Windows VM freezes for a few 
seconds.  Oh well, no biggie.  Except, every so often, these I/O bottlenecks 
start to affect other VM's and they freeze as well for a few seconds.  I 
don't really care of one VM does so much I/O that it freezes itself 
temporarily.  I just don't want I/O bottlenecks on one VM to affect other 
VM's.

My questions are:

1. Is this a problem anyone else has experienced and has it been fixed in a 
later KVM release?

2. I'm using the CFQ scheduler.  Would the deadline scheduler do a better 
job?

3. Any other suggestions to improve this problem?

Re: Avoiding I/O bottlenecks between VM's

[Marcelo Tosatti]

Hi Alberto,

On Fri, Sep 19, 2008 at 11:26:09AM -0600, Alberto Treviño wrote:
> I am using KVM 69 and kernel 2.6.25.17 to host several VM's in a large 
> server.  So far, everything has been great.  Except I'm adding a Windows 
> Server VM that will run a SQL Server database.  A few times I've noticed 
> that I/O becomes a bottleneck for the VM and Windows VM freezes for a few 
> seconds.  Oh well, no biggie.  Except, every so often, these I/O bottlenecks 
> start to affect other VM's and they freeze as well for a few seconds.  I 
> don't really care of one VM does so much I/O that it freezes itself 
> temporarily.  I just don't want I/O bottlenecks on one VM to affect other 
> VM's.
> 
> My questions are:
> 
> 1. Is this a problem anyone else has experienced and has it been fixed in a 
> later KVM release?
> 
> 2. I'm using the CFQ scheduler.  Would the deadline scheduler do a better 
> job?
> 
> 3. Any other suggestions to improve this problem?

Are you using filesystem backed storage for the guest images or direct
block device storage? I assume there's heavy write activity on the
guests when these hangs happen?

ext3 with ordered data mode has latency issues on fsync.

Re: Avoiding I/O bottlenecks between VM's

[Alberto Treviño]

On Friday 19 September 2008 12:41:46 pm you wrote:
> Are you using filesystem backed storage for the guest images or direct
> block device storage? I assume there's heavy write activity on the
> guests when these hangs happen?

Yes, they happen when one VM is doing heavy writes.  I'm actually using a 
whole stack of things:

OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted 
partition.  Fun debugging that, heh?

In trying to figure out the problem, I tried to reconfigure DRBD to use 
Protocol B instead of C.  However, it failed to make the switch and both 
nodes disconnected so now I have a split-brain.  In try to fix the split 
brain I'm taking down on one node all the VM's one by one, copying the VM 
drives from one node to the next, and starting up on the other node (old-
fashioned migration).  Yes, I'm having *lots* of fun!  Perfect way to end 
the week!

So, any ideas on how to solve the bottleneck?  Isn't the CFQ scheduler 
supposed to grant every processes the same amount of I/O?  Is there a way to 
change something in proc to avoid this situation?

-- 
Alberto Treviño
BYU Testing Center
Brigham Young University

Re: Avoiding I/O bottlenecks between VM's

[Javier Guerra]

On Fri, Sep 19, 2008 at 1:53 PM, Alberto Treviño <alberto@byu.edu> wrote:
> On Friday 19 September 2008 12:41:46 pm you wrote:
>> Are you using filesystem backed storage for the guest images or direct
>> block device storage? I assume there's heavy write activity on the
>> guests when these hangs happen?
>
> Yes, they happen when one VM is doing heavy writes.  I'm actually using a
> whole stack of things:
>
> OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted
> partition.  Fun debugging that, heh?

a not-so-wild guess might be the inter-node locking needed by any
cluster FS.  you'd do much better using just CLVM or EVMS-Ha

if it's a single box, it would be interesting to compare with ext3

> So, any ideas on how to solve the bottleneck?  Isn't the CFQ scheduler
> supposed to grant every processes the same amount of I/O?  Is there a way to
> change something in proc to avoid this situation?

i don't think CFQ can do much to alleviate the heavy lock-dependency
of a cluster FS

-- 
Javier

Re: Avoiding I/O bottlenecks between VM's

[Marcelo Tosatti]

On Fri, Sep 19, 2008 at 02:14:32PM -0500, Javier Guerra wrote:
> On Fri, Sep 19, 2008 at 1:53 PM, Alberto Treviño <alberto@byu.edu> wrote:
> > On Friday 19 September 2008 12:41:46 pm you wrote:
> >> Are you using filesystem backed storage for the guest images or direct
> >> block device storage? I assume there's heavy write activity on the
> >> guests when these hangs happen?
> >
> > Yes, they happen when one VM is doing heavy writes.  I'm actually using a
> > whole stack of things:
> >
> > OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted
> > partition.  Fun debugging that, heh?

Heh. Lots of variables there.

> a not-so-wild guess might be the inter-node locking needed by any
> cluster FS.  you'd do much better using just CLVM or EVMS-Ha
> 
> if it's a single box, it would be interesting to compare with ext3
> 
> > So, any ideas on how to solve the bottleneck?  Isn't the CFQ scheduler
> > supposed to grant every processes the same amount of I/O?  

Yes, but if the filesystem on top is at fault, the IO scheduler can't
help (this is the case with ext3 ordered mode and fsync latency, which
could last for hundreds of seconds last time I checked).

> > Is there a way to
> > change something in proc to avoid this situation?
> 
> i don't think CFQ can do much to alleviate the heavy lock-dependency
> of a cluster FS

Perhaps isolate the problem by having the guest images directly on
partitions first (or ext3 with writeback data mode).