Poor performance with qemu

Poor performance with qemu

[Diego Calleja]

Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw"
qemu format according to virt-manager and, of course, placed on a btrfs
filesystem, running the latest mainline git) is awfully slow, no matter
what OS is running inside the VM. The PCBSD installer says it's copying
data at a 40-50 KB/s rate. Is someone using KVM and having better numbers
than me? How can I help to debug this workload?

Poor performance with qemu

[Markus Suvanto]

>Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw"
>qemu format according to virt-manager and, of course, placed on a btrfs
>filesystem, running the latest mainline git) is awfully slow, no matter
>what OS is running inside the VM. The PCBSD installer says it's copying
>data at a 40-50 KB/s rate. Is someone using KVM and having better numbers
>than me? How can I help to debug this workload?

Maybe cache=writeback helps?
-drive  file=image.raw,cache=writeback

-Markus

Re: Poor performance with qemu

[Chris Mason]

On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote:
> Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw"
> qemu format according to virt-manager and, of course, placed on a btrfs
> filesystem, running the latest mainline git) is awfully slow, no matter
> what OS is running inside the VM. The PCBSD installer says it's copying
> data at a 40-50 KB/s rate. Is someone using KVM and having better numbers
> than me? How can I help to debug this workload?

The problem is that qemu uses O_SYNC by default, which makes btrfs do
log commits for every write.

Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
to use a writeback cache instead.

-chris

Re: Poor performance with qemu

[Avi Kivity]

On 03/30/2010 03:56 PM, Chris Mason wrote:
> On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote:
>    
>> Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw"
>> qemu format according to virt-manager and, of course, placed on a btrfs
>> filesystem, running the latest mainline git) is awfully slow, no matter
>> what OS is running inside the VM. The PCBSD installer says it's copying
>> data at a 40-50 KB/s rate. Is someone using KVM and having better numbers
>> than me? How can I help to debug this workload?
>>      
> The problem is that qemu uses O_SYNC by default, which makes btrfs do
> log commits for every write.
>    

Problem is, btrfs takes the 50 KB/s guest rate and inflates it to 
something much larger (megabytes/sec).  Are there plans to reduce the 
amount of O_SYNC overhead writes?

I saw this too, but with 2.6.31 or 2.6.32 IIRC.

> Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
> to use a writeback cache instead.
>    

Even with writeback qemu will issue a lot of fsyncs.

-- 
error compiling committee.c: too many arguments to function

Re: Poor performance with qemu

[Gordan Bobic]

Avi Kivity wrote:
> On 03/30/2010 03:56 PM, Chris Mason wrote:
>> On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote:
>>   
>>> Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw"
>>> qemu format according to virt-manager and, of course, placed on a btrfs
>>> filesystem, running the latest mainline git) is awfully slow, no matter
>>> what OS is running inside the VM. The PCBSD installer says it's copying
>>> data at a 40-50 KB/s rate. Is someone using KVM and having better 
>>> numbers
>>> than me? How can I help to debug this workload?
>>>      
>> The problem is that qemu uses O_SYNC by default, which makes btrfs do
>> log commits for every write.
>>    
> 
> Problem is, btrfs takes the 50 KB/s guest rate and inflates it to 
> something much larger (megabytes/sec).  Are there plans to reduce the 
> amount of O_SYNC overhead writes?
> 
> I saw this too, but with 2.6.31 or 2.6.32 IIRC.
> 
>> Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
>> to use a writeback cache instead.
>>    
> 
> Even with writeback qemu will issue a lot of fsyncs.
> 

My understanding was that with cache=writeback qemu shouldn't issue any 
fsyncs at all, but I could be wrong.

Gordan

Re: Poor performance with qemu

[Chris Mason]

On Thu, Apr 08, 2010 at 05:58:17PM +0300, Avi Kivity wrote:
> On 03/30/2010 03:56 PM, Chris Mason wrote:
> >On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote:
> >>Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw"
> >>qemu format according to virt-manager and, of course, placed on a btrfs
> >>filesystem, running the latest mainline git) is awfully slow, no matter
> >>what OS is running inside the VM. The PCBSD installer says it's copying
> >>data at a 40-50 KB/s rate. Is someone using KVM and having better numbers
> >>than me? How can I help to debug this workload?
> >The problem is that qemu uses O_SYNC by default, which makes btrfs do
> >log commits for every write.
> 
> Problem is, btrfs takes the 50 KB/s guest rate and inflates it to
> something much larger (megabytes/sec).  Are there plans to reduce
> the amount of O_SYNC overhead writes?
> 
> I saw this too, but with 2.6.31 or 2.6.32 IIRC.

With O_DIRECT the writeback rates are very reasonable.  I'll work up a
way to pass the barrier down from the guest to btrfs to force logging of
updated metadata when required.

> 
> >Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
> >to use a writeback cache instead.
> 
> Even with writeback qemu will issue a lot of fsyncs.

Oh, I didn't see that when I was testing, when does it fsync?

-chris

Re: Poor performance with qemu

[Avi Kivity]

On 04/08/2010 06:26 PM, Chris Mason wrote:
>>
>>> Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
>>> to use a writeback cache instead.
>>>        
>> Even with writeback qemu will issue a lot of fsyncs.
>>      
> Oh, I didn't see that when I was testing, when does it fsync?
>    

When it updates qcow2 metadata or when the guest issues a barrier.  It's 
relatively new.  I have a patch that introduces cache=volatile somewhere.

-- 
error compiling committee.c: too many arguments to function

Re: Poor performance with qemu

[Christoph Hellwig]

On Thu, Apr 08, 2010 at 11:26:15AM -0400, Chris Mason wrote:
> With O_DIRECT the writeback rates are very reasonable.  I'll work up a
> way to pass the barrier down from the guest to btrfs to force logging of
> updated metadata when required.

Barriers are implemented in the guest kernel using queue drains and
cache flush commands.  Qemu maps the cache flush to fdatasync.

> > >Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
> > >to use a writeback cache instead.
> > 
> > Even with writeback qemu will issue a lot of fsyncs.
> 
> Oh, I didn't see that when I was testing, when does it fsync?

When the guest issues a barrier (and it's actually a fdatasync)

Re: Poor performance with qemu

[Chris Mason]

On Thu, Apr 08, 2010 at 06:28:54PM +0300, Avi Kivity wrote:
> On 04/08/2010 06:26 PM, Chris Mason wrote:
> >>
> >>>Once the O_DIRECT read patch is in, you can switch to that, or tell qemu
> >>>to use a writeback cache instead.
> >>Even with writeback qemu will issue a lot of fsyncs.
> >Oh, I didn't see that when I was testing, when does it fsync?
> 
> When it updates qcow2 metadata or when the guest issues a barrier.
> It's relatively new.  I have a patch that introduces cache=volatile
> somewhere.

Ok, that's actually perfect.  I'll retest with that if I can get
qemu-git to work here.

-chris

Re: Poor performance with qemu

[Christoph Hellwig]

On Thu, Apr 08, 2010 at 06:28:54PM +0300, Avi Kivity wrote:
> When it updates qcow2 metadata or when the guest issues a barrier.  It's  
> relatively new.  I have a patch that introduces cache=volatile somewhere.

qcow2 does not issues any fsyncs by itself, it only passes throught the
guests ones.  The only other placess issueing fsyncs is commit a COW
image back to the base image, and on migreation.

Re: Poor performance with qemu

[Avi Kivity]

On 04/08/2010 06:34 PM, Christoph Hellwig wrote:
> On Thu, Apr 08, 2010 at 06:28:54PM +0300, Avi Kivity wrote:
>    
>> When it updates qcow2 metadata or when the guest issues a barrier.  It's
>> relatively new.  I have a patch that introduces cache=volatile somewhere.
>>      
> qcow2 does not issues any fsyncs by itself, it only passes throught the
> guests ones.  The only other placess issueing fsyncs is commit a COW
> image back to the base image, and on migreation.
>    

Shouldn't it do that then?  What's the point of fsyncing guest data if 
qcow2 metadata is volatile?

-- 
error compiling committee.c: too many arguments to function

Re: Poor performance with qemu

[Christoph Hellwig]

On Thu, Apr 08, 2010 at 06:36:15PM +0300, Avi Kivity wrote:
> Shouldn't it do that then?  What's the point of fsyncing guest data if  
> qcow2 metadata is volatile?

Not my territory - but in the end getting qcow2 as-is solid in face
of crashes will be an uphill battel - I'd rather recommend not using it.