Re: sync guest calls made async on host - SQLite performance

Re: sync guest calls made async on host - SQLite performance

[RW]

I've read the article a few days ago and it was interesting.
As I upgraded vom 2.6.29 to 2.6.30 (Gentoo) I also saw a dramatic
increase disk and filesystem performance. But then I realized
that the default mode for ext3 changed to "data=writeback".
So I changed that back to "data=ordered" and performance was
as it was with 2.6.29.

I think ext3 with "data=writeback" in a KVM and KVM started
with "if=virtio,cache=none" is a little bit crazy. I don't know
if this is the case with current Ubuntu Alpha but it looks
like so.

Regards,
Robert

> I would like to call attention to the SQLite performance under KVM in
> the current Ubuntu Alpha.
>
> http://www.phoronix.com/scan.php?page=article&item=linux_2631_kvm&num=3
>
> SQLite's benchmark as part of the Phoronix Test Suite is typically IO
> limited and is affected by both disk and filesystem performance.
>
> en comparing SQLite under the host against the guest OS,  there is an
> der of magnitude _IMPROVEMENT_ in the measured performance  of the guest.
>
> I am expecting that the host is doing synchronous IO operations but
> somewhere in the stack the calls are ultimately being made asynchronous
> or at the very least batched for writing.
>
> On the surface, this represents a data integrity issue and  I am
> interested in the KVM communities thoughts on this behaviour.  Is it
> expected? Is it acceptable?  Is it safe?

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 09/25/2009 10:00 AM, RW wrote:
> I think ext3 with "data=writeback" in a KVM and KVM started
> with "if=virtio,cache=none" is a little bit crazy. I don't know
> if this is the case with current Ubuntu Alpha but it looks
> like so.
>    

It's not crazy, qemu bypasses the cache with cache=none so the ext3 
data= setting is immaterial.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

I have created a launchpad bug against qemu-kvm in Ubuntu.

https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/437473

Just re-iterating, my concern isn't so much performance, but integrity 
of stock KVM configurations with server or other workloads that expect 
sync fileIO requests to be honored and synchronous to the underlying 
physical disk.

(That and ensuring that sanity reigns where a benchmark doesn't show a 
guest operating 10 times faster than a host for the same test :).

Regards,

Matthew
-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Avi Kivity <avi@redhat.com>
To: RW <kvm@tauceti.net>
Cc: kvm@vger.kernel.org
Date: 09/27/2009 07:37 AM

> On 09/25/2009 10:00 AM, RW wrote:
>> I think ext3 with "data=writeback" in a KVM and KVM started
>> with "if=virtio,cache=none" is a little bit crazy. I don't know
>> if this is the case with current Ubuntu Alpha but it looks
>> like so.
>>    
> 
> It's not crazy, qemu bypasses the cache with cache=none so the ext3 
> data= setting is immaterial.
> 

Re: sync guest calls made async on host - SQLite performance

[Anthony Liguori]

Matthew Tippett wrote:
> I have created a launchpad bug against qemu-kvm in Ubuntu.
>
> https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/437473
>
> Just re-iterating, my concern isn't so much performance, but integrity 
> of stock KVM configurations with server or other workloads that expect 
> sync fileIO requests to be honored and synchronous to the underlying 
> physical disk.
>
> (That and ensuring that sanity reigns where a benchmark doesn't show a 
> guest operating 10 times faster than a host for the same test :).

And I've closed it.  In the future, please actually reproduce a bug 
before filing it.  Reading it on a website doesn't mean it's true :-)

Regards,

Anthony Liguori

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

Okay, bringing the leafs of the discussions onto this thread.

As per

http://www.phoronix.com/scan.php?page=article&item=linux_2631_kvm&num=1&single=1

"The host OS (as well as the guest OS when testing under KVM) was 
running an Ubuntu 9.10 daily snapshot with the Linux 2.6.31 (final) kernel"

I am attempting to get the actual "daily" snapshot to provide the 
precise version.  I should have that information shortly.  It is likely 
that it was within 1-2 weeks prior to the article posting.


 > Ubuntu's Karmic release has _not_ been released yet.  For this
 > particular test, Phoronix was probably using an alpha drop before
 > Ubuntu switched from kvm-84 to qemu-kvm-0.11.0.

The "probably" was described above - it was a snapshot after the 2.6.31 
final as September 9th, the article was published on September 21st, so 
there is a finite window.

I have high confidence in the testing that Phoronix has done and don't 
expect to need to confirm the results explicitly, and I have pieced 
together the following information.  I should be able to get the actual 
daily build number but broadly it looks like it was

   Ubuntu 9.10 daily snapshot (~ 9th - 21st September)
   Linux 2.6.31 (packaged as 2.6.31-10.30 to 2.6.31-10.32)
   qemu-kvm 0.11 (packaged as 0.11.0~rc2-0ubuntu to 0.11.0~rc2-0ubuntu5

Once I get confirmation of the actual date, digger deeping can occur.



But, if it turned out to be Ubuntu 9.10, linux 2.6.31, qemu-kvm 0.11 
would there be any concerns?



I would prefer rather than riling against Phoronix or the results as 
presented, ask questions to seek further information about what was 
tested rather than writing off all of it as completely invalid.

Regards,

Matthew
-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Anthony Liguori <anthony@codemonkey.ws>
To: Matthew Tippett <tippettm@gmail.com>
Cc: Avi Kivity <avi@redhat.com>, RW <kvm@tauceti.net>, kvm@vger.kernel.org
Date: 09/29/2009 03:02 PM

> Matthew Tippett wrote:
>> I have created a launchpad bug against qemu-kvm in Ubuntu.
>>
>> https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/437473
>>
>> Just re-iterating, my concern isn't so much performance, but integrity 
>> of stock KVM configurations with server or other workloads that expect 
>> sync fileIO requests to be honored and synchronous to the underlying 
>> physical disk.
>>
>> (That and ensuring that sanity reigns where a benchmark doesn't show a 
>> guest operating 10 times faster than a host for the same test :).
> 
> And I've closed it.  In the future, please actually reproduce a bug 
> before filing it.  Reading it on a website doesn't mean it's true :-)
> 
> Regards,
> 
> Anthony Liguori

Re: sync guest calls made async on host - SQLite performance

[Dustin Kirkland]

On Tue, Sep 29, 2009 at 2:32 PM, Matthew Tippett <tippettm@gmail.com> wrote:
> I would prefer rather than riling against Phoronix or the results as
> presented, ask questions to seek further information about what was tested
> rather than writing off all of it as completely invalid.

Matthew-

If you could please provide very specific instructions that you have
personally used to reproduce this problem on the latest Ubuntu Karmic
qemu-kvm-0.11.0 package, that would help very much.

I have personally tried to reproduce this problem and I don't see the
problem manifested in Ubuntu's qemu-kvm package.

For the technical people on the list, our configure line in Ubuntu
looks like this:

./configure --prefix=/usr --disable-blobs --audio-drv-list="alsa pa
oss sdl" --audio-card-list="ac97 es1370 sb16 cs4231a adlib gus" ...

I don't think we're doing anything exotic, and we're not carrying any
major patches.  I have submitted each patch that we are carrying
upstream to the KVM and/or QEMU mailing lists as appropriate.

:-Dustin

Re: sync guest calls made async on host - SQLite performance

[Anthony Liguori]

Matthew Tippett wrote:
> Okay, bringing the leafs of the discussions onto this thread.
>
> As per
>
> http://www.phoronix.com/scan.php?page=article&item=linux_2631_kvm&num=1&single=1 
>
>
> "The host OS (as well as the guest OS when testing under KVM) was 
> running an Ubuntu 9.10 daily snapshot with the Linux 2.6.31 (final) 
> kernel"
>
> I am attempting to get the actual "daily" snapshot to provide the 
> precise version.  I should have that information shortly.  It is 
> likely that it was within 1-2 weeks prior to the article posting.
>
>
> > Ubuntu's Karmic release has _not_ been released yet.  For this
> > particular test, Phoronix was probably using an alpha drop before
> > Ubuntu switched from kvm-84 to qemu-kvm-0.11.0.
>
> The "probably" was described above - it was a snapshot after the 
> 2.6.31 final as September 9th, the article was published on September 
> 21st, so there is a finite window.
>
> I have high confidence in the testing that Phoronix has done and don't 
> expect to need to confirm the results explicitly,

Your confidence is misplaced apparently.

> and I have pieced together the following information.  I should be 
> able to get the actual daily build number but broadly it looks like it 
> was
>
>   Ubuntu 9.10 daily snapshot (~ 9th - 21st September)
>   Linux 2.6.31 (packaged as 2.6.31-10.30 to 2.6.31-10.32)
>   qemu-kvm 0.11 (packaged as 0.11.0~rc2-0ubuntu to 0.11.0~rc2-0ubuntu5

That's extremely unlikely.

> But, if it turned out to be Ubuntu 9.10, linux 2.6.31, qemu-kvm 0.11 
> would there be any concerns?

It's not relevant because it's not qemu-kvm-0.11.

Regards,

Anthony Liguori

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

I now have more information.

Dustin,

The version used was 0.11.0-rc2, from the 2009-09-11 karmic daily build. 
  The VM identifies itself as "AMD QEMU Virtual CPU version 0.10.92 
stepping 03".

When you indicated that you had attempted to reproduce the problem, what 
mechanism did you use?  Was it Karmic + KVM as the host and Karmic as 
the guest?  What test did you use?

I will re-open the launchpad bug if you believe it makes sense to 
continue the discussions there.

Anthony,

If you can suspend your disbelief for a short while and ask questions to 
clarify the details.  My only interest here is to understand the results 
presented by the benchmark and determine if there are data integrity risks.

Fundamentally, if there are modes of operation that applications can get 
a considerable performance boost by running the same OS under KVM then 
there will be lots of people happy.  But realistically it is an 
indication of something wrong, misconfigured or just broken it bears at 
least some discussion.

Bear in mind that upstream is relevant for KVM, but for distributions 
shipping KVM, they may have secondary concerns about patchesets and 
upstream changes that may be relevant for how they support their customers.

Regards,

Matthew



-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Anthony Liguori <anthony@codemonkey.ws>
To: Matthew Tippett <tippettm@gmail.com>
Cc: Avi Kivity <avi@redhat.com>, RW <kvm@tauceti.net>, kvm@vger.kernel.org
Date: 09/29/2009 04:51 PM

> Matthew Tippett wrote:
> 
> Your confidence is misplaced apparently.
> 
>> and I have pieced together the following information.  I should be 
>> able to get the actual daily build number but broadly it looks like it 
>> was
>>
>>   Ubuntu 9.10 daily snapshot (~ 9th - 21st September)
>>   Linux 2.6.31 (packaged as 2.6.31-10.30 to 2.6.31-10.32)
>>   qemu-kvm 0.11 (packaged as 0.11.0~rc2-0ubuntu to 0.11.0~rc2-0ubuntu5
> 
> That's extremely unlikely.
> 
>> But, if it turned out to be Ubuntu 9.10, linux 2.6.31, qemu-kvm 0.11 
>> would there be any concerns?
> 
> It's not relevant because it's not qemu-kvm-0.11.

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

(Resending to the list without multi-part).

I now have more information.

Dustin,

The version used was 0.11.0-rc2, from the 2009-09-11 karmic daily build.
  The VM identifies itself as "AMD QEMU Virtual CPU version 0.10.92
stepping 03".

When you indicated that you had attempted to reproduce the problem, what
mechanism did you use?  Was it Karmic + KVM as the host and Karmic as
the guest?  What test did you use?

I will re-open the launchpad bug if you believe it makes sense to
continue the discussions there.

Anthony,

If you can suspend your disbelief for a short while and ask questions to
clarify the details.  My only interest here is to understand the results
presented by the benchmark and determine if there are data integrity risks.

Fundamentally, if there are modes of operation that applications can get
a considerable performance boost by running the same OS under KVM then
there will be lots of people happy.  But realistically it is an
indication of something wrong, misconfigured or just broken it bears at
least some discussion.

Bear in mind that upstream is relevant for KVM, but for distributions
shipping KVM, they may have secondary concerns about patchesets and
upstream changes that may be relevant for how they support their customers.

Regards,

Matthew



-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Anthony Liguori <anthony@codemonkey.ws>
To: Matthew Tippett <tippettm@gmail.com>
Cc: Avi Kivity <avi@redhat.com>, RW <kvm@tauceti.net>, kvm@vger.kernel.org
Date: 09/29/2009 04:51 PM

> Matthew Tippett wrote:
> 
> Your confidence is misplaced apparently.
> 
>> and I have pieced together the following information.  I should be 
>> able to get the actual daily build number but broadly it looks like it 
>> was
>>
>>   Ubuntu 9.10 daily snapshot (~ 9th - 21st September)
>>   Linux 2.6.31 (packaged as 2.6.31-10.30 to 2.6.31-10.32)
>>   qemu-kvm 0.11 (packaged as 0.11.0~rc2-0ubuntu to 0.11.0~rc2-0ubuntu5
> 
> That's extremely unlikely.
> 
>> But, if it turned out to be Ubuntu 9.10, linux 2.6.31, qemu-kvm 0.11 
>> would there be any concerns?
> 
> It's not relevant because it's not qemu-kvm-0.11.

Re: sync guest calls made async on host - SQLite performance

[Dustin Kirkland]

On Wed, Oct 7, 2009 at 11:53 AM, Matthew Tippett <tippettm@gmail.com> wrote:
> When you indicated that you had attempted to reproduce the problem, what
> mechanism did you use?  Was it Karmic + KVM as the host and Karmic as
> the guest?  What test did you use?


I ran the following in several places:
  a) on the system running on real hardware,
    time dd if=/dev/zero of=$HOME/foo bs=1M count=500
    524288000 bytes (524 MB) copied, 9.72614 s, 53.9 MB/s
  b) in an vm running on qemu-kvm-0.11 on Karmic
    time dd if=/dev/zero of=$HOME/foo bs=1M count=500 oflag=direct
    524288000 bytes (524 MB) copied, 31.6961 s, 16.5 MB/s
  c) in a vm running on kvm-84 on Jaunty
    time dd if=/dev/zero of=$HOME/foo bs=1M count=500 oflag=direct
    524288000 bytes (524 MB) copied, 22.2169 s, 23.6 MB/s

Looking at the time it takes to write a 500MB file to a real hard
disk, and then inside of the VM.  If I were to experience the problem
on Karmic, I would have seen this dd of a 500MB file take far, far
less time than it takes to write that file to disk on the real
hardware.  This was not the case in my testing.

> I will re-open the launchpad bug if you believe it makes sense to
> continue the discussions there.

Please re-open the bug if you can describe a real test case that you
used to demonstrate the problem.  Without being rude, it's hard for me
to work from a bug that says a magazine article says that there's a
bug in the Ubuntu distribution of qemu-kvm-0.11.

If you can provide clear steps that you have used to experience the
problem, then I will be able to take this issue seriously, reproduce
it myself, and develop a fix.

:-Dustin

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

The benchmark used was the sqlite subtest in the phoronix test suite.

My awareness and involvement is beyond "reading a magazine article", I 
can elaborate if needed, but I don't believe it is necessary.

Process for reproduction, assuming Karmic,

	# apt-get install phoronix-test-suite

         $ phoronix-test-suite benchmark sqlite

Answer the questions (test-names, etc, etc), it will download sqlite, 
build it and execute the test.  By default the test runs three timesand 
averages the results.  The results experienced should be similar to the 
values identified at

http://www.phoronix.com/scan.php?page=article&item=linux_2631_kvm&num=3

Which is approximately 12 minutes for the native, and about 60 seconds 
for the guest.

Given that the performance under the guest is expected to be around 60 
seconds, I would suggest confirming performance there first.

Regards,

Matthew

-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Dustin Kirkland <dustin.kirkland@gmail.com>
To: Matthew Tippett <tippettm@gmail.com>
Cc: Anthony Liguori <anthony@codemonkey.ws>, Avi Kivity 
<avi@redhat.com>, RW <kvm@tauceti.net>, kvm@vger.kernel.org
Date: 10/07/2009 02:59 PM

> On Wed, Oct 7, 2009 at 11:53 AM, Matthew Tippett <tippettm@gmail.com> wrote:
>> When you indicated that you had attempted to reproduce the problem, what
>> mechanism did you use?  Was it Karmic + KVM as the host and Karmic as
>> the guest?  What test did you use?
> 
> 
> I ran the following in several places:
>   a) on the system running on real hardware,
>     time dd if=/dev/zero of=$HOME/foo bs=1M count=500
>     524288000 bytes (524 MB) copied, 9.72614 s, 53.9 MB/s
>   b) in an vm running on qemu-kvm-0.11 on Karmic
>     time dd if=/dev/zero of=$HOME/foo bs=1M count=500 oflag=direct
>     524288000 bytes (524 MB) copied, 31.6961 s, 16.5 MB/s
>   c) in a vm running on kvm-84 on Jaunty
>     time dd if=/dev/zero of=$HOME/foo bs=1M count=500 oflag=direct
>     524288000 bytes (524 MB) copied, 22.2169 s, 23.6 MB/s
> 
> Looking at the time it takes to write a 500MB file to a real hard
> disk, and then inside of the VM.  If I were to experience the problem
> on Karmic, I would have seen this dd of a 500MB file take far, far
> less time than it takes to write that file to disk on the real
> hardware.  This was not the case in my testing.
> 
>> I will re-open the launchpad bug if you believe it makes sense to
>> continue the discussions there.
> 
> Please re-open the bug if you can describe a real test case that you
> used to demonstrate the problem.  Without being rude, it's hard for me
> to work from a bug that says a magazine article says that there's a
> bug in the Ubuntu distribution of qemu-kvm-0.11.
> 
> If you can provide clear steps that you have used to experience the
> problem, then I will be able to take this issue seriously, reproduce
> it myself, and develop a fix.
> 
> :-Dustin

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 10/07/2009 09:31 PM, Matthew Tippett wrote:
> The benchmark used was the sqlite subtest in the phoronix test suite.
>
> My awareness and involvement is beyond "reading a magazine article", I 
> can elaborate if needed, but I don't believe it is necessary.
>
> Process for reproduction, assuming Karmic,
>
>     # apt-get install phoronix-test-suite
>
>         $ phoronix-test-suite benchmark sqlite
>
> Answer the questions (test-names, etc, etc), it will download sqlite, 
> build it and execute the test.  By default the test runs three 
> timesand averages the results.  The results experienced should be 
> similar to the values identified at
>
> http://www.phoronix.com/scan.php?page=article&item=linux_2631_kvm&num=3
>
> Which is approximately 12 minutes for the native, and about 60 seconds 
> for the guest.
>
> Given that the performance under the guest is expected to be around 60 
> seconds, I would suggest confirming performance there first.

What is the data set for this benchmark?  If it's much larger than guest 
RAM, but smaller than host RAM, you could be seeing the effects of read 
caching.

Another possiblity is barriers and flushing.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Avi Kivity <avi@redhat.com>
To: Matthew Tippett <tippettm@gmail.com>
Cc: Dustin Kirkland <dustin.kirkland@gmail.com>, Anthony Liguori 
<anthony@codemonkey.ws>, RW <kvm@tauceti.net>, kvm@vger.kernel.org
Date: 10/07/2009 04:12 PM

> 
> What is the data set for this benchmark?  If it's much larger than guest 
> RAM, but smaller than host RAM, you could be seeing the effects of read 
> caching.

2GB for host, 1.7GB accessible for the guest (although highly unlikely 
that the memory usage went very high at all.

> 
> Another possiblity is barriers and flushing.
> 

That is what I am expecting, remember that the host and the guest were 
the same OS, same config, nothing special.  So the variable in the mix 
is how Ubuntu Karmic interacts with the bare metal vs the qemu-kvm 
virtual metal.

The test itself is simply 12500 sequential inserts, designed to model a 
simple high-transactional load single-tier system.  I still have some 
investigations pending on how sqlite responds at the syscall level, but 
I believe it is requesting synchronous writes and then doing many writes.

The consequence of the structure of the benchmark is that if there is 
any caching occurring at all from the sqlite library down, then it tends 
to show.  And I believe that it is unexpectedly showing here (since the 
writes are expected to be synchronous to a physical disk).

If there is a clear rationale that the KVM community is comfortable 
with, then it becomes a distribution or deployment issue relative to 
data integrity where a synchronous write within a guest may not be 
synchronous to a physical disk.  I assume this would concern commercial 
and server users of virtual machines.

Regards,

Matthew

Re: sync guest calls made async on host - SQLite performance

[Dustin Kirkland]

On Wed, Oct 7, 2009 at 2:31 PM, Matthew Tippett <tippettm@gmail.com> wrote:
> The benchmark used was the sqlite subtest in the phoronix test suite.
>
> My awareness and involvement is beyond "reading a magazine article", I can
> elaborate if needed, but I don't believe it is necessary.
>
> Process for reproduction, assuming Karmic,
>
>        # apt-get install phoronix-test-suite
>
>        $ phoronix-test-suite benchmark sqlite
>
> Answer the questions (test-names, etc, etc), it will download sqlite, build
> it and execute the test.  By default the test runs three timesand averages
> the results.  The results experienced should be similar to the values
> identified at
>
> http://www.phoronix.com/scan.php?page=article&item=linux_2631_kvm&num=3
>
> Which is approximately 12 minutes for the native, and about 60 seconds for
> the guest.

I have tried this, exactly as you have described.  The tests took:

 * 1162.08033204 seconds on native hardware
 * 2306.68306303 seconds in a kvm using if=scsi disk
 * 405.382308006 seconds in a kvm using if=virtio

I am using an up-to-date Karmic amd64 system, running qemu-kvm-0.11.0,
on a Thinkpad x200, dual 2.4GHz, 4GB, and a somewhat slow 5400rpm SATA
disk.  The filesystem is ext4 in both the guest and the host.

So I'm not seeing a 10x or order-of-magnitude improvement by doing
this in the guest.  With a scsi interface, it's twice as slow.  With
virtio, it's a good bit faster, but not 10x faster.

That said, I don't know that I'm all that concerned about this, right
now.  I haven't looked in detail at what this test from phoronix is
actually doing (nor do I really have the time to do so).  Sorry.

:-Dustin

Re: sync guest calls made async on host - SQLite performance

[Dustin Kirkland]

On Fri, Oct 9, 2009 at 6:25 AM, Matthew Tippett <tippettm@gmail.com> wrote:
> Can I ask you to do the following...
>
>  1) Re-affirm that Ubuntu does not carry any non-stream patches and
> the build command and possibly any other unusual patches or
> commandline options.  This should push it back onto Avi and Anthony's
> plate.

I have put the patches we're carrying here, for your review:
 * http://rookery.canonical.com/~kirkland/patches

There's nothing exotic in here.  Most of these have been committed
upstream already.  All of them have been at least posted on these
lists.  None of these should affect your test case.

We configure with:
  ./configure --prefix=/usr --disable-blobs --audio-drv-list="alsa pa
oss sdl" --audio-card-list="ac97 es1370 sb16 cs4231a adlib gus"
--target-list="$(TARGET_SYSTEM_TCG) $(TARGET_LINUX_TCG)"

We carry a number of compiler options, mostly in the interest of
hardening and security:
  * https://wiki.ubuntu.com/CompilerFlags

The #define'd variables on my local system (which should be similar,
though not identical to our build servers) can be seen here:
 * http://rookery.canonical.com/~kirkland/defined

>  2) Carefully consider risks to virtualized environments in the
> server space and consider noting it in release notes.

Thank you for the suggestion.  I will take it under consideration.

:-Dustin

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

Thanks Duncan for reproducing the behavior outside myself and Phoronix.

I dug deeper into the actual syscalls being made by sqlite.  The salient 
part of the behaviour is small sequential writes followed by a fdatasync 
(effectively a metadata-free fsync).

As Dustin indicates,

    if scsi is used, you incur the cost of virtualization,
    if virtio is used, your guests fsyncs incur less cost.

So back to the question to the kvm team.  It appears that with the stock 
KVM setup customers who need higher data integrity (through fsync) 
should steer away from virtio for the moment.

Is that assessment correct?

Regards,

Matthew


-------- Original Message  --------
Subject: Re: sync guest calls made async on host - SQLite performance
From: Dustin Kirkland <dustin.kirkland@gmail.com>
To: Matthew Tippett <tippettm@gmail.com>, Anthony Liguori 
<anthony@codemonkey.ws>, Avi Kivity <avi@redhat.com>, RW 
<kvm@tauceti.net>, kvm@vger.kernel.org
Date: 10/09/2009 11:18 AM

> On Fri, Oct 9, 2009 at 6:25 AM, Matthew Tippett <tippettm@gmail.com> wrote:
>> Can I ask you to do the following...
>>
>>  1) Re-affirm that Ubuntu does not carry any non-stream patches and
>> the build command and possibly any other unusual patches or
>> commandline options.  This should push it back onto Avi and Anthony's
>> plate.
> 
> I have put the patches we're carrying here, for your review:
>  * http://rookery.canonical.com/~kirkland/patches
> 
> There's nothing exotic in here.  Most of these have been committed
> upstream already.  All of them have been at least posted on these
> lists.  None of these should affect your test case.
> 
> We configure with:
>   ./configure --prefix=/usr --disable-blobs --audio-drv-list="alsa pa
> oss sdl" --audio-card-list="ac97 es1370 sb16 cs4231a adlib gus"
> --target-list="$(TARGET_SYSTEM_TCG) $(TARGET_LINUX_TCG)"
> 
> We carry a number of compiler options, mostly in the interest of
> hardening and security:
>   * https://wiki.ubuntu.com/CompilerFlags
> 
> The #define'd variables on my local system (which should be similar,
> though not identical to our build servers) can be seen here:
>  * http://rookery.canonical.com/~kirkland/defined
> 
>>  2) Carefully consider risks to virtualized environments in the
>> server space and consider noting it in release notes.
> 
> Thank you for the suggestion.  I will take it under consideration.
> 
> :-Dustin

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 10/09/2009 09:06 PM, Matthew Tippett wrote:
> Thanks Duncan for reproducing the behavior outside myself and Phoronix.
>
> I dug deeper into the actual syscalls being made by sqlite.  The 
> salient part of the behaviour is small sequential writes followed by a 
> fdatasync (effectively a metadata-free fsync).
>
> As Dustin indicates,
>
>    if scsi is used, you incur the cost of virtualization,
>    if virtio is used, your guests fsyncs incur less cost.
>
> So back to the question to the kvm team.  It appears that with the 
> stock KVM setup customers who need higher data integrity (through 
> fsync) should steer away from virtio for the moment.
>
> Is that assessment correct?
>

Christoph, wasn't there a bug where the guest didn't wait for requests 
in response to a barrier request?

-- 
error compiling committee.c: too many arguments to function

Re: sync guest calls made async on host - SQLite performance

[Christoph Hellwig]

On Sun, Oct 11, 2009 at 11:16:42AM +0200, Avi Kivity wrote:
> >   if scsi is used, you incur the cost of virtualization,
> >   if virtio is used, your guests fsyncs incur less cost.
> >
> >So back to the question to the kvm team.  It appears that with the 
> >stock KVM setup customers who need higher data integrity (through 
> >fsync) should steer away from virtio for the moment.
> >
> >Is that assessment correct?
> >
> 
> Christoph, wasn't there a bug where the guest didn't wait for requests 
> in response to a barrier request?

Can't remember anything like that.  The "bug" was the complete lack of
cache flush infrastructure for virtio, and the lack of advertising a
volative write cache on ide.

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 10/14/2009 07:37 AM, Christoph Hellwig wrote:
>> Christoph, wasn't there a bug where the guest didn't wait for requests
>> in response to a barrier request?
>>      
> Can't remember anything like that.  The "bug" was the complete lack of
> cache flush infrastructure for virtio, and the lack of advertising a
> volative write cache on ide.
>    

By "complete flush infrastructure", you mean host-side and guest-side 
support for a new barrier command, yes?

But can't this be also implemented using QUEUE_ORDERED_DRAIN, and on the 
host side disabling the backing device write cache?  I'm talking about 
cache=none, primarily.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

[PATCH] virtio-blk: fallback to draining the queue if barrier ops are not supported

[Avi Kivity]

Early implementations of virtio devices did not support barrier operations,
but did commit the data to disk.  In such cases, drain the queue to emulate
barrier operations.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 drivers/block/virtio_blk.c |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 43f1938..2627cc3 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -354,12 +354,16 @@ static int __devinit virtblk_probe(struct virtio_device *vdev)
 	vblk->disk->driverfs_dev = &vdev->dev;
 	index++;
 
-	/* If barriers are supported, tell block layer that queue is ordered */
+	/* If barriers are supported, tell block layer that queue is ordered;
+	 * otherwise just drain the queue.
+	 */
 	if (virtio_has_feature(vdev, VIRTIO_BLK_F_FLUSH))
 		blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_DRAIN_FLUSH,
 				  virtblk_prepare_flush);
 	else if (virtio_has_feature(vdev, VIRTIO_BLK_F_BARRIER))
 		blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_TAG, NULL);
+	else
+		blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_DRAIN, NULL);
 
 	/* If disk is read-only in the host, the guest should obey */
 	if (virtio_has_feature(vdev, VIRTIO_BLK_F_RO))
-- 
1.6.2.5

Re: [PATCH] virtio-blk: fallback to draining the queue if barrier ops are not supported

[Javier Guerra]

On Wed, Oct 14, 2009 at 7:03 AM, Avi Kivity <avi@redhat.com> wrote:
> Early implementations of virtio devices did not support barrier operations,
> but did commit the data to disk.  In such cases, drain the queue to emulate
> barrier operations.

would this help on the (i think common) situation with XFS on a
virtio-enabled VM, using LVM-backed storage; where LVM just loses
barriers.

-- 
Javier

Re: [PATCH] virtio-blk: fallback to draining the queue if barrier ops are not supported

[Avi Kivity]

On 10/14/2009 11:46 PM, Javier Guerra wrote:
> On Wed, Oct 14, 2009 at 7:03 AM, Avi Kivity<avi@redhat.com>  wrote:
>    
>> Early implementations of virtio devices did not support barrier operations,
>> but did commit the data to disk.  In such cases, drain the queue to emulate
>> barrier operations.
>>      
> would this help on the (i think common) situation with XFS on a
> virtio-enabled VM, using LVM-backed storage; where LVM just loses
> barriers.
>    

No, it's a guest only patch.  If LVM loses barriers, I don't think 
anything can restore them.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [PATCH] virtio-blk: fallback to draining the queue if barrier ops are not supported

[Michael Tokarev]

Avi Kivity wrote:
> Early implementations of virtio devices did not support barrier operations,
> but did commit the data to disk.  In such cases, drain the queue to emulate
> barrier operations.

Are there any implementation currently that actually supports barriers?
As far as I remember there's no way to invoke barriers from a user-space
application on linux, and this is how kvm/qemu is running on this OS.

Thanks!

/mjt

Re: [PATCH] virtio-blk: fallback to draining the queue if barrier ops are not supported

[Christoph Hellwig]

On Wed, Oct 14, 2009 at 07:38:45PM +0400, Michael Tokarev wrote:
> Avi Kivity wrote:
> >Early implementations of virtio devices did not support barrier operations,
> >but did commit the data to disk.  In such cases, drain the queue to emulate
> >barrier operations.
> 
> Are there any implementation currently that actually supports barriers?
> As far as I remember there's no way to invoke barriers from a user-space
> application on linux, and this is how kvm/qemu is running on this OS.

Ignore all the barrier talk.  The way Linux uses the various storage
transport the primitives are queue draining (done entirely in the guest
block layer) and cache flushes.  Fdatasync is exactly the same primitive
as a WIN FLUSH CACHE in ATA or SYNCHRONIZE cache in SCSI module the lack
or ranges in fdatasync - but that is just a performance optimization and
not actually used by Linux guests for now.

Re: sync guest calls made async on host - SQLite performance

[Christoph Hellwig]

On Wed, Oct 14, 2009 at 08:03:41PM +0900, Avi Kivity wrote:
> >Can't remember anything like that.  The "bug" was the complete lack of
> >cache flush infrastructure for virtio, and the lack of advertising a
> >volative write cache on ide.
> >   
> 
> By "complete flush infrastructure", you mean host-side and guest-side 
> support for a new barrier command, yes?

The cache flush command, not barrier command.  The new virtio code
implements barrier the same way we do for IDE and SCSI - all barrier
semantics are implemented by generic code in the block layer by draining
the queues, the only thing we send over the wire are cache flush
commands in strategic places.

> But can't this be also implemented using QUEUE_ORDERED_DRAIN, and on the 
> host side disabling the backing device write cache?  I'm talking about 
> cache=none, primarily.

Yes, it could.  But as I found out in a long discussion with Stephen
it's not actually nessecary.  All filesystems do the right thing for
a device not claiming to support barriers if it doesn't include write
caches, that is implement ordering internally.  So there is no urge to
set QUEUE_ORDERED_DRAIN for the case without write cache.

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 10/14/2009 10:41 PM, Christoph Hellwig wrote:
>> But can't this be also implemented using QUEUE_ORDERED_DRAIN, and on the
>> host side disabling the backing device write cache?  I'm talking about
>> cache=none, primarily.
>>      
> Yes, it could.  But as I found out in a long discussion with Stephen
> it's not actually nessecary.  All filesystems do the right thing for
> a device not claiming to support barriers if it doesn't include write
> caches, that is implement ordering internally.  So there is no urge to
> set QUEUE_ORDERED_DRAIN for the case without write cache.
>    

Does virtio say it has a write cache or not (and how does one say it?)?

According to the report, a write+fdatasync completes too fast, at least 
on Ubuntu's qemu.  So perhaps somewhere this information is lost.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: sync guest calls made async on host - SQLite performance

[Christoph Hellwig]

On Thu, Oct 15, 2009 at 01:56:40AM +0900, Avi Kivity wrote:
> Does virtio say it has a write cache or not (and how does one say it?)?

Historically it didn't and the only safe way to use virtio was in
cache=writethrough mode.  Since qemu git as of 4th Sempember and Linux
2.6.32-rc there is a virtio-blk feature to communicate the existance
of a volatile write cache, and the support for a cache flush command.
With the combination of these two data=writeback and data=none modes
are safe for the first time.

Re: sync guest calls made async on host - SQLite performance

[Anthony Liguori]

Christoph Hellwig wrote:
> On Thu, Oct 15, 2009 at 01:56:40AM +0900, Avi Kivity wrote:
>   
>> Does virtio say it has a write cache or not (and how does one say it?)?
>>     
>
> Historically it didn't and the only safe way to use virtio was in
> cache=writethrough mode.

Which should be the default on Ubuntu's kvm that this report is 
concerned with so I'm a bit confused.

Avi's patch is a performance optimization, not a correctness issue?

Regards,

Anthony Liguori

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 10/15/2009 07:54 AM, Anthony Liguori wrote:
> Christoph Hellwig wrote:
>> On Thu, Oct 15, 2009 at 01:56:40AM +0900, Avi Kivity wrote:
>>> Does virtio say it has a write cache or not (and how does one say it?)?
>>
>> Historically it didn't and the only safe way to use virtio was in
>> cache=writethrough mode.

It didn't say?  So it's up to the default, which is what?

>
> Which should be the default on Ubuntu's kvm that this report is 
> concerned with so I'm a bit confused.
>
> Avi's patch is a performance optimization, not a correctness issue?

If filesystems do drain by default, it should be a no-op on 
cache!=writeback.

However if lseek(0); write(1); fdatasync(); are faster than disk speed, 
then something in our assumptions has to be wrong.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: sync guest calls made async on host - SQLite performance

[Christoph Hellwig]

On Wed, Oct 14, 2009 at 05:54:23PM -0500, Anthony Liguori wrote:
> >Historically it didn't and the only safe way to use virtio was in
> >cache=writethrough mode.
> 
> Which should be the default on Ubuntu's kvm that this report is 
> concerned with so I'm a bit confused.

So can we please get the detailed setup where this happens, that is:

filesystem used in the guest
any volume manager / software raid used in the guest
kernel version in the guest
image format used
qemu command line including caching mode, using ide/scsi/virtio, etc
qemu/kvm version
filesystem used in the host
any volume manager / software raid used in the host
kernel version in the host


> Avi's patch is a performance optimization, not a correctness issue?

It could actually minimally degrade performace.  For the existing
filesystems as upper layer it doesn not improve correctness either.

Re: sync guest calls made async on host - SQLite performance

[Christoph Hellwig]

On Thu, Oct 15, 2009 at 02:17:02PM +0200, Christoph Hellwig wrote:
> On Wed, Oct 14, 2009 at 05:54:23PM -0500, Anthony Liguori wrote:
> > >Historically it didn't and the only safe way to use virtio was in
> > >cache=writethrough mode.
> > 
> > Which should be the default on Ubuntu's kvm that this report is 
> > concerned with so I'm a bit confused.
> 
> So can we please get the detailed setup where this happens, that is:
> 
> filesystem used in the guest
> any volume manager / software raid used in the guest
> kernel version in the guest
> image format used
> qemu command line including caching mode, using ide/scsi/virtio, etc
> qemu/kvm version
> filesystem used in the host
> any volume manager / software raid used in the host
> kernel version in the host

And very important the mount options (/proc/self/mounts) of both host
and guest.

Re: sync guest calls made async on host - SQLite performance

[Avi Kivity]

On 10/15/2009 09:17 PM, Christoph Hellwig wrote:
>
> So can we please get the detailed setup where this happens, that is:
>
>    

Here's a setup where it doesn't happen (pwrite() + fdatasync() get to 
the disk):

> filesystem used in the guest
>    

ext4

> any volume manager / software raid used in the guest
>    

lvm over one physical device

> kernel version in the guest
>    

2.6.30.8 (F11)

> image format used
>    

qcow2

> qemu command line including caching mode, using ide/scsi/virtio, etc
>    

-drive file=...,if=virtio

> qemu/kvm version
>    

qemu-kvm.git master

> filesystem used in the host
>    

ext4

> any volume manager / software raid used in the host
>    

lvm over one device

> kernel version in the host
>    
2.6.30.8 (F11)

So it seems this is no longer an issue, but may be an issue on older 
setups.  I suspect LVM as it used to swallow barriers.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: sync guest calls made async on host - SQLite performance

[Anthony Liguori]

Matthew Tippett wrote:
> Thanks Duncan for reproducing the behavior outside myself and Phoronix.
>
> I dug deeper into the actual syscalls being made by sqlite.  The 
> salient part of the behaviour is small sequential writes followed by a 
> fdatasync (effectively a metadata-free fsync).
>
> As Dustin indicates,
>
>    if scsi is used, you incur the cost of virtualization,
>    if virtio is used, your guests fsyncs incur less cost.
>
> So back to the question to the kvm team.  It appears that with the 
> stock KVM setup customers who need higher data integrity (through 
> fsync) should steer away from virtio for the moment.
>
> Is that assessment correct?

No, it's an absurd assessment.

You have additional layers of caching happening because you're running a 
guest from a filesystem on the host.

A benchmark running under a guest that happens do be faster than the 
host does not indicate anything.  It could be that the benchmark is 
poorly written.

What operation, specifically, do you think is not behaving properly 
under kvm?  ext4 (karmic's default filesystem) does not enable barriers 
by default so it's unlikely this is anything barrier related.

> Regards,
>
> Matthew

Regards,

Anthony Liguori

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

> No, it's an absurd assessment.
> 
> You have additional layers of caching happening because you're running a 
> guest from a filesystem on the host.

Comments below.

> A benchmark running under a guest that happens do be faster than the 
> host does not indicate anything.  It could be that the benchmark is 
> poorly written.

I believe that I have removed the benchmark from discussion, we are now 
looking at semantics of small writes followed by
> 
> What operation, specifically, do you think is not behaving properly 
> under kvm?  ext4 (karmic's default filesystem) does not enable barriers 
> by default so it's unlikely this is anything barrier related.
>

Re-quoting me from two replies ago.

===
I dug deeper into the actual syscalls being made by sqlite.  The salient 
part of the behaviour is small sequential writes followed by a
fdatasync (effectively a metadata-free fsync).
===

And quoting from Dustin

===
I have tried this, exactly as you have described.  The tests took:

  * 1162.08033204 seconds on native hardware
  * 2306.68306303 seconds in a kvm using if=scsi disk
  * 405.382308006 seconds in a kvm using if=virtio
===

And finally Christoph

===
Can't remember anything like that.  The "bug" was the complete lack of
cache flush infrastructure for virtio, and the lack of advertising a
volative write cache on ide.
===

The _Operation_ that I believe is not behaving as expected is fdatasync 
under virtio. I understand your position that this is not a bug, but a 
configuration/packaging issue.

So I'll put it to you differently.  When a Linux guest issues a fsync or 
fdatasync what should occur?

o If the system has been configured in writeback mode then you don't 
worry about getting the data to the disk, so when the hypervisor has 
received the data, be happy with it.

o If the system is configured in writethrough mode, shouldn't the 
hypervisor look to get the data to disk ASAP?  Whether this is 
immediately, or batched with other data, I'll leave it to you guys.

As mentioned above, I am not saying it is a bug in KVM, and may well be 
a poor choice of configuration options within distributions.  From what 
I can interpret from above, scsi and writethrough is the safest model to 
go for.  By extension, for enterprise workloads where data integrity is 
more critical the default configuration of KVM under Ubuntu and possibly 
other distributions may be a poor choice.

Regards,

Matthew

Re: sync guest calls made async on host - SQLite performance

[Dustin Kirkland]

On Tue, Oct 13, 2009 at 9:09 PM, Matthew Tippett <tippettm@gmail.com> wrote:
> I believe that I have removed the benchmark from discussion, we are now
> looking at semantics of small writes followed by
...
> And quoting from Dustin
>
> ===
> I have tried this, exactly as you have described.  The tests took:
>
>  * 1162.08033204 seconds on native hardware
>  * 2306.68306303 seconds in a kvm using if=scsi disk
>  * 405.382308006 seconds in a kvm using if=virtio

Hang on now...

My timings are from running the Phoronix test *as you described*.  I
have not looked at what magic is happening inside of this Phoronix
test.  I am most certainly *not* speaking as to the quality or
legitimacy of the test.

:-Dustin

Re: sync guest calls made async on host - SQLite performance

[Matthew Tippett]

I understand.  However the test itself is fairly trivial
representation of a single teir high-transactional load system.  (Ie:
a system that is logging a large number of events).

The phoronix test suite simply hands over to a binary using sqlite and
does 25000 sequential inserts.  The overhead of the suite would be
measured in milliseconds at the start and end.  Over the life of the
test (100-2500 seconds), it becomes insignificant noise.

As I said, the relevant system calls itself for the running of the
test are expressed as

    write
    write
    write
    fdatasync

The writes are typically small (5-100) bytes.

With that information, I believe the method of execution is mostly
irrelevant.  If people are still concerned, I can write a trivial
application that should reproduce the behaviour.

It still ultimately comes down to the guests expected semantics of
fdatasync, and the actual behaviour relative to the hosts physical
device.  I am not saying that the currentl behaviour is wrong, I just
want a clear understanding of what is expected by the kvm team vs what
we are seeing.

Regards... Matthew


On 10/14/09, Dustin Kirkland <kirkland@canonical.com> wrote:
> On Tue, Oct 13, 2009 at 9:09 PM, Matthew Tippett <tippettm@gmail.com> wrote:
>> I believe that I have removed the benchmark from discussion, we are now
>> looking at semantics of small writes followed by
> ...
>> And quoting from Dustin
>>
>> ===
>> I have tried this, exactly as you have described.  The tests took:
>>
>>  * 1162.08033204 seconds on native hardware
>>  * 2306.68306303 seconds in a kvm using if=scsi disk
>>  * 405.382308006 seconds in a kvm using if=virtio
>
> Hang on now...
>
> My timings are from running the Phoronix test *as you described*.  I
> have not looked at what magic is happening inside of this Phoronix
> test.  I am most certainly *not* speaking as to the quality or
> legitimacy of the test.
>
> :-Dustin
>

-- 
Sent from my mobile device