Re: virtio disk slower than IDE?

Re: virtio disk slower than IDE?

[john cooper]

[prior attempts from elsewhere kept bouncing, apologies for any replication]

Gordan Bobic wrote:
> The test is building the Linux kernel (only taking the second run to give the test the benefit of local cache):
>
> make clean; make -j8 all; make clean; sync; time make -j8 all
>
> This takes about 10 minutes with IDE disk emulation and about 13 minutes with virtio. I ran the tests multiple time with most non-essential services on the host switched off (including cron/atd), and the guest in single-user mode to reduce the "noise" in the test to the minimum, and the results are pretty consistent, with virtio being about 30% behind.

I'd expect for an observed 30% wall clock time difference
of an operation as complex as a kernel build the base i/o
throughput disparity is substantially greater.  Did you
try a more simple/regular load, eg: a streaming dd read
of various block sizes from guest raw disk devices?
This is also considerably easier to debug vs. the complex
i/o load generated by a build.

One way to chop up the problem space is using blktrace
on the host to observe both the i/o patterns coming out
of qemu and the host's response to them in terms of
turn around time.  I expect you'll see somewhat different
nature requests generated by qemu w/r/t blocking and
number of threads serving virtio_blk requests relative
to ide but the host response should be essentially the
same in terms of data returned per unit time.

If the host looks to be turning around i/o request with
similar latency in both cases, the problem would be lower
frequency of requests generated by qemu in the case of
virtio_blk.   Here it would be useful to know the host
load generated by the guest for both cases.

-john


-- 
john.cooper@redhat.com

Re: virtio disk slower than IDE?

[Gordan Bobic]

john cooper wrote:

>> The test is building the Linux kernel (only taking the second run to give the test the benefit of local cache):
>>
>> make clean; make -j8 all; make clean; sync; time make -j8 all
>>
>> This takes about 10 minutes with IDE disk emulation and about 13 minutes with virtio. I ran the tests multiple time with most non-essential services on the host switched off (including cron/atd), and the guest in single-user mode to reduce the "noise" in the test to the minimum, and the results are pretty consistent, with virtio being about 30% behind.
> 
> I'd expect for an observed 30% wall clock time difference
> of an operation as complex as a kernel build the base i/o
> throughput disparity is substantially greater.  Did you
> try a more simple/regular load, eg: a streaming dd read
> of various block sizes from guest raw disk devices?
> This is also considerably easier to debug vs. the complex
> i/o load generated by a build.

I'm not convinced it's the read performance, since it's the second pass 
that is time, by which time all the source files will be in the guest's 
cache. I verified this by doing just one pass and priming it with:

find . -type f -exec cat '{}' > /dev/null \;

The execution times are indistinguishable from the second pass in the 
two-pass test.

To me that would indicate the the problem is with write performance, 
rather than read performance.

> One way to chop up the problem space is using blktrace
> on the host to observe both the i/o patterns coming out
> of qemu and the host's response to them in terms of
> turn around time.  I expect you'll see somewhat different
> nature requests generated by qemu w/r/t blocking and
> number of threads serving virtio_blk requests relative
> to ide but the host response should be essentially the
> same in terms of data returned per unit time.
> 
> If the host looks to be turning around i/o request with
> similar latency in both cases, the problem would be lower
> frequency of requests generated by qemu in the case of
> virtio_blk.   Here it would be useful to know the host
> load generated by the guest for both cases.

With virtio the CPU usage did seem to be noticeably lower. I figured 
that was because it was spending more time waiting for I/O to finish, 
since it was clearly bottlenecking on disk I/O (since that's the only 
thing that changed).

I'll try iozone's write tests and see how that compares. If I'm right 
about write performance being problematic, iozone might show the same 
performance deterioration on write tests compared to the IDE emulation.

Gordan

virtio disk slower than IDE?

[Gordan Bobic]

I just tried paravirtualized virtio block devices, and my tests show 
that they are approximately 30% slower than emulated IDE devices. I'm 
guessing this isn't normal. Is this a known issue or am I likely to have 
mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and 
guest).

Thanks.

Gordan

Re: virtio disk slower than IDE?

[Dor Laor]

On 11/14/2009 04:23 PM, Gordan Bobic wrote:
> I just tried paravirtualized virtio block devices, and my tests show
> that they are approximately 30% slower than emulated IDE devices. I'm
> guessing this isn't normal. Is this a known issue or am I likely to have
> mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and
> guest).

Please try to change the io scheduler on the host to io scheduler, it 
should boost your performance back.

>
> Thanks.
>
> Gordan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: virtio disk slower than IDE?

[Gordan Bobic]

Dor Laor wrote:
> On 11/14/2009 04:23 PM, Gordan Bobic wrote:
>> I just tried paravirtualized virtio block devices, and my tests show
>> that they are approximately 30% slower than emulated IDE devices. I'm
>> guessing this isn't normal. Is this a known issue or am I likely to have
>> mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and
>> guest).
> 
> Please try to change the io scheduler on the host to io scheduler, it 
> should boost your performance back.

I presume you mean the deadline io scheduler. I tried that (kernel 
parameter elevator=deadline) and it made no measurable difference 
compared to the cfq scheduler.

Gordan

Re: virtio disk slower than IDE?

[Dor Laor]

On 11/15/2009 02:00 PM, Gordan Bobic wrote:
> Dor Laor wrote:
>> On 11/14/2009 04:23 PM, Gordan Bobic wrote:
>>> I just tried paravirtualized virtio block devices, and my tests show
>>> that they are approximately 30% slower than emulated IDE devices. I'm
>>> guessing this isn't normal. Is this a known issue or am I likely to have
>>> mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and
>>> guest).
>>
>> Please try to change the io scheduler on the host to io scheduler, it
>> should boost your performance back.
>
> I presume you mean the deadline io scheduler. I tried that (kernel
> parameter elevator=deadline) and it made no measurable difference
> compared to the cfq scheduler.

What version of kvm do you use? Is it rhel5.4?
Can you post the qemu cmdline and the perf test in the guest?

Lastly, do you use cache=wb on qemu? it's just a fun mode, we use 
cache=off only.

>
> Gordan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: virtio disk slower than IDE?

[Gordan Bobic]

Dor Laor wrote:
> On 11/15/2009 02:00 PM, Gordan Bobic wrote:
>> Dor Laor wrote:
>>> On 11/14/2009 04:23 PM, Gordan Bobic wrote:
>>>> I just tried paravirtualized virtio block devices, and my tests show
>>>> that they are approximately 30% slower than emulated IDE devices. I'm
>>>> guessing this isn't normal. Is this a known issue or am I likely to 
>>>> have
>>>> mosconfigured something? I'm using 64-bit RHEL/CentOS 5 (both host and
>>>> guest).
>>>
>>> Please try to change the io scheduler on the host to io scheduler, it
>>> should boost your performance back.
>>
>> I presume you mean the deadline io scheduler. I tried that (kernel
>> parameter elevator=deadline) and it made no measurable difference
>> compared to the cfq scheduler.
> 
> What version of kvm do you use? Is it rhel5.4?

It's RHEL 5.4.

$ rpm -qa | grep -i kvm
kmod-kvm-83-105.el5_4.9
kvm-83-105.el5_4.9



> Can you post the qemu cmdline and the perf test in the guest?

Here is what is in the libvirt log:

For IDE emulation:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root 
LOGNAME=root /usr/libexec/qemu-kvm -S -M pc -m 2048 -smp 4 -name 
RHEL_5_x86-64 -uuid cb44b2c5-e64b-848f-77af-f8e7f02fa2ca 
-no-kvm-pit-reinjection -monitor pty -pidfile 
/var/run/libvirt/qemu//RHEL_5_x86-64.pid -boot c -drive 
file=/var/lib/libvirt/images/RHEL_5_x86-64.img,if=ide,index=0,boot=on 
-net nic,macaddr=54:52:00:5a:67:4b,vlan=0,model=e1000 -net 
tap,fd=15,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb 
-vnc 127.0.0.1:0 -k en-gb

For virtio:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root 
LOGNAME=root /usr/libexec/qemu-kvm -S -M pc -m 2048 -smp 4 -name 
RHEL_5_x86-64 -uuid cb44b2c5-e64b-848f-77af-f8e7f02fa2ca 
-no-kvm-pit-reinjection -monitor pty -pidfile 
/var/run/libvirt/qemu//RHEL_5_x86-64.pid -boot c -drive 
file=/var/lib/libvirt/images/CentOS_5_x86-64.img,if=virtio,index=0,boot=on 
-net nic,macaddr=54:52:00:5a:67:4b,vlan=0,model=e1000 -net 
tap,fd=15,script=,vlan=0,ifname=vnet0 -serial pty -parallel none -usb 
-vnc 127.0.0.1:0 -k en-gb

The test is building the Linux kernel (only taking the second run to 
give the test the benefit of local cache):

make clean; make -j8 all; make clean; sync; time make -j8 all

This takes about 10 minutes with IDE disk emulation and about 13 minutes 
with virtio. I ran the tests multiple time with most non-essential 
services on the host switched off (including cron/atd), and the guest in 
single-user mode to reduce the "noise" in the test to the minimum, and 
the results are pretty consistent, with virtio being about 30% behind.

> Lastly, do you use cache=wb on qemu? it's just a fun mode, we use 
> cache=off only.

I don't see the option being set in the logs, so I'd guess it's whatever 
qemu-kvm defaults to.

Gordan

Re: virtio disk slower than IDE?

[john cooper]

Gordan Bobic wrote:
> The test is building the Linux kernel (only taking the second run to 
> give the test the benefit of local cache):
> 
> make clean; make -j8 all; make clean; sync; time make -j8 all
> 
> This takes about 10 minutes with IDE disk emulation and about 13 minutes 
> with virtio. I ran the tests multiple time with most non-essential 
> services on the host switched off (including cron/atd), and the guest in 
> single-user mode to reduce the "noise" in the test to the minimum, and 
> the results are pretty consistent, with virtio being about 30% behind.

I'd expect for an observed 30% wall clock time difference
of an operation as complex as a kernel build the base i/o
throughput disparity is substantially greater.  Did you
try a more simple/regular load, eg: a streaming dd read
of various block sizes from guest raw disk devices?
This is also considerably easier to debug vs. the complex
i/o load generated by a build.

One way to chop up the problem space is using blktrace
on the host to observe both the i/o patterns coming out
of qemu and the host's response to them in terms of
turn around time.  I expect you'll see somewhat different
nature requests generated by qemu w/r/t blocking and
number of threads serving virtio_blk requests relative
to ide but the host response should be essentially the
same in terms of data returned per unit time.

If the host looks to be turning around i/o request with
similar latency in both cases, the problem would be lower
frequency of requests generated by qemu in the case of
virtio_blk.   Here it would be useful to know the host
load generated by the guest for both cases.

-john


-- 
john.cooper@third-harmonic.com

Re: virtio disk slower than IDE?

[Charles Duffy]

Gordan Bobic wrote:
>> Lastly, do you use cache=wb on qemu? it's just a fun mode, we use 
>> cache=off only.
> 
> I don't see the option being set in the logs, so I'd guess it's whatever 
> qemu-kvm defaults to.

You can set this through libvirt by putting an element such as the 
following within your <disk> element:

       <driver name='qemu' type='qcow2' cache='none'/>

(Setting the type is preferred to avoid security issues wherein a guest 
writes an arbitrary qcow2 header to the beginning of a raw disk, reboots 
and allows qemu's autodetection to decide that this formerly-raw disk 
should now be treated as a delta against a file they otherwise might not 
have access to read; as such, it's particularly important if you intend 
that the type be raw).

Re: virtio disk slower than IDE?

[Dor Laor]

On 11/16/2009 08:11 PM, Charles Duffy wrote:
> Gordan Bobic wrote:
>>> Lastly, do you use cache=wb on qemu? it's just a fun mode, we use
>>> cache=off only.
>>
>> I don't see the option being set in the logs, so I'd guess it's
>> whatever qemu-kvm defaults to.
>
> You can set this through libvirt by putting an element such as the
> following within your <disk> element:
>
> <driver name='qemu' type='qcow2' cache='none'/>

It's not needed on rhel5.4 qemu - we have cache=none as a default

>
> (Setting the type is preferred to avoid security issues wherein a guest
> writes an arbitrary qcow2 header to the beginning of a raw disk, reboots
> and allows qemu's autodetection to decide that this formerly-raw disk
> should now be treated as a delta against a file they otherwise might not
> have access to read; as such, it's particularly important if you intend
> that the type be raw).
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html