[Qemu-devel] virtio-scsi vs. virtio-blk

[Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe]

Hello list,

i wanted to start using virtio-scsi instead of virtio-blk, cause it 
offers the possibility to use discard / trim support.

Kernel: 3.5.0 on host and guest
Qemu-kvm: 1.1.1 stable

But i'm not seeing the same or nearly the same speed:

virtio-scsi:
rand. 4k:
   write: io=677628KB, bw=67709KB/s, iops=16927, runt= 10008msec
   read : io=697648KB, bw=69646KB/s, iops=17411, runt= 10017msec
seq 4m:
   write: io=8192MB, bw=817683KB/s, iops=199, runt= 10259msec
   read : io=5672MB, bw=565378KB/s, iops=138, runt= 10273msec

virtio-blk:
rand. 4k:
   write: io=1750MB, bw=179169KB/s, iops=44792, runt= 10001msec
   read : io=1388MB, bw=142073KB/s, iops=35518, runt= 10006msec
seq 4m:
   write: io=9776MB, bw=984522KB/s, iops=240, runt= 10168msec
   read : io=2980MB, bw=288970KB/s, iops=70, runt= 10560msec

Any ideas?

Greets,
Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 08/08/2012 17:21, Stefan Priebe ha scritto:
> Hello list,
> 
> i wanted to start using virtio-scsi instead of virtio-blk, cause it
> offers the possibility to use discard / trim support.
> 
> Kernel: 3.5.0 on host and guest
> Qemu-kvm: 1.1.1 stable
> 
> But i'm not seeing the same or nearly the same speed:

1) How did you start the guest?  Did you use cache=none?

2) Please try the latest qemu-kvm release.  1.1 didn't use ioeventfd for
virtio-scsi by mistake.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe]

Yes cache none. Is there a bugfix for 1.1.1?

Stefan

Am 08.08.2012 um 18:17 schrieb Paolo Bonzini <pbonzini@redhat.com>:

> Il 08/08/2012 17:21, Stefan Priebe ha scritto:
>> Hello list,
>> 
>> i wanted to start using virtio-scsi instead of virtio-blk, cause it
>> offers the possibility to use discard / trim support.
>> 
>> Kernel: 3.5.0 on host and guest
>> Qemu-kvm: 1.1.1 stable
>> 
>> But i'm not seeing the same or nearly the same speed:
> 
> 1) How did you start the guest?  Did you use cache=none?
> 
> 2) Please try the latest qemu-kvm release.  1.1 didn't use ioeventfd for
> virtio-scsi by mistake.
> 
> Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 08/08/2012 19:12, Stefan Priebe ha scritto:
> Yes cache none. Is there a bugfix for 1.1.1?

It's not fixed in 1.1.1, but it's fixed in git.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe]

i really would like to test with actual git. But my VMs run awfully SLOW 
with actual git version. Boot process prints one line every two seconds. 
So i can't test. Is there a patch or backport for this problem?

Thanks,

Stefan

Am 08.08.2012 18:17, schrieb Paolo Bonzini:
> Il 08/08/2012 17:21, Stefan Priebe ha scritto:
>> Hello list,
>>
>> i wanted to start using virtio-scsi instead of virtio-blk, cause it
>> offers the possibility to use discard / trim support.
>>
>> Kernel: 3.5.0 on host and guest
>> Qemu-kvm: 1.1.1 stable
>>
>> But i'm not seeing the same or nearly the same speed:
>
> 1) How did you start the guest?  Did you use cache=none?
>
> 2) Please try the latest qemu-kvm release.  1.1 didn't use ioeventfd for
> virtio-scsi by mistake.
>
> Paolo
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 08:13, Stefan Priebe ha scritto:
> i really would like to test with actual git. But my VMs run awfully SLOW
> with actual git version. Boot process prints one line every two seconds.
> So i can't test. Is there a patch or backport for this problem?

Hmm, no, I haven't seen it reported either...  How are you running the
VMs?  What guest is in there?

Also, perhaps you could try bisecting the issue?  I'm going on holiday
next week, but I'm sure other people could help.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe]

Yes should be possible. guest is Debian or Ubuntu. I couldn't find a tag for V1.1.1 which I ran from source. So where to start bisect?

Stefan

Am 09.08.2012 um 09:01 schrieb Paolo Bonzini <pbonzini@redhat.com>:

> Il 09/08/2012 08:13, Stefan Priebe ha scritto:
>> i really would like to test with actual git. But my VMs run awfully SLOW
>> with actual git version. Boot process prints one line every two seconds.
>> So i can't test. Is there a patch or backport for this problem?
> 
> Hmm, no, I haven't seen it reported either...  How are you running the
> VMs?  What guest is in there?
> 
> Also, perhaps you could try bisecting the issue?  I'm going on holiday
> next week, but I'm sure other people could help.
> 
> Paolo
> 

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 09:07, Stefan Priebe ha scritto:
> Yes should be possible. guest is Debian or Ubuntu. I couldn't find a
> tag for V1.1.1 which I ran from source. So where to start bisect?

You can start from the v1.1.0 tag.

Can you give the command line, perhaps it is enough to reproduce?

Paolo

> Stefan
> 
> Am 09.08.2012 um 09:01 schrieb Paolo Bonzini <pbonzini@redhat.com>:
> 
>> Il 09/08/2012 08:13, Stefan Priebe ha scritto:
>>> i really would like to test with actual git. But my VMs run
>>> awfully SLOW with actual git version. Boot process prints one
>>> line every two seconds. So i can't test. Is there a patch or
>>> backport for this problem?
>> 
>> Hmm, no, I haven't seen it reported either...  How are you running
>> the VMs?  What guest is in there?
>> 
>> Also, perhaps you could try bisecting the issue?  I'm going on
>> holiday next week, but I'm sure other people could help.
>> 
>> Paolo
>> 

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe]

starting line:
/usr/bin/qemu-x86_64 -chardev 
socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait -mon 
chardev=qmp,mode=control -pidfile /var/run/qemu-server/103.pid 
-daemonize -usbdevice tablet -name kvmcrash -smp sockets=1,cores=8 
-nodefaults -boot menu=on -vga cirrus -k de -device 
virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive 
file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/0,if=none,id=drive-scsi1,cache=writethrough,aio=native 
-device 
scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1 
-drive 
file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/2,if=none,id=drive-virtio0,cache=writethrough,aio=native 
-device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa 
-drive 
file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/1,if=none,id=drive-scsi0,cache=writethrough,aio=native 
-device 
scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=102 
-m 4096 -netdev 
type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,vhost=on 
-device 
virtio-net-pci,mac=BA:5B:86:AD:14:3A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

I've also needed this patch:
https://github.com/stefanha/qemu/commit/d3e3082e6fe63d45cea2d92c41ad148dccf1b63e

otherwise kvm crash while hardware detection of ubuntu / debian.

Greets,
Stefan
Am 09.08.2012 09:19, schrieb Paolo Bonzini:
> Il 09/08/2012 09:07, Stefan Priebe ha scritto:
>> Yes should be possible. guest is Debian or Ubuntu. I couldn't find a
>> tag for V1.1.1 which I ran from source. So where to start bisect?
>
> You can start from the v1.1.0 tag.
>
> Can you give the command line, perhaps it is enough to reproduce?
>
> Paolo
>
>> Stefan
>>
>> Am 09.08.2012 um 09:01 schrieb Paolo Bonzini <pbonzini@redhat.com>:
>>
>>> Il 09/08/2012 08:13, Stefan Priebe ha scritto:
>>>> i really would like to test with actual git. But my VMs run
>>>> awfully SLOW with actual git version. Boot process prints one
>>>> line every two seconds. So i can't test. Is there a patch or
>>>> backport for this problem?
>>>
>>> Hmm, no, I haven't seen it reported either...  How are you running
>>> the VMs?  What guest is in there?
>>>
>>> Also, perhaps you could try bisecting the issue?  I'm going on
>>> holiday next week, but I'm sure other people could help.
>>>
>>> Paolo
>>>
>
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 09:41, Stefan Priebe ha scritto:
> -drive file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/0,if=none,id=drive-scsi1,cache=writethrough,aio=native

Are you sure you want cache=writethrough here?

Also, please try either updating libiscsi to the latest, or using the
kernel iSCSI initiator.  This should ensure that the bug is not in libiscsi.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe]

@writethrough: why not? 

@libiscsi Same speed problem with cache=none and with just local lvm disks.

Stefan

Am 09.08.2012 um 09:53 schrieb Paolo Bonzini <pbonzini@redhat.com>:

> Il 09/08/2012 09:41, Stefan Priebe ha scritto:
>> -drive file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/0,if=none,id=drive-scsi1,cache=writethrough,aio=native
> 
> Are you sure you want cache=writethrough here?
> 
> Also, please try either updating libiscsi to the latest, or using the
> kernel iSCSI initiator.  This should ensure that the bug is not in libiscsi.
> 
> Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 10:00, Stefan Priebe ha scritto:
> @writethrough: why not? 

Because it's slow, and unnecessary if you're running kernel >= 2.6.32 or
recent RHEL/CentOS.

> @libiscsi Same speed problem with cache=none and with just local lvm disks.

Cool, please use these settings while bisecting.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Hajnoczi]

On Thu, Aug 9, 2012 at 8:41 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> starting line:
> /usr/bin/qemu-x86_64 -chardev
> socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait -mon
> chardev=qmp,mode=control -pidfile /var/run/qemu-server/103.pid -daemonize
> -usbdevice tablet -name kvmcrash -smp sockets=1,cores=8 -nodefaults -boot
> menu=on -vga cirrus -k de -device
> virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/0,if=none,id=drive-scsi1,cache=writethrough,aio=native
> -device
> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1
> -drive
> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/2,if=none,id=drive-virtio0,cache=writethrough,aio=native
> -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa
> -drive
> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/1,if=none,id=drive-scsi0,cache=writethrough,aio=native
> -device
> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=102
> -m 4096 -netdev
> type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
> -device
> virtio-net-pci,mac=BA:5B:86:AD:14:3A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

You're running qemu.git?  In that case make sure to use -enable-kvm,
otherwise you get TCG (dynamic binary translator) instead of using the
CPU's hardware virtualization extensions.

qemu-kvm.git enables KVM by default so this could explain slowness if
you've run distro qemu-kvm binaries or qemu-kvm.git in the past.

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

That looks better - thanks for the hint. But now network isn't working 
at all ;-(

Stefan
Am 09.08.2012 11:18, schrieb Stefan Hajnoczi:
> On Thu, Aug 9, 2012 at 8:41 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> starting line:
>> /usr/bin/qemu-x86_64 -chardev
>> socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait -mon
>> chardev=qmp,mode=control -pidfile /var/run/qemu-server/103.pid -daemonize
>> -usbdevice tablet -name kvmcrash -smp sockets=1,cores=8 -nodefaults -boot
>> menu=on -vga cirrus -k de -device
>> virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
>> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/0,if=none,id=drive-scsi1,cache=writethrough,aio=native
>> -device
>> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1
>> -drive
>> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/2,if=none,id=drive-virtio0,cache=writethrough,aio=native
>> -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa
>> -drive
>> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/1,if=none,id=drive-scsi0,cache=writethrough,aio=native
>> -device
>> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=102
>> -m 4096 -netdev
>> type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
>> -device
>> virtio-net-pci,mac=BA:5B:86:AD:14:3A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
>
> You're running qemu.git?  In that case make sure to use -enable-kvm,
> otherwise you get TCG (dynamic binary translator) instead of using the
> CPU's hardware virtualization extensions.
>
> qemu-kvm.git enables KVM by default so this could explain slowness if
> you've run distro qemu-kvm binaries or qemu-kvm.git in the past.
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Hajnoczi]

On Thu, Aug 9, 2012 at 11:17 AM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> That looks better - thanks for the hint. But now network isn't working at
> all ;-(

You need to have commit 26b9b5fe17cc1b6be2e8bf8b9d16094f420bb8ad
("virtio: fix vhost handling").  Pull the latest qemu.git/master
changes if you don't have it.

Besides that recent vhost-net fix I'm not sure what the problem could
be.  Your command-line looks sane.  Can you share errors messages or
details on the failure?

With tap the most common problem is permissions since the QEMU process
needs to have access to the /dev/tap* device.

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 13:04, Stefan Hajnoczi ha scritto:
>> > That looks better - thanks for the hint. But now network isn't working at
>> > all ;-(
> You need to have commit 26b9b5fe17cc1b6be2e8bf8b9d16094f420bb8ad
> ("virtio: fix vhost handling").  Pull the latest qemu.git/master
> changes if you don't have it.

Or for bisection, just disable vhost.

Paolo

> Besides that recent vhost-net fix I'm not sure what the problem could
> be.  Your command-line looks sane.  Can you share errors messages or
> details on the failure?
> 
> With tap the most common problem is permissions since the QEMU process
> needs to have access to the /dev/tap* device.

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 09.08.2012 13:04, schrieb Stefan Hajnoczi:
> On Thu, Aug 9, 2012 at 11:17 AM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> That looks better - thanks for the hint. But now network isn't working at
>> all ;-(
>
> You need to have commit 26b9b5fe17cc1b6be2e8bf8b9d16094f420bb8ad
> ("virtio: fix vhost handling").  Pull the latest qemu.git/master
> changes if you don't have it.
>
> Besides that recent vhost-net fix I'm not sure what the problem could
> be.  Your command-line looks sane.  Can you share errors messages or
> details on the failure?
>
> With tap the most common problem is permissions since the QEMU process
> needs to have access to the /dev/tap* device.

no error messages. I just can't connect to the gateway. installing 1.1.1 
results in working network again. Same command start line.

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

sorry guys i mixed qemu vs. qemu-kvm. Network is now working fine.

but still virtio-scsi vs. virtio-blk:

virtio-scsi:
rand 4k:
   write: io=822448KB, bw=82228KB/s, iops=20557, runt= 10002msec
   read : io=950920KB, bw=94694KB/s, iops=23673, runt= 10042msec
seq:
   write: io=2436MB, bw=231312KB/s, iops=56, runt= 10784msec
   read : io=3248MB, bw=313799KB/s, iops=76, runt= 10599msec

virtio-blk:
rand 4k:
   write: io=896472KB, bw=89051KB/s, iops=22262, runt= 10067msec
   read : io=1710MB, bw=175073KB/s, iops=43768, runt= 10002msec
seq:
   write: io=4008MB, bw=391285KB/s, iops=95, runt= 10489msec
   read : io=5748MB, bw=570178KB/s, iops=139, runt= 10323msec

Stefan

Am 09.08.2012 13:04, schrieb Stefan Hajnoczi:
> On Thu, Aug 9, 2012 at 11:17 AM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> That looks better - thanks for the hint. But now network isn't working at
>> all ;-(
>
> You need to have commit 26b9b5fe17cc1b6be2e8bf8b9d16094f420bb8ad
> ("virtio: fix vhost handling").  Pull the latest qemu.git/master
> changes if you don't have it.
>
> Besides that recent vhost-net fix I'm not sure what the problem could
> be.  Your command-line looks sane.  Can you share errors messages or
> details on the failure?
>
> With tap the most common problem is permissions since the QEMU process
> needs to have access to the /dev/tap* device.
>
> Stefan
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 14:08, Stefan Priebe - Profihost AG ha scritto:
> 
> virtio-scsi:
> rand 4k:
>   write: io=822448KB, bw=82228KB/s, iops=20557, runt= 10002msec
>   read : io=950920KB, bw=94694KB/s, iops=23673, runt= 10042msec
> seq:
>   write: io=2436MB, bw=231312KB/s, iops=56, runt= 10784msec
>   read : io=3248MB, bw=313799KB/s, iops=76, runt= 10599msec
> 
> virtio-blk:
> rand 4k:
>   write: io=896472KB, bw=89051KB/s, iops=22262, runt= 10067msec
>   read : io=1710MB, bw=175073KB/s, iops=43768, runt= 10002msec
> seq:
>   write: io=4008MB, bw=391285KB/s, iops=95, runt= 10489msec
>   read : io=5748MB, bw=570178KB/s, iops=139, runt= 10323msec

Thanks; some overhead is expected, but not this much.  Especially the
sequential case is bad, what disk is this?

Things to test include:

- using the deadline I/O scheduler on at least the host, and possibly
the guest too

- running perf on the guest and the host (separately) to get profiles

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 09.08.2012 14:19, schrieb Paolo Bonzini:
> Il 09/08/2012 14:08, Stefan Priebe - Profihost AG ha scritto:
>>
>> virtio-scsi:
>> rand 4k:
>>    write: io=822448KB, bw=82228KB/s, iops=20557, runt= 10002msec
>>    read : io=950920KB, bw=94694KB/s, iops=23673, runt= 10042msec
>> seq:
>>    write: io=2436MB, bw=231312KB/s, iops=56, runt= 10784msec
>>    read : io=3248MB, bw=313799KB/s, iops=76, runt= 10599msec
>>
>> virtio-blk:
>> rand 4k:
>>    write: io=896472KB, bw=89051KB/s, iops=22262, runt= 10067msec
>>    read : io=1710MB, bw=175073KB/s, iops=43768, runt= 10002msec
>> seq:
>>    write: io=4008MB, bw=391285KB/s, iops=95, runt= 10489msec
>>    read : io=5748MB, bw=570178KB/s, iops=139, runt= 10323msec
>
> Thanks; some overhead is expected, but not this much.  Especially the
> sequential case is bad, what disk is this?

right now this is an external iscsi Nexentastor. Locally i can't get 
this bandwith nor these iops to test.


> Things to test include:
>
> - using the deadline I/O scheduler on at least the host, and possibly
> the guest too
guest uses noop right now. Disk Host is nexentastor running open 
solaris. I use libiscsi right now so the disks are not visible in both 
cases (virtio-blk and virtio-scsi) to the host right now.

> - running perf on the guest and the host (separately) to get profiles
Which command of perf? Just perf top?

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 14:31, Stefan Priebe - Profihost AG ha scritto:
> Am 09.08.2012 14:19, schrieb Paolo Bonzini:
>> Il 09/08/2012 14:08, Stefan Priebe - Profihost AG ha scritto:
>>>
>>> virtio-scsi:
>>> rand 4k:
>>>    write: io=822448KB, bw=82228KB/s, iops=20557, runt= 10002msec
>>>    read : io=950920KB, bw=94694KB/s, iops=23673, runt= 10042msec
>>> seq:
>>>    write: io=2436MB, bw=231312KB/s, iops=56, runt= 10784msec
>>>    read : io=3248MB, bw=313799KB/s, iops=76, runt= 10599msec
>>>
>>> virtio-blk:
>>> rand 4k:
>>>    write: io=896472KB, bw=89051KB/s, iops=22262, runt= 10067msec
>>>    read : io=1710MB, bw=175073KB/s, iops=43768, runt= 10002msec
>>> seq:
>>>    write: io=4008MB, bw=391285KB/s, iops=95, runt= 10489msec
>>>    read : io=5748MB, bw=570178KB/s, iops=139, runt= 10323msec
>>
>> Thanks; some overhead is expected, but not this much.  Especially the
>> sequential case is bad, what disk is this?
> 
> right now this is an external iscsi Nexentastor. Locally i can't get
> this bandwith nor these iops to test.

Thanks for the information.

>> Things to test include:
>>
>> - using the deadline I/O scheduler on at least the host, and possibly
>> the guest too
> guest uses noop right now. Disk Host is nexentastor running open
> solaris. I use libiscsi right now so the disks are not visible in both
> cases (virtio-blk and virtio-scsi) to the host right now.

Ok, try deadline in the guest then.  Using noop amplifies bad
performance, because you lose request merging.  With no host scheduler,
as is the case with libiscsi, noop is rarely a good idea.

>> - running perf on the guest and the host (separately) to get profiles
> Which command of perf? Just perf top?

Yeah, perhaps you could make the benchmarks a bit longer so that perf
top has more time to stabilize.  Then you can just cut-and-paste perf
top output.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

OK VMs do work fine now. Sorry for missing the patch after switching to 
qemu-kvm.

Am 09.08.2012 14:44, schrieb Paolo Bonzini:
> Ok, try deadline in the guest then.  Using noop amplifies bad
> performance, because you lose request merging.  With no host scheduler,
> as is the case with libiscsi, noop is rarely a good idea.

Tests with cache=none and deadline:
virtio-scsi:
   write: io=8267MB, bw=94023KB/s, iops=23505, runt= 90032msec
   read : io=14576MB, bw=165783KB/s, iops=41445, runt= 90035msec
   write: io=21312MB, bw=240782KB/s, iops=58, runt= 90636msec
   read : io=48072MB, bw=544882KB/s, iops=133, runt= 90342msec

virtio-blk:
   write: io=9210MB, bw=104710KB/s, iops=26177, runt= 90070msec
   read : io=18804MB, bw=213929KB/s, iops=53482, runt= 90006msec
   write: io=29080MB, bw=329478KB/s, iops=80, runt= 90379msec
   read : io=26328MB, bw=298159KB/s, iops=72, runt= 90421msec

So this looks OK to me. Do you agree? So with virtio-scsi discard should 
work?

Greets and many thanks
Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[ronnie sahlberg]

On Thu, Aug 9, 2012 at 10:31 PM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Am 09.08.2012 14:19, schrieb Paolo Bonzini:
>
>> Il 09/08/2012 14:08, Stefan Priebe - Profihost AG ha scritto:
>>>
>>>
>>> virtio-scsi:
>>> rand 4k:
>>>    write: io=822448KB, bw=82228KB/s, iops=20557, runt= 10002msec
>>>    read : io=950920KB, bw=94694KB/s, iops=23673, runt= 10042msec
>>> seq:
>>>    write: io=2436MB, bw=231312KB/s, iops=56, runt= 10784msec
>>>    read : io=3248MB, bw=313799KB/s, iops=76, runt= 10599msec
>>>
>>> virtio-blk:
>>> rand 4k:
>>>    write: io=896472KB, bw=89051KB/s, iops=22262, runt= 10067msec
>>>    read : io=1710MB, bw=175073KB/s, iops=43768, runt= 10002msec
>>> seq:
>>>    write: io=4008MB, bw=391285KB/s, iops=95, runt= 10489msec
>>>    read : io=5748MB, bw=570178KB/s, iops=139, runt= 10323msec
>>
>>
>> Thanks; some overhead is expected, but not this much.  Especially the
>> sequential case is bad, what disk is this?
>
>
> right now this is an external iscsi Nexentastor. Locally i can't get this
> bandwith nor these iops to test.
>
>
>
>> Things to test include:
>>
>> - using the deadline I/O scheduler on at least the host, and possibly
>> the guest too
>
> guest uses noop right now. Disk Host is nexentastor running open solaris. I
> use libiscsi right now so the disks are not visible in both cases
> (virtio-blk and virtio-scsi) to the host right now.
>

And if you mount the disks locally on the host using open-iscsi, and
access them as /dev/sg* from qemu, what performance do you get?


virtio-blk would first go to scsi emulation and then call out to
block/iscsi.c to translate back to scsi commands to send to libiscsi

while virtio-scsi (I think) would treat libiscsi as a generic scsi
passthrough device.  I.e. all commands just go straight through
bdrv_aio_ioctl(SG_IO)


If anything, I think the codepaths should be shorter for the
virtio-scsi case, and it should due to the lack of scsi emulate and
scsi re-encode perform better.


Can you try also using normal scsi-generic and see how it performs
compared to virtio-blk/-scsi ?

git show 983924532f61091fd90d1f2fafa4aa938c414dbb
This command shows how to set up libiscsi with passthrough via an
emulated scsi hba.


as  virtio-blk/-scsi both use libiscsi, i think the bottleneck might
either be the interface between guest and qemu,  or the difference to
the guest when talking to local scsi-emulation vs talking to the
passthrough remote target.

also is it possible to map the LUNs locally on the host using
open-iscsi and then use the scsi-generic devices /dev/sg*  for qemu
and see how it compares?

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 14:52, ronnie sahlberg ha scritto:
>> >
>> > guest uses noop right now. Disk Host is nexentastor running open solaris. I
>> > use libiscsi right now so the disks are not visible in both cases
>> > (virtio-blk and virtio-scsi) to the host right now.
>> >
> And if you mount the disks locally on the host using open-iscsi, and
> access them as /dev/sg* from qemu, what performance do you get?

Good question.

> virtio-blk would first go to scsi emulation and then call out to
> block/iscsi.c to translate back to scsi commands to send to libiscsi
> 
> while virtio-scsi (I think) would treat libiscsi as a generic scsi
> passthrough device.  I.e. all commands just go straight through
> bdrv_aio_ioctl(SG_IO)

I think he's not using scsi-block or scsi-generic, because 1.0 libiscsi
didn't support that.

scsi-generic would indeed incur some overhead because it does not do
scatter/gather I/O directly, but scsi-hd/scsi-block do not have this
overhead.  In any case, that should be visible through the output of
perf if it is significant.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 09.08.2012 15:15, schrieb Paolo Bonzini:
> Il 09/08/2012 14:52, ronnie sahlberg ha scritto:
>>>>
>>>> guest uses noop right now. Disk Host is nexentastor running open solaris. I
>>>> use libiscsi right now so the disks are not visible in both cases
>>>> (virtio-blk and virtio-scsi) to the host right now.
>>>>
>> And if you mount the disks locally on the host using open-iscsi, and
>> access them as /dev/sg* from qemu, what performance do you get?
>
> Good question.
>
>> virtio-blk would first go to scsi emulation and then call out to
>> block/iscsi.c to translate back to scsi commands to send to libiscsi
>>
>> while virtio-scsi (I think) would treat libiscsi as a generic scsi
>> passthrough device.  I.e. all commands just go straight through
>> bdrv_aio_ioctl(SG_IO)
>
> I think he's not using scsi-block or scsi-generic, because 1.0 libiscsi
> didn't support that.
>
> scsi-generic would indeed incur some overhead because it does not do
> scatter/gather I/O directly, but scsi-hd/scsi-block do not have this
> overhead.  In any case, that should be visible through the output of
> perf if it is significant.

Thanks for your help and replies. I'm a little bit lost on all these 
comments. So what to check / do next?

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 15:39, Stefan Priebe - Profihost AG ha scritto:
>> scsi-generic would indeed incur some overhead because it does not do
>> scatter/gather I/O directly, but scsi-hd/scsi-block do not have this
>> overhead.  In any case, that should be visible through the output of
>> perf if it is significant.
> 
> Thanks for your help and replies. I'm a little bit lost on all these
> comments. So what to check / do next?

Try with the deadline scheduler, and run "perf top" in guest/host if the
problems persist.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 09.08.2012 15:42, schrieb Paolo Bonzini:
> Il 09/08/2012 15:39, Stefan Priebe - Profihost AG ha scritto:
>>> scsi-generic would indeed incur some overhead because it does not do
>>> scatter/gather I/O directly, but scsi-hd/scsi-block do not have this
>>> overhead.  In any case, that should be visible through the output of
>>> perf if it is significant.
>>
>> Thanks for your help and replies. I'm a little bit lost on all these
>> comments. So what to check / do next?
>
> Try with the deadline scheduler, and run "perf top" in guest/host if the
> problems persist.

Thanks will do so. Right now i'm getting virtio_net virtio2: output:id 1 
is not a head!

and then network is down.

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 09/08/2012 15:52, Stefan Priebe - Profihost AG ha scritto:
> 
> Am 09.08.2012 15:42, schrieb Paolo Bonzini:
>> Il 09/08/2012 15:39, Stefan Priebe - Profihost AG ha scritto:
>>>> scsi-generic would indeed incur some overhead because it does not do
>>>> scatter/gather I/O directly, but scsi-hd/scsi-block do not have this
>>>> overhead.  In any case, that should be visible through the output of
>>>> perf if it is significant.
>>>
>>> Thanks for your help and replies. I'm a little bit lost on all these
>>> comments. So what to check / do next?
>>
>> Try with the deadline scheduler, and run "perf top" in guest/host if the
>> problems persist.
> 
> Thanks will do so. Right now i'm getting virtio_net virtio2: output:id 1
> is not a head!
> 
> and then network is down.

Please disable vhost, or ensure that you have the patch that Stefan
pointed you to.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 09.08.2012 15:42, schrieb Paolo Bonzini:
> Il 09/08/2012 15:39, Stefan Priebe - Profihost AG ha scritto:
>>> scsi-generic would indeed incur some overhead because it does not do
>>> scatter/gather I/O directly, but scsi-hd/scsi-block do not have this
>>> overhead.  In any case, that should be visible through the output of
>>> perf if it is significant.
>>
>> Thanks for your help and replies. I'm a little bit lost on all these
>> comments. So what to check / do next?
>
> Try with the deadline scheduler, and run "perf top" in guest/host if the
> problems persist.

Right now i'm just starting to ping from the guest to a random domain. 
it works for around 65s then i get the message:
ping: senmsg: No buffer space available

and the network connection is away until i reboot the guest.

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

virtio-scsi is now working fine. Could you please help me to get discard 
/ trim running? I can't find any information what is needed to get 
discard / trim working.

Thanks,
Stefan
Am 09.08.2012 12:17, schrieb Stefan Priebe - Profihost AG:
> That looks better - thanks for the hint. But now network isn't working
> at all ;-(
>
> Stefan
> Am 09.08.2012 11:18, schrieb Stefan Hajnoczi:
>> On Thu, Aug 9, 2012 at 8:41 AM, Stefan Priebe <s.priebe@profihost.ag>
>> wrote:
>>> starting line:
>>> /usr/bin/qemu-x86_64 -chardev
>>> socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait -mon
>>> chardev=qmp,mode=control -pidfile /var/run/qemu-server/103.pid
>>> -daemonize
>>> -usbdevice tablet -name kvmcrash -smp sockets=1,cores=8 -nodefaults
>>> -boot
>>> menu=on -vga cirrus -k de -device
>>> virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
>>> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/0,if=none,id=drive-scsi1,cache=writethrough,aio=native
>>>
>>> -device
>>> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1
>>>
>>> -drive
>>> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/2,if=none,id=drive-virtio0,cache=writethrough,aio=native
>>>
>>> -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa
>>> -drive
>>> file=iscsi://10.0.255.100/iqn.1986-03.com.sun:02:8a9019a4-4aa3-cd8a-f723-f05db9085ef9/1,if=none,id=drive-scsi0,cache=writethrough,aio=native
>>>
>>> -device
>>> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=102
>>>
>>> -m 4096 -netdev
>>> type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
>>>
>>> -device
>>> virtio-net-pci,mac=BA:5B:86:AD:14:3A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
>>>
>>
>> You're running qemu.git?  In that case make sure to use -enable-kvm,
>> otherwise you get TCG (dynamic binary translator) instead of using the
>> CPU's hardware virtualization extensions.
>>
>> qemu-kvm.git enables KVM by default so this could explain slowness if
>> you've run distro qemu-kvm binaries or qemu-kvm.git in the past.
>>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 10/08/2012 11:22, Stefan Priebe - Profihost AG ha scritto:
> virtio-scsi is now working fine. Could you please help me to get discard
> / trim running? I can't find any information what is needed to get
> discard / trim working.

You need to add discard_granularity=NNN, where NNN is usually 512 for
raw and the cluster size (65536) for qcow2.

However, discard is supported only for XFS on raw, and on qcow2 it will
not reclaim space---the space will only be reused for future qcow2
allocation.

Better support for discard on raw (other filesystems + block devices),
and more efficient support also on other formats is on my todo list for
the future.  However, an efficient implementation unfortunately requires
changes at all levels including the kernel.

I hope to present something about it at KVM Forum.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

I'm using iscsi. So no raw or qcow2. XFS as FS.

Thanks,

Stefan

Am 10.08.2012 12:20, schrieb Paolo Bonzini:
> Il 10/08/2012 11:22, Stefan Priebe - Profihost AG ha scritto:
>> virtio-scsi is now working fine. Could you please help me to get discard
>> / trim running? I can't find any information what is needed to get
>> discard / trim working.
>
> You need to add discard_granularity=NNN, where NNN is usually 512 for
> raw and the cluster size (65536) for qcow2.
>
> However, discard is supported only for XFS on raw, and on qcow2 it will
> not reclaim space---the space will only be reused for future qcow2
> allocation.
>
> Better support for discard on raw (other filesystems + block devices),
> and more efficient support also on other formats is on my todo list for
> the future.  However, an efficient implementation unfortunately requires
> changes at all levels including the kernel.
>
> I hope to present something about it at KVM Forum.
>
> Paolo
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
> I'm using iscsi. So no raw or qcow2.

Ok, then you need to use scsi-block as your device instead of scsi-disk
or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
talk directly to the NAS.

CCing Ronnie who may be interested in bug reports since I'm on holiday
starting "soon".

Paolo

> 
> Thanks,
> 
> Stefan
> 
> Am 10.08.2012 12:20, schrieb Paolo Bonzini:
>> Il 10/08/2012 11:22, Stefan Priebe - Profihost AG ha scritto:
>>> virtio-scsi is now working fine. Could you please help me to get discard
>>> / trim running? I can't find any information what is needed to get
>>> discard / trim working.
>>
>> You need to add discard_granularity=NNN, where NNN is usually 512 for
>> raw and the cluster size (65536) for qcow2.
>>
>> However, discard is supported only for XFS on raw, and on qcow2 it will
>> not reclaim space---the space will only be reused for future qcow2
>> allocation.
>>
>> Better support for discard on raw (other filesystems + block devices),
>> and more efficient support also on other formats is on my todo list for
>> the future.  However, an efficient implementation unfortunately requires
>> changes at all levels including the kernel.
>>
>> I hope to present something about it at KVM Forum.
>>
>> Paolo
>>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[ronnie sahlberg]

You want discard to work?



That should not be a problem with iscsi.

You are using qemu 1.0 ?
So you dont have the qemu support for scsi-generic passthrough to iscsi devices.


This should though work without too much trouble



First you need an iscsi target that supports SBC UNMAP command.
STGT  does support this :   http://stgt.sourceforge.net/
This is a userspace software iscsi target that works on most distros of linux.

Support for thin-provisioning in STGT is very very recent :
    commit 9a855ac0026971c2b5c7f7133febfaf9462410dc
    Author: Ronnie Sahlberg <ronniesahlberg@gmail.com>
    Date:   Sun Apr 15 12:07:32 2012 +1000

    SBC: Add support for thin-provisioning and the UNMAP command

    The UNMAP command is implemented using FALLOC_FL_PUNCH_HOLE and
    release UNMAPPED blocks back to the underlying filesystem.

    FALLOC_FL_PUNCH_HOLE is fairly new addition to Linux but works o
    ext4 and XFS filesystems currently.

    Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
    Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>


That means STGT version 1.0.27 or later.
As this is very recent your distro probably does not have support for this yet,
so you probably want to download, compile and install STGT from the
current git tree.

Thin-provisioning in STGT requires the also very recent addition on
FALLOC_FL_PUNCH_HOLE
(and also SEEK_HOLE/SEEK_DATA if you want "get_lba_status" support)
Because STGT just calls out to these functions.

I think you need to run the target on linux 3.2 or later kernels using
ext4/xfs filessytem for this to work since I dont think any
other filesystems support it. Never tested XFS myself but google
claims it works.


Once you have STGT running on a 3.2 or later kernel and using a
filesystem that supports discard,  this is the command to tell TGTD to
activate
thin-provisioning support for the LUN :

   tgtadm --lld iscsi --mode logicalunit --op update --tid $TID --lun
1 --params thin_provisioning=1

STGT will show the thin provisioning status like this

        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 105 MB, Block size: 512
            Online: Yes
            Removable media: Yes
            Prevent removal: No
            Readonly: No
            Thin-provisioning: Yes
            Backing store type: rdwr
            Backing store path: /100mb.img
            Backing store flags:
...

Once you got STGT up and running and setup for thin provisioning that
should be almost all you need.
(Other iscsi targets may also probably support thin-provisioning but I
have no idea on how to set them up)


Once you have set STGT up,  you just need a guest that supports discard.
Any recent linux distro with a kernel 3.2 or later should do.
I often use latest mint when I test.

Just set it up and put a ext4 filesystem on the iscsi lun, and use the
'discard' mount option in the guest.
Use 'ls -ls <path-to-lun>' on the target and see that the file
'shrinks' in size when  you delete files from the ext4 filesystem.

You can also use wireshark, it understands and decodes the unmap UNMAP
that is used to punch holes in the medium.




NOTE: STGT does not yet support/implement the "thresholds" for
thin-provisioning so there is not yet any mechanism to automatically
inform your guest when the unterlying storage is running low on space.
So you do need to track space utilization on the target backing
filesystem youself!
At some stage I will add the thresholds from sbc 4.7.3.8 but it wont
be anytime soon. (patches are likely welcome)



That should be all you need to do to get it running. It is pretty easy.
Ping me if you have any trouble.



regards
ronnie sahlberg



On Fri, Aug 10, 2012 at 8:30 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
>> I'm using iscsi. So no raw or qcow2.
>
> Ok, then you need to use scsi-block as your device instead of scsi-disk
> or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
> talk directly to the NAS.
>
> CCing Ronnie who may be interested in bug reports since I'm on holiday
> starting "soon".
>
> Paolo
>
>>
>> Thanks,
>>
>> Stefan
>>
>> Am 10.08.2012 12:20, schrieb Paolo Bonzini:
>>> Il 10/08/2012 11:22, Stefan Priebe - Profihost AG ha scritto:
>>>> virtio-scsi is now working fine. Could you please help me to get discard
>>>> / trim running? I can't find any information what is needed to get
>>>> discard / trim working.
>>>
>>> You need to add discard_granularity=NNN, where NNN is usually 512 for
>>> raw and the cluster size (65536) for qcow2.
>>>
>>> However, discard is supported only for XFS on raw, and on qcow2 it will
>>> not reclaim space---the space will only be reused for future qcow2
>>> allocation.
>>>
>>> Better support for discard on raw (other filesystems + block devices),
>>> and more efficient support also on other formats is on my todo list for
>>> the future.  However, an efficient implementation unfortunately requires
>>> changes at all levels including the kernel.
>>>
>>> I hope to present something about it at KVM Forum.
>>>
>>> Paolo
>>>
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 10.08.2012 13:12, schrieb ronnie sahlberg:
> You want discard to work?
Yes

> You are using qemu 1.0 ?
actual qemu-kvm git

> So you dont have the qemu support for scsi-generic passthrough to iscsi devices.
Why?

> I think you need to run the target on linux 3.2 or later kernels using
> ext4/xfs filessytem for this to work since I dont think any
> other filesystems support it. Never tested XFS myself but google
> claims it works.
Host and guestlinux have 3.5.0.

> That should be all you need to do to get it running. It is pretty easy.
> Ping me if you have any trouble.
Thanks to offer help. I'm trying to get it running with Nexentastor.

Greets,
Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[ronnie sahlberg]

On Fri, Aug 10, 2012 at 9:57 PM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Am 10.08.2012 13:12, schrieb ronnie sahlberg:
>
>> You want discard to work?
>
> Yes
>
>
>> You are using qemu 1.0 ?
>
> actual qemu-kvm git
>
>
>> So you dont have the qemu support for scsi-generic passthrough to iscsi
>> devices.
>
> Why?
>

scsi-generic passthrough I think was added for iscsi in 1.1
so in 1.0 your guest will talk scsi to qemu, and invoke the
scsi-emulation in qemu.
It then will call functions like 'bdrv_aio_discard()" in libiscsi
that will translate it back into a scsi command again and pass it to
the target.

It still works, it just means you have a small degradation of
performance compared to if you could send the SCSI CDB straight
through to the iscsi target as you can in qemu 1.1
Very likely so small performance hit that you can not even measure it.


Biggest difference is cosmetic if you run 'sg_inq' in your guest.  in
1.0 it will talk to the qemu scsi emulation layer. in 1.1 when scsi
passthrough is use you will talk to the iscsi target.

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 10.08.2012 14:04, schrieb ronnie sahlberg:
> On Fri, Aug 10, 2012 at 9:57 PM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> Am 10.08.2012 13:12, schrieb ronnie sahlberg:
>>
>>> You want discard to work?
>>
>> Yes
>>
>>
>>> You are using qemu 1.0 ?
>>
>> actual qemu-kvm git
>>
>>
>>> So you dont have the qemu support for scsi-generic passthrough to iscsi
>>> devices.
>>
>> Why?
>>
>
> scsi-generic passthrough I think was added for iscsi in 1.1
> so in 1.0 your guest will talk scsi to qemu, and invoke the
> scsi-emulation in qemu.
> It then will call functions like 'bdrv_aio_discard()" in libiscsi
> that will translate it back into a scsi command again and pass it to
> the target.
>
> It still works, it just means you have a small degradation of
> performance compared to if you could send the SCSI CDB straight
> through to the iscsi target as you can in qemu 1.1
> Very likely so small performance hit that you can not even measure it.

which version are you talking about? I use qemu-kvm.git so this is 
upcomming 1.2 and i use libiscsi 1.5.0.

Greets,
Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[ronnie sahlberg]

On Fri, Aug 10, 2012 at 10:14 PM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Am 10.08.2012 14:04, schrieb ronnie sahlberg:
>
>> On Fri, Aug 10, 2012 at 9:57 PM, Stefan Priebe - Profihost AG
>> <s.priebe@profihost.ag> wrote:
>>>
>>> Am 10.08.2012 13:12, schrieb ronnie sahlberg:
>>>
>>>> You want discard to work?
>>>
>>>
>>> Yes
>>>
>>>
>>>> You are using qemu 1.0 ?
>>>
>>>
>>> actual qemu-kvm git
>>>
>>>
>>>> So you dont have the qemu support for scsi-generic passthrough to iscsi
>>>> devices.
>>>
>>>
>>> Why?
>>>
>>
>> scsi-generic passthrough I think was added for iscsi in 1.1
>> so in 1.0 your guest will talk scsi to qemu, and invoke the
>> scsi-emulation in qemu.
>> It then will call functions like 'bdrv_aio_discard()" in libiscsi
>> that will translate it back into a scsi command again and pass it to
>> the target.
>>
>> It still works, it just means you have a small degradation of
>> performance compared to if you could send the SCSI CDB straight
>> through to the iscsi target as you can in qemu 1.1
>> Very likely so small performance hit that you can not even measure it.
>
>
> which version are you talking about? I use qemu-kvm.git so this is upcomming
> 1.2 and i use libiscsi 1.5.0.

I dont know the kvm version numbers.

But you can check the file
block/iscsi.c for the version you use  for this :

  .bdrv_aio_discard = iscsi_aio_discard,

If it has bdrv_aio_discard then you have support for 'discard' when
using the scsi emulation. i.e.   -drive ...,if=scsi,...



#ifdef __linux__
    .bdrv_ioctl       = iscsi_ioctl,
    .bdrv_aio_ioctl   = iscsi_aio_ioctl,
#endif

If it has these two lines too, then you have scsi-passthrough and can
bypass the qemu scsi emulation.
One way to activate passthough is via scsi-generic:
    Example:
        -device lsi -device scsi-generic,drive=MyISCSI \
        -drive file=iscsi://10.1.1.125/iqn.ronnie.test/1,if=none,id=MyI


regards
ronnie sahlberg

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Am 10.08.2012 14:24, schrieb ronnie sahlberg:
> On Fri, Aug 10, 2012 at 10:14 PM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
> I dont know the kvm version numbers.
They're the same as qemu.

> But you can check the file
> block/iscsi.c for the version you use  for this :
>
>    .bdrv_aio_discard = iscsi_aio_discard,
# grep 'scsi_aio_discard' block/iscsi.c
  iscsi_aio_discard(BlockDriverState *bs,
      .bdrv_aio_discard = iscsi_aio_discard,

=> so yes

> If it has bdrv_aio_discard then you have support for 'discard' when
> using the scsi emulation. i.e.   -drive ...,if=scsi,...
>
> #ifdef __linux__
>      .bdrv_ioctl       = iscsi_ioctl,
>      .bdrv_aio_ioctl   = iscsi_aio_ioctl,
> #endif

yes too.

> If it has these two lines too, then you have scsi-passthrough and can
> bypass the qemu scsi emulation.
> One way to activate passthough is via scsi-generic:
>      Example:
>          -device lsi -device scsi-generic,drive=MyISCSI \
>          -drive file=iscsi://10.1.1.125/iqn.ronnie.test/1,if=none,id=MyI

When i do this the guest system always uses the sym53c8xx kernel module. 
This results in 70 iops instead of 30000 iops. Is this really correct 
that it uses the very old sym53c8xx kernel module for this device?

used start command:
http://pastebin.com/raw.php?i=23fkaQgc

dmesg from guest:
  dmesg|egrep "sym|scsi"
[    0.000000] Linux version 3.5.0intel (root@neuertestswerverscsi) (gcc 
version 4.4.5 (Debian 4.4.5-8) ) #5 SMP Thu Aug 9 20:27:29 CEST 2012
[    0.291949] scsi0 : ata_piix
[    0.292109] scsi1 : ata_piix
[    0.495217] sym0: <895a> rev 0x0 at pci 0000:00:03.0 irq 10
[    0.498694] sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
[    0.500565] sym0: SCSI BUS has been reset.
[    0.508603] scsi2 : sym-2.2.3
[    3.512544] sym0: unknown interrupt(s) ignored, ISTAT=0x5 DSTAT=0x80 
SIST=0x0
[    3.514004] scsi 2:0:0:0: Direct-Access     NEXENTA  NEXENTASTOR 
  1.0  PQ: 0 ANSI: 5
[    3.514414] scsi target2:0:0: tagged command queuing enabled, command 
queue depth 16.
[    3.514820] scsi target2:0:0: Beginning Domain Validation
[    3.518370] scsi target2:0:0: Domain Validation skipping write tests
[    3.518772] scsi target2:0:0: Ending Domain Validation


Thanks a lot!

Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Paolo Bonzini]

Il 10/08/2012 14:35, Stefan Priebe - Profihost AG ha scritto:
>>
>> One way to activate passthough is via scsi-generic:
>>      Example:
>>          -device lsi -device scsi-generic,drive=MyISCSI \
>>          -drive file=iscsi://10.1.1.125/iqn.ronnie.test/1,if=none,id=MyI
> 
> When i do this the guest system always uses the sym53c8xx kernel module.
> This results in 70 iops instead of 30000 iops. Is this really correct
> that it uses the very old sym53c8xx kernel module for this device?
> 
> used start command:
> http://pastebin.com/raw.php?i=23fkaQgc

Just replace lsi with virtio-scsi-pci again.  Also, using scsi-block
instead of scsi-generic should have better performance, but I'm not sure
Ronnie tested it with libiscsi.

Paolo

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Hi Paolo,

Am 10.08.2012 14:39, schrieb Paolo Bonzini:
> Il 10/08/2012 14:35, Stefan Priebe - Profihost AG ha scritto:
>>>
>>> One way to activate passthough is via scsi-generic:
>>>       Example:
>>>           -device lsi -device scsi-generic,drive=MyISCSI \
>>>           -drive file=iscsi://10.1.1.125/iqn.ronnie.test/1,if=none,id=MyI
>>
>> When i do this the guest system always uses the sym53c8xx kernel module.
>> This results in 70 iops instead of 30000 iops. Is this really correct
>> that it uses the very old sym53c8xx kernel module for this device?
>>
>> used start command:
>> http://pastebin.com/raw.php?i=23fkaQgc
>
> Just replace lsi with virtio-scsi-pci again.  Also, using scsi-block
> instead of scsi-generic should have better performance, but I'm not sure
> Ronnie tested it with libiscsi.

I tried this earlier but this results in:

kvm: -device scsi-block,drive=drive-scsi0: scsi-block: INQUIRY failed
kvm: -device scsi-block,drive=drive-scsi0: Device 'scsi-block' could not 
be initialized

Greets,
Stefan

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[ronnie sahlberg]

On Fri, Aug 10, 2012 at 8:30 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
>> I'm using iscsi. So no raw or qcow2.
>
> Ok, then you need to use scsi-block as your device instead of scsi-disk
> or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
> talk directly to the NAS.
>
> CCing Ronnie who may be interested in bug reports since I'm on holiday
> starting "soon".
>

I think it works on any,
You can of course not boot from a if=scsi disk in qemu,

but any '-drive file=iscsi://...,if=scsi' should work as long as it is
not the boot device.

SCSI emulation in qemu picks this up via WRITESAME10/16 and then calls
 bdrv_aio_discard()
block/iscsi.c is invoked for discard and then translates this back to
a SBC UNMAP command it sends to the target.


Now, block/iscsi.c does assume that any target that reports that it
supports thin-provisioning actually implements UNMAP command.
There could be targets that support thin-provision ing that does NOT
support UNMAP and unly support discard via WRITESAME10/16
so at some stage I should send a patch to iscsi.c to check which
commands the target supprots and use one of the supported ones instead
of a blanket
"you say you support thin-provisioning, I take that as confirmation
you support SBC UNMAP"


regards
ronnie sahlberg

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

http://www.nexenta.com/corp/products/what-is-openstorage/nexentastor

tells me:
"SCSI UNMAP as a client-side feature frees up storage in the back end, 
in the context of thin provisioning (a 100-to-one reduction in space for 
VDI when using NexentaStor)."

So i would say nexenta supports it. But i'm using virtio-scsi-pci? I'm 
really sorry to ask so many questions.

Stefan
Am 10.08.2012 13:20, schrieb ronnie sahlberg:
> On Fri, Aug 10, 2012 at 8:30 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
>>> I'm using iscsi. So no raw or qcow2.
>>
>> Ok, then you need to use scsi-block as your device instead of scsi-disk
>> or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
>> talk directly to the NAS.
>>
>> CCing Ronnie who may be interested in bug reports since I'm on holiday
>> starting "soon".
>>
>
> I think it works on any,
> You can of course not boot from a if=scsi disk in qemu,
>
> but any '-drive file=iscsi://...,if=scsi' should work as long as it is
> not the boot device.
>
> SCSI emulation in qemu picks this up via WRITESAME10/16 and then calls
>   bdrv_aio_discard()
> block/iscsi.c is invoked for discard and then translates this back to
> a SBC UNMAP command it sends to the target.
>
>
> Now, block/iscsi.c does assume that any target that reports that it
> supports thin-provisioning actually implements UNMAP command.
> There could be targets that support thin-provision ing that does NOT
> support UNMAP and unly support discard via WRITESAME10/16
> so at some stage I should send a patch to iscsi.c to check which
> commands the target supprots and use one of the supported ones instead
> of a blanket
> "you say you support thin-provisioning, I take that as confirmation
> you support SBC UNMAP"
>
>
> regards
> ronnie sahlberg
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[ronnie sahlberg]

You can easily verify if your target supports thin-provisioning via
the UNMAP command.

Download the sg3-utils package
 and either mount the LUN locally via the kernel open-iscsi or apply
the libiscsi patch to sg3-utils to make it iscsi-aware

then use the commands    "sg_unmap"  to try to unmap regions   and
"sg_get_lba_status" to check that the regions are now unmapped.



On Fri, Aug 10, 2012 at 9:54 PM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> http://www.nexenta.com/corp/products/what-is-openstorage/nexentastor
>
> tells me:
> "SCSI UNMAP as a client-side feature frees up storage in the back end, in
> the context of thin provisioning (a 100-to-one reduction in space for VDI
> when using NexentaStor)."
>
> So i would say nexenta supports it. But i'm using virtio-scsi-pci? I'm
> really sorry to ask so many questions.
>
> Stefan
> Am 10.08.2012 13:20, schrieb ronnie sahlberg:
>
>> On Fri, Aug 10, 2012 at 8:30 PM, Paolo Bonzini <pbonzini@redhat.com>
>> wrote:
>>>
>>> Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
>>>>
>>>> I'm using iscsi. So no raw or qcow2.
>>>
>>>
>>> Ok, then you need to use scsi-block as your device instead of scsi-disk
>>> or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
>>> talk directly to the NAS.
>>>
>>> CCing Ronnie who may be interested in bug reports since I'm on holiday
>>> starting "soon".
>>>
>>
>> I think it works on any,
>> You can of course not boot from a if=scsi disk in qemu,
>>
>> but any '-drive file=iscsi://...,if=scsi' should work as long as it is
>> not the boot device.
>>
>> SCSI emulation in qemu picks this up via WRITESAME10/16 and then calls
>>   bdrv_aio_discard()
>> block/iscsi.c is invoked for discard and then translates this back to
>> a SBC UNMAP command it sends to the target.
>>
>>
>> Now, block/iscsi.c does assume that any target that reports that it
>> supports thin-provisioning actually implements UNMAP command.
>> There could be targets that support thin-provision ing that does NOT
>> support UNMAP and unly support discard via WRITESAME10/16
>> so at some stage I should send a patch to iscsi.c to check which
>> commands the target supprots and use one of the supported ones instead
>> of a blanket
>> "you say you support thin-provisioning, I take that as confirmation
>> you support SBC UNMAP"
>>
>>
>> regards
>> ronnie sahlberg
>>
>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

never used this tool. Output is:
This looks like this:
# sg_unmap --lba=0x1000 --num=1 /dev/sde

# sg_get_lba_status --lba=0x1000 /dev/sde
get LBA status: transport: Host_status=0x10 is invalid
Driver_status=0x08 [DRIVER_SENSE, SUGGEST_OK]

Get LBA Status command failed
     try '-v' option for more information

Stefan
Am 10.08.2012 13:58, schrieb ronnie sahlberg:
> You can easily verify if your target supports thin-provisioning via
> the UNMAP command.
>
> Download the sg3-utils package
>   and either mount the LUN locally via the kernel open-iscsi or apply
> the libiscsi patch to sg3-utils to make it iscsi-aware
>
> then use the commands    "sg_unmap"  to try to unmap regions   and
> "sg_get_lba_status" to check that the regions are now unmapped.
>
>
>
> On Fri, Aug 10, 2012 at 9:54 PM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> http://www.nexenta.com/corp/products/what-is-openstorage/nexentastor
>>
>> tells me:
>> "SCSI UNMAP as a client-side feature frees up storage in the back end, in
>> the context of thin provisioning (a 100-to-one reduction in space for VDI
>> when using NexentaStor)."
>>
>> So i would say nexenta supports it. But i'm using virtio-scsi-pci? I'm
>> really sorry to ask so many questions.
>>
>> Stefan
>> Am 10.08.2012 13:20, schrieb ronnie sahlberg:
>>
>>> On Fri, Aug 10, 2012 at 8:30 PM, Paolo Bonzini <pbonzini@redhat.com>
>>> wrote:
>>>>
>>>> Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
>>>>>
>>>>> I'm using iscsi. So no raw or qcow2.
>>>>
>>>>
>>>> Ok, then you need to use scsi-block as your device instead of scsi-disk
>>>> or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
>>>> talk directly to the NAS.
>>>>
>>>> CCing Ronnie who may be interested in bug reports since I'm on holiday
>>>> starting "soon".
>>>>
>>>
>>> I think it works on any,
>>> You can of course not boot from a if=scsi disk in qemu,
>>>
>>> but any '-drive file=iscsi://...,if=scsi' should work as long as it is
>>> not the boot device.
>>>
>>> SCSI emulation in qemu picks this up via WRITESAME10/16 and then calls
>>>    bdrv_aio_discard()
>>> block/iscsi.c is invoked for discard and then translates this back to
>>> a SBC UNMAP command it sends to the target.
>>>
>>>
>>> Now, block/iscsi.c does assume that any target that reports that it
>>> supports thin-provisioning actually implements UNMAP command.
>>> There could be targets that support thin-provision ing that does NOT
>>> support UNMAP and unly support discard via WRITESAME10/16
>>> so at some stage I should send a patch to iscsi.c to check which
>>> commands the target supprots and use one of the supported ones instead
>>> of a blanket
>>> "you say you support thin-provisioning, I take that as confirmation
>>> you support SBC UNMAP"
>>>
>>>
>>> regards
>>> ronnie sahlberg
>>>
>>

Re: [Qemu-devel] virtio-scsi vs. virtio-blk

[Stefan Priebe - Profihost AG]

Hi,

i tried that but i then get:
kvm: -device 
scsi-block,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0: 
scsi-block: INQUIRY failed
kvm: -device 
scsi-block,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0: 
Device 'scsi-block' could not be initialized

VM start command was:
http://pastebin.com/raw.php?i=6WNLPemy

Stefan
Am 10.08.2012 12:30, schrieb Paolo Bonzini:
> Il 10/08/2012 12:28, Stefan Priebe - Profihost AG ha scritto:
>> I'm using iscsi. So no raw or qcow2.
>
> Ok, then you need to use scsi-block as your device instead of scsi-disk
> or scsi-hd.  This will disable the QEMU SCSI emulation and let your VM
> talk directly to the NAS.
>
> CCing Ronnie who may be interested in bug reports since I'm on holiday
> starting "soon".
>
> Paolo
>
>>
>> Thanks,
>>
>> Stefan
>>
>> Am 10.08.2012 12:20, schrieb Paolo Bonzini:
>>> Il 10/08/2012 11:22, Stefan Priebe - Profihost AG ha scritto:
>>>> virtio-scsi is now working fine. Could you please help me to get discard
>>>> / trim running? I can't find any information what is needed to get
>>>> discard / trim working.
>>>
>>> You need to add discard_granularity=NNN, where NNN is usually 512 for
>>> raw and the cluster size (65536) for qcow2.
>>>
>>> However, discard is supported only for XFS on raw, and on qcow2 it will
>>> not reclaim space---the space will only be reused for future qcow2
>>> allocation.
>>>
>>> Better support for discard on raw (other filesystems + block devices),
>>> and more efficient support also on other formats is on my todo list for
>>> the future.  However, an efficient implementation unfortunately requires
>>> changes at all levels including the kernel.
>>>
>>> I hope to present something about it at KVM Forum.
>>>
>>> Paolo
>>>
>