* [Qemu-devel] is it possible to use a disk with multiple iothreads ? [not found] <391d6bb1-e0ad-49ea-afdc-fff462081f24@mailpro> @ 2014-10-23 18:26 ` Alexandre DERUMIER 2014-10-24 9:04 ` Stefan Hajnoczi 0 siblings, 1 reply; 7+ messages in thread From: Alexandre DERUMIER @ 2014-10-23 18:26 UTC (permalink / raw) To: stefanha; +Cc: qemu-devel Hi, I was reading this interesting presentation, http://vmsplice.net/~stefan/stefanha-kvm-forum-2014.pdf and I have a specific question. I'm currently evaluate ceph/rbd storage performance through qemu, and the current bottleneck seem to be the cpu usage of the iothread. (rbd procotol cpu usage is really huge). Currently It's around 10000iops on 1core of recent xeons. So, I had tried with virtio-scsi multiqueue, but it doesn't help. (Not sure, but I think that all queues are mapped on 1 iothread ?) I had tried with with virtio-blk and iothread/dataplane, it's help a little , but not too much too. So, my question is : Is it possible to use multiple iothreads with 1 disk ? (something like 1 queue - 1 iothread?) Regards, Alexandre ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] is it possible to use a disk with multiple iothreads ? 2014-10-23 18:26 ` [Qemu-devel] is it possible to use a disk with multiple iothreads ? Alexandre DERUMIER @ 2014-10-24 9:04 ` Stefan Hajnoczi 2014-10-24 13:39 ` Alexandre DERUMIER 0 siblings, 1 reply; 7+ messages in thread From: Stefan Hajnoczi @ 2014-10-24 9:04 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: josh.durgin, qemu-devel [-- Attachment #1: Type: text/plain, Size: 1256 bytes --] On Thu, Oct 23, 2014 at 08:26:17PM +0200, Alexandre DERUMIER wrote: > I was reading this interesting presentation, > > http://vmsplice.net/~stefan/stefanha-kvm-forum-2014.pdf > > and I have a specific question. > > > I'm currently evaluate ceph/rbd storage performance through qemu, > > and the current bottleneck seem to be the cpu usage of the iothread. > (rbd procotol cpu usage is really huge). > > Currently It's around 10000iops on 1core of recent xeons. > > > > So, I had tried with virtio-scsi multiqueue, but it doesn't help. (Not sure, but I think that all queues are mapped on 1 iothread ?) > > I had tried with with virtio-blk and iothread/dataplane, it's help a little , but not too much too. > > So, my question is : Is it possible to use multiple iothreads with 1 disk ? > > (something like 1 queue - 1 iothread?) virtio-blk and virtio-scsi emulation only runs in 1 thread at a time. It is currently not possible to achieve true multiqueue from guest, through QEMU, and down to the host. This is what the final slides in my presentation were about. Regarding Ceph, do you know why it burns a lot of CPU? CCed Josh in case he has ideas for reducing Ceph library CPU utilization. Stefan [-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] is it possible to use a disk with multiple iothreads ? 2014-10-24 9:04 ` Stefan Hajnoczi @ 2014-10-24 13:39 ` Alexandre DERUMIER 2014-10-27 13:19 ` Stefan Hajnoczi 0 siblings, 1 reply; 7+ messages in thread From: Alexandre DERUMIER @ 2014-10-24 13:39 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: josh durgin, qemu-devel >>virtio-blk and virtio-scsi emulation only runs in 1 thread at a time. >>It is currently not possible to achieve true multiqueue from guest, >>through QEMU, and down to the host. >> >>This is what the final slides in my presentation were about. Ok Thanks ! >>Regarding Ceph, do you know why it burns a lot of CPU? I'm currently doing some perfs to find why is so huge. One thing is that qemu+ qemu rbd block driver + librbd seem to use 2-3x more cpu, than passing krbd /dev/rbd0 from host to qemu. Don't known yet if the problem is librbd or qemu rbd block driver. ----- Mail original ----- De: "Stefan Hajnoczi" <stefanha@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: "qemu-devel" <qemu-devel@nongnu.org>, "josh durgin" <josh.durgin@inktank.com> Envoyé: Vendredi 24 Octobre 2014 11:04:06 Objet: Re: is it possible to use a disk with multiple iothreads ? On Thu, Oct 23, 2014 at 08:26:17PM +0200, Alexandre DERUMIER wrote: > I was reading this interesting presentation, > > http://vmsplice.net/~stefan/stefanha-kvm-forum-2014.pdf > > and I have a specific question. > > > I'm currently evaluate ceph/rbd storage performance through qemu, > > and the current bottleneck seem to be the cpu usage of the iothread. > (rbd procotol cpu usage is really huge). > > Currently It's around 10000iops on 1core of recent xeons. > > > > So, I had tried with virtio-scsi multiqueue, but it doesn't help. (Not sure, but I think that all queues are mapped on 1 iothread ?) > > I had tried with with virtio-blk and iothread/dataplane, it's help a little , but not too much too. > > So, my question is : Is it possible to use multiple iothreads with 1 disk ? > > (something like 1 queue - 1 iothread?) virtio-blk and virtio-scsi emulation only runs in 1 thread at a time. It is currently not possible to achieve true multiqueue from guest, through QEMU, and down to the host. This is what the final slides in my presentation were about. Regarding Ceph, do you know why it burns a lot of CPU? CCed Josh in case he has ideas for reducing Ceph library CPU utilization. Stefan ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] is it possible to use a disk with multiple iothreads ? 2014-10-24 13:39 ` Alexandre DERUMIER @ 2014-10-27 13:19 ` Stefan Hajnoczi 2014-10-27 14:13 ` Alexandre DERUMIER 0 siblings, 1 reply; 7+ messages in thread From: Stefan Hajnoczi @ 2014-10-27 13:19 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: josh durgin, qemu-devel [-- Attachment #1: Type: text/plain, Size: 505 bytes --] On Fri, Oct 24, 2014 at 03:39:52PM +0200, Alexandre DERUMIER wrote: > >>Regarding Ceph, do you know why it burns a lot of CPU? > > I'm currently doing some perfs to find why is so huge. > > One thing is that qemu+ qemu rbd block driver + librbd seem to use 2-3x more cpu, than passing krbd /dev/rbd0 from host to qemu. > Don't known yet if the problem is librbd or qemu rbd block driver. That's very interesting! Please keep Josh and me in CC when you want to discuss the results. Stefan [-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] is it possible to use a disk with multiple iothreads ? 2014-10-27 13:19 ` Stefan Hajnoczi @ 2014-10-27 14:13 ` Alexandre DERUMIER 2014-10-27 14:47 ` Paolo Bonzini 0 siblings, 1 reply; 7+ messages in thread From: Alexandre DERUMIER @ 2014-10-27 14:13 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: josh durgin, qemu-devel [-- Attachment #1: Type: text/plain, Size: 3813 bytes --] >>That's very interesting! Please keep Josh and me in CC when you want to >>discuss the results. Here the aggregate results (perf report details attached in this mail) + 33,02% 3974368751 kvm [kernel.kallsyms] + 23,66% 2847206635 kvm libc-2.13.so + 18,79% 2262052133 kvm librados.so.2.0.0 + 11,04% 1328581527 kvm librbd.so.1.0.0 + 5,87% 706713737 kvm libpthread-2.13.so + 3,75% 451690142 kvm kvm + 2,74% 329457334 kvm libstdc++.so.6.0.17 + 0,51% 61519664 kvm [vdso] + 0,42% 50000089 kvm libglib-2.0.so.0.3200.4 + 0,15% 18119658 kvm libm-2.13.so + 0,05% 5705776 kvm librt-2.13.so + 0,00% 356625 kvm libz.so.1.2.7 >> 23,66% 2847206635 kvm libc-2.13.so This one is mostly malloc,free,... I see almost same results using fio with rbdengine on the host (outside the kvm process). So I think they are all mostly related to librbd. ----- Mail original ----- De: "Stefan Hajnoczi" <stefanha@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: "qemu-devel" <qemu-devel@nongnu.org>, "josh durgin" <josh.durgin@inktank.com> Envoyé: Lundi 27 Octobre 2014 14:19:58 Objet: Re: is it possible to use a disk with multiple iothreads ? On Fri, Oct 24, 2014 at 03:39:52PM +0200, Alexandre DERUMIER wrote: > >>Regarding Ceph, do you know why it burns a lot of CPU? > > I'm currently doing some perfs to find why is so huge. > > One thing is that qemu+ qemu rbd block driver + librbd seem to use 2-3x more cpu, than passing krbd /dev/rbd0 from host to qemu. > Don't known yet if the problem is librbd or qemu rbd block driver. That's very interesting! Please keep Josh and me in CC when you want to discuss the results. Stefan [-- Attachment #2: report.txt.gz --] [-- Type: application/x-gzip, Size: 111472 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] is it possible to use a disk with multiple iothreads ? 2014-10-27 14:13 ` Alexandre DERUMIER @ 2014-10-27 14:47 ` Paolo Bonzini 2014-10-27 16:42 ` Alexandre DERUMIER 0 siblings, 1 reply; 7+ messages in thread From: Paolo Bonzini @ 2014-10-27 14:47 UTC (permalink / raw) To: Alexandre DERUMIER, Stefan Hajnoczi; +Cc: josh durgin, qemu-devel On 10/27/2014 03:13 PM, Alexandre DERUMIER wrote: >>> That's very interesting! Please keep Josh and me in CC when you want to >>> >>discuss the results. > Here the aggregate results (perf report details attached in this mail) > > + 33,02% 3974368751 kvm [kernel.kallsyms] > + 23,66% 2847206635 kvm libc-2.13.so > + 18,79% 2262052133 kvm librados.so.2.0.0 > + 11,04% 1328581527 kvm librbd.so.1.0.0 > + 5,87% 706713737 kvm libpthread-2.13.so > + 3,75% 451690142 kvm kvm > + 2,74% 329457334 kvm libstdc++.so.6.0.17 > + 0,51% 61519664 kvm [vdso] + 0,42% 50000089 kvm libglib-2.0.so.0.3200.4 + 0,15% 18119658 kvm libm-2.13.so + 0,05% 5705776 kvm librt-2.13.so + 0,00% 356625 kvm libz.so.1.2.7 > > > > >>> >> 23,66% 2847206635 kvm libc-2.13.so > This one is mostly malloc,free,... > > > I see almost same results using fio with rbdengine on the host (outside the kvm process). > So I think they are all mostly related to librbd. You are missing debug information unfortunately, but the slowdown seems to be related to futexes. Paolo ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] is it possible to use a disk with multiple iothreads ? 2014-10-27 14:47 ` Paolo Bonzini @ 2014-10-27 16:42 ` Alexandre DERUMIER 0 siblings, 0 replies; 7+ messages in thread From: Alexandre DERUMIER @ 2014-10-27 16:42 UTC (permalink / raw) To: Paolo Bonzini; +Cc: josh durgin, qemu-devel, Stefan Hajnoczi >>You are missing debug information unfortunately, Ok thanks, I'll try to add qemu debug symbols. (I have already libc6,librbd,librados debug symbols installed) ----- Mail original ----- De: "Paolo Bonzini" <pbonzini@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com>, "Stefan Hajnoczi" <stefanha@redhat.com> Cc: "josh durgin" <josh.durgin@inktank.com>, "qemu-devel" <qemu-devel@nongnu.org> Envoyé: Lundi 27 Octobre 2014 15:47:19 Objet: Re: is it possible to use a disk with multiple iothreads ? On 10/27/2014 03:13 PM, Alexandre DERUMIER wrote: >>> That's very interesting! Please keep Josh and me in CC when you want to >>> >>discuss the results. > Here the aggregate results (perf report details attached in this mail) > > + 33,02% 3974368751 kvm [kernel.kallsyms] > + 23,66% 2847206635 kvm libc-2.13.so > + 18,79% 2262052133 kvm librados.so.2.0.0 > + 11,04% 1328581527 kvm librbd.so.1.0.0 > + 5,87% 706713737 kvm libpthread-2.13.so > + 3,75% 451690142 kvm kvm > + 2,74% 329457334 kvm libstdc++.so.6.0.17 > + 0,51% 61519664 kvm [vdso] + 0,42% 50000089 kvm libglib-2.0.so.0.3200.4 + 0,15% 18119658 kvm libm-2.13.so + 0,05% 5705776 kvm librt-2.13.so + 0,00% 356625 kvm libz.so.1. 2.7 > > > > >>> >> 23,66% 2847206635 kvm libc-2.13.so > This one is mostly malloc,free,... > > > I see almost same results using fio with rbdengine on the host (outside the kvm process). > So I think they are all mostly related to librbd. You are missing debug information unfortunately, but the slowdown seems to be related to futexes. Paolo ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-10-27 16:42 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <391d6bb1-e0ad-49ea-afdc-fff462081f24@mailpro>
2014-10-23 18:26 ` [Qemu-devel] is it possible to use a disk with multiple iothreads ? Alexandre DERUMIER
2014-10-24 9:04 ` Stefan Hajnoczi
2014-10-24 13:39 ` Alexandre DERUMIER
2014-10-27 13:19 ` Stefan Hajnoczi
2014-10-27 14:13 ` Alexandre DERUMIER
2014-10-27 14:47 ` Paolo Bonzini
2014-10-27 16:42 ` Alexandre DERUMIER
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).