* fio read and randread cpu usage results for qemu and host machine
[not found] <ff0167fd-ba60-48d2-b735-63569a5a6dd4@mailpro>
@ 2014-10-23 12:30 ` Alexandre DERUMIER
2014-10-23 13:07 ` Mark Nelson
0 siblings, 1 reply; 4+ messages in thread
From: Alexandre DERUMIER @ 2014-10-23 12:30 UTC (permalink / raw)
To: Ceph Devel
Hi,
I have done fio tests on multiple qemu setups and host machine,
to see if librbd use more cpu than krbd
and also find the best qemu optimisations.
Fio test was done with blocksize=4K , read and randread with num_jobs=32.
1) First, I have done more tests with qemu optimisation, like iothread (dataplane) for virtio-blk disk,
and num_queue=8 for virtio-scsi disks.
for virtio iothread:
--------------------
It's working only with qemu + krbd, and for sequential reads.
And It's seem that qemu aggregate reads in bigger ceph reads. (I see 4x more iops on fio than on ceph, with same bandwith)
for virtio-scsi num_queue = 8:
-------------------------------
works with krbd and librbd
-for random read : I jump from 7000 to 12000iops
-for sequential, qemu aggreate reads in bigger ceph reads. (Same, I see 4xmore iops on fio than on ceph, with same bandwith).
So it seem to be useful for some specific workloads.
2 ) Now, about cpu usage, it's seem than librbd use really more cpu than krbd,
on host, librbd use 4x more cpu than krbd
on qemu, librbd use 2x more cpu than krbd
So, what could explain so much difference between both ?
Regards,
Alexandre
fio iops seq read summary results
----------------------------------
qemu virtio iothread krbd vs qemu virtio iothread librbd : 27000 iops vs 15000 ipos
qemu virtio krbd vs qemu virtio librbd : 19000 iops vs 15000 iops
qemu virtio-scsi krbd vs qemu virtio librbd : 50000 iops vs 48000 iops
host krbd vs host librbd : 36000 iops vs 25000 iops
fio iops randread summary results
------------------------------
qemu virtio iothread krbd vs qemu virtio iothread librbd : 15000 iops vs 14000 iops
qemu virtio krbd vs qemu virtio librbd : 14000 iops vs 15000 iops
qemu virtio-scsi krbd vs qemu virtio librbd : 7500 iops vs 12000 iops
host krbd vs host librbd : 38000 iops vs 25000 iops
cpu usage ratio summary
------------------------
qemu virtio krbd vs qemu virtio librbd : 2x more cpu usage for librbd
qemu virtio-scsi krbd vs qemu virtio-scsi librbd : 2x more cpu usage for librbd
host krbd vs host librbd : 4x more cpu usage for fio-rbd
RESULTS
-------
host + fio - krbd
-------------------
read
-----
fio iops : 142.9MB/Ss , 36000 iops
ceph : 134 MB/s rd, 34319 op/s
fio : 70,4% kworker : 93,9% cpu = 164% cpu
100%cpu : 21000iops
randread
--------
fio: 151MB/S,38000 iops
ceph :148 MB/s rd, 37932 op/s
fio : 80%cpu kwoker : 110,3%cpu = 180% cpu
100%cpu : 21000 iops
host + fio-rbdengin :
---------------------
randread (cpu bound)
--------------------
fio iops : 25000 ops
ceph iops : 99636 kB/s rd, 24909 op/s
fio : 460%cpu
100%cpu : 5415iops
read (cpu bound)
-----------------
fio iops : 25000 ops
ceph ios : 94212 kB/s rd, 23553 op/s
fio : 480%cpu
100%cpu : 5323iops
qemu + krbd + virtio + iothread
---------------------------------
read
----
fio :107MB/S : 27000 iops >>>SEEM THAT QEMU AGGREGATE READS ops
ceph : 93942 kB/s rd, 12430 op/s
kvm: 130% cpu - kworker : 41,2% = 171,2% cpu
100%cpu ratio : 7260iops
randread
--------
fio : 60MBS - 15000 iops
ceph : 54400 kB/s rd, 13600 op/s
kvm: 95,0% cpu - kworker : 42,1 % cpu = 137,1%cpu
100%cpu ratio : 9919 iops
qemu + krbd + virtio
----------------------
read
-----
fio : 70mbs/ , 19000iops
ceph:75705 kB/s rd, 18926 op/s
kvm : 164% cpu - kworker : 48,5% cpu = 212,5%cpu
100%cpu ratio : 8906 iops
randread
--------
fio : 54mbs/ , 14000iops
ceph : 54800 kB/s rd, 13700 op/s
kvm: 103% cpu - kworker 41,2% cpu = 144,2%cpu
100%cpu ratio : 9513 iops
qemu + krbd + virtio-scsi (num_queue 8)
--------------------------------------
read:
----
fio : 200MB / 50000 iops >>>SEEM THAT QEMU AGGREGATE READS ops
ceph : 205 MB/s rd, 7648 op/s
kvm: 145% kworker : 46,5% = 191,5%cpu
100%cpu ratio : 3993 iops
randread:
----------
fio : 30MB/S / 7500 iops
ceph : 29318 kB/s rd, 7329 op/s
kvm : 150% kworker : 21,4% cpu = 171,4% cpu
100%cpu ratio : 4275 iops
qemu + librbd + virtio + iothread
----------------------------------
read
----
fio : 60MBS/s , 15000iops
ceph: 56199 kB/s rd, 14052 op/s
kvm: 300% cpu
100%cpu : 4666iops
randread
---------
fio : 56MBS/s, 14000iops
ceph : 55916 kB/s rd, 13979 op/s
kvm: 300% cpu
100%cpu : 4659 iops
qemu + librbd + virtio
-------------------------
read
-----
fio : 60MBS/s, 15000iops
ceph : 63021 kB/s rd, 15755 op/s
kvm: 300% cpu
100%cpu : 5233 iops
randread
--------
fio : 60MBS/s, 15000iops
ceph : 55916 kB/s rd, 13979 op/s
kvm : 300%cpu
100%cpu : 4659 iops
qemu + librbd + virtio-scsi (num_queue 8)
----------------------------------------
read
----
fio : 256 MB/S , 48000iops >>>SEEM THAT QEMU AGGREGATE READS ops
ceph : 244 MB/s rd, 12002 op/s
kvm : 300% cpu
100%cpu : 4000 iops
randread
--------
fio: 12000iops
ceph iops : 47511 kB/s rd, 11877 op/s
kvm: 300% cpu
100%cpu: 3959 iops
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: fio read and randread cpu usage results for qemu and host machine
2014-10-23 12:30 ` fio read and randread cpu usage results for qemu and host machine Alexandre DERUMIER
@ 2014-10-23 13:07 ` Mark Nelson
2014-10-23 15:20 ` Alexandre DERUMIER
0 siblings, 1 reply; 4+ messages in thread
From: Mark Nelson @ 2014-10-23 13:07 UTC (permalink / raw)
To: Alexandre DERUMIER, Ceph Devel
On 10/23/2014 07:30 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I have done fio tests on multiple qemu setups and host machine,
> to see if librbd use more cpu than krbd
> and also find the best qemu optimisations.
>
> Fio test was done with blocksize=4K , read and randread with num_jobs=32.
>
>
>
>
>
> 1) First, I have done more tests with qemu optimisation, like iothread (dataplane) for virtio-blk disk,
> and num_queue=8 for virtio-scsi disks.
>
> for virtio iothread:
> --------------------
> It's working only with qemu + krbd, and for sequential reads.
> And It's seem that qemu aggregate reads in bigger ceph reads. (I see 4x more iops on fio than on ceph, with same bandwith)
>
>
> for virtio-scsi num_queue = 8:
> -------------------------------
> works with krbd and librbd
>
> -for random read : I jump from 7000 to 12000iops
> -for sequential, qemu aggreate reads in bigger ceph reads. (Same, I see 4xmore iops on fio than on ceph, with same bandwith).
>
>
> So it seem to be useful for some specific workloads.
>
>
>
> 2 ) Now, about cpu usage, it's seem than librbd use really more cpu than krbd,
>
> on host, librbd use 4x more cpu than krbd
> on qemu, librbd use 2x more cpu than krbd
>
>
> So, what could explain so much difference between both ?
Hi Alexandre, do you have access to perf? Especially on newer
kernels/distributions where you can use dwarf symbols, perf can give you
a lot of information about what is using CPU where. vtune is also a
nice option if you ave a license.
>
>
> Regards,
>
> Alexandre
>
>
>
> fio iops seq read summary results
> ----------------------------------
> qemu virtio iothread krbd vs qemu virtio iothread librbd : 27000 iops vs 15000 ipos
> qemu virtio krbd vs qemu virtio librbd : 19000 iops vs 15000 iops
> qemu virtio-scsi krbd vs qemu virtio librbd : 50000 iops vs 48000 iops
> host krbd vs host librbd : 36000 iops vs 25000 iops
>
>
>
> fio iops randread summary results
> ------------------------------
> qemu virtio iothread krbd vs qemu virtio iothread librbd : 15000 iops vs 14000 iops
> qemu virtio krbd vs qemu virtio librbd : 14000 iops vs 15000 iops
> qemu virtio-scsi krbd vs qemu virtio librbd : 7500 iops vs 12000 iops
> host krbd vs host librbd : 38000 iops vs 25000 iops
>
>
> cpu usage ratio summary
> ------------------------
> qemu virtio krbd vs qemu virtio librbd : 2x more cpu usage for librbd
> qemu virtio-scsi krbd vs qemu virtio-scsi librbd : 2x more cpu usage for librbd
> host krbd vs host librbd : 4x more cpu usage for fio-rbd
>
>
>
>
>
>
> RESULTS
> -------
>
> host + fio - krbd
> -------------------
> read
> -----
> fio iops : 142.9MB/Ss , 36000 iops
> ceph : 134 MB/s rd, 34319 op/s
>
> fio : 70,4% kworker : 93,9% cpu = 164% cpu
>
> 100%cpu : 21000iops
>
> randread
> --------
> fio: 151MB/S,38000 iops
> ceph :148 MB/s rd, 37932 op/s
>
> fio : 80%cpu kwoker : 110,3%cpu = 180% cpu
>
> 100%cpu : 21000 iops
>
>
>
>
> host + fio-rbdengin :
> ---------------------
> randread (cpu bound)
> --------------------
> fio iops : 25000 ops
> ceph iops : 99636 kB/s rd, 24909 op/s
>
> fio : 460%cpu
>
> 100%cpu : 5415iops
>
> read (cpu bound)
> -----------------
> fio iops : 25000 ops
> ceph ios : 94212 kB/s rd, 23553 op/s
>
> fio : 480%cpu
>
> 100%cpu : 5323iops
>
>
>
>
>
> qemu + krbd + virtio + iothread
> ---------------------------------
> read
> ----
> fio :107MB/S : 27000 iops >>>SEEM THAT QEMU AGGREGATE READS ops
> ceph : 93942 kB/s rd, 12430 op/s
>
> kvm: 130% cpu - kworker : 41,2% = 171,2% cpu
>
> 100%cpu ratio : 7260iops
>
> randread
> --------
> fio : 60MBS - 15000 iops
> ceph : 54400 kB/s rd, 13600 op/s
>
> kvm: 95,0% cpu - kworker : 42,1 % cpu = 137,1%cpu
>
> 100%cpu ratio : 9919 iops
>
>
>
>
>
> qemu + krbd + virtio
> ----------------------
> read
> -----
> fio : 70mbs/ , 19000iops
> ceph:75705 kB/s rd, 18926 op/s
> kvm : 164% cpu - kworker : 48,5% cpu = 212,5%cpu
>
> 100%cpu ratio : 8906 iops
>
> randread
> --------
> fio : 54mbs/ , 14000iops
> ceph : 54800 kB/s rd, 13700 op/s
> kvm: 103% cpu - kworker 41,2% cpu = 144,2%cpu
>
> 100%cpu ratio : 9513 iops
>
>
>
> qemu + krbd + virtio-scsi (num_queue 8)
> --------------------------------------
> read:
> ----
> fio : 200MB / 50000 iops >>>SEEM THAT QEMU AGGREGATE READS ops
> ceph : 205 MB/s rd, 7648 op/s
>
> kvm: 145% kworker : 46,5% = 191,5%cpu
>
> 100%cpu ratio : 3993 iops
>
> randread:
> ----------
> fio : 30MB/S / 7500 iops
> ceph : 29318 kB/s rd, 7329 op/s
> kvm : 150% kworker : 21,4% cpu = 171,4% cpu
>
> 100%cpu ratio : 4275 iops
>
>
>
>
> qemu + librbd + virtio + iothread
> ----------------------------------
> read
> ----
> fio : 60MBS/s , 15000iops
> ceph: 56199 kB/s rd, 14052 op/s
>
> kvm: 300% cpu
>
> 100%cpu : 4666iops
>
>
> randread
> ---------
> fio : 56MBS/s, 14000iops
> ceph : 55916 kB/s rd, 13979 op/s
>
> kvm: 300% cpu
>
> 100%cpu : 4659 iops
>
>
>
> qemu + librbd + virtio
> -------------------------
> read
> -----
> fio : 60MBS/s, 15000iops
> ceph : 63021 kB/s rd, 15755 op/s
>
> kvm: 300% cpu
>
> 100%cpu : 5233 iops
>
> randread
> --------
> fio : 60MBS/s, 15000iops
> ceph : 55916 kB/s rd, 13979 op/s
>
> kvm : 300%cpu
>
> 100%cpu : 4659 iops
>
>
> qemu + librbd + virtio-scsi (num_queue 8)
> ----------------------------------------
> read
> ----
> fio : 256 MB/S , 48000iops >>>SEEM THAT QEMU AGGREGATE READS ops
> ceph : 244 MB/s rd, 12002 op/s
>
> kvm : 300% cpu
>
> 100%cpu : 4000 iops
>
> randread
> --------
> fio: 12000iops
> ceph iops : 47511 kB/s rd, 11877 op/s
>
> kvm: 300% cpu
>
> 100%cpu: 3959 iops
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: fio read and randread cpu usage results for qemu and host machine
2014-10-23 13:07 ` Mark Nelson
@ 2014-10-23 15:20 ` Alexandre DERUMIER
2014-10-23 21:38 ` Alexandre DERUMIER
0 siblings, 1 reply; 4+ messages in thread
From: Alexandre DERUMIER @ 2014-10-23 15:20 UTC (permalink / raw)
To: Mark Nelson; +Cc: Ceph Devel
>>Hi Alexandre, do you have access to perf?
yes. I can send perf report if you want. (I'm not sure how to analyze the results)
>>Especially on newer kernels/distributions where you can use dwarf symbols, perf can give you
>>a lot of information about what is using CPU where.
I'm using kernel 3.16, so I think It'll be ok
>> vtune is also a nice option if you ave a license.
Don't have it sorry.
BTW, about seq read results with virtio iothread and virtio-scsi multiqueue
it's seem that it's enabling linux queues merging
I had try to disable it with echo 2 > /sys/block/sda/queue/nomerges,
and I have the same result than without multiqueue.
So it's more an optimization for qemu,
but this not improving the reals maximum ios we can reach from rbd.
----- Mail original -----
De: "Mark Nelson" <mark.nelson@inktank.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>, "Ceph Devel" <ceph-devel@vger.kernel.org>
Envoyé: Jeudi 23 Octobre 2014 15:07:29
Objet: Re: fio read and randread cpu usage results for qemu and host machine
On 10/23/2014 07:30 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I have done fio tests on multiple qemu setups and host machine,
> to see if librbd use more cpu than krbd
> and also find the best qemu optimisations.
>
> Fio test was done with blocksize=4K , read and randread with num_jobs=32.
>
>
>
>
>
> 1) First, I have done more tests with qemu optimisation, like iothread (dataplane) for virtio-blk disk,
> and num_queue=8 for virtio-scsi disks.
>
> for virtio iothread:
> --------------------
> It's working only with qemu + krbd, and for sequential reads.
> And It's seem that qemu aggregate reads in bigger ceph reads. (I see 4x more iops on fio than on ceph, with same bandwith)
>
>
> for virtio-scsi num_queue = 8:
> -------------------------------
> works with krbd and librbd
>
> -for random read : I jump from 7000 to 12000iops
> -for sequential, qemu aggreate reads in bigger ceph reads. (Same, I see 4xmore iops on fio than on ceph, with same bandwith).
>
>
> So it seem to be useful for some specific workloads.
>
>
>
> 2 ) Now, about cpu usage, it's seem than librbd use really more cpu than krbd,
>
> on host, librbd use 4x more cpu than krbd
> on qemu, librbd use 2x more cpu than krbd
>
>
> So, what could explain so much difference between both ?
Hi Alexandre, do you have access to perf? Especially on newer
kernels/distributions where you can use dwarf symbols, perf can give you
a lot of information about what is using CPU where. vtune is also a
nice option if you ave a license.
>
>
> Regards,
>
> Alexandre
>
>
>
> fio iops seq read summary results
> ----------------------------------
> qemu virtio iothread krbd vs qemu virtio iothread librbd : 27000 iops vs 15000 ipos
> qemu virtio krbd vs qemu virtio librbd : 19000 iops vs 15000 iops
> qemu virtio-scsi krbd vs qemu virtio librbd : 50000 iops vs 48000 iops
> host krbd vs host librbd : 36000 iops vs 25000 iops
>
>
>
> fio iops randread summary results
> ------------------------------
> qemu virtio iothread krbd vs qemu virtio iothread librbd : 15000 iops vs 14000 iops
> qemu virtio krbd vs qemu virtio librbd : 14000 iops vs 15000 iops
> qemu virtio-scsi krbd vs qemu virtio librbd : 7500 iops vs 12000 iops
> host krbd vs host librbd : 38000 iops vs 25000 iops
>
>
> cpu usage ratio summary
> ------------------------
> qemu virtio krbd vs qemu virtio librbd : 2x more cpu usage for librbd
> qemu virtio-scsi krbd vs qemu virtio-scsi librbd : 2x more cpu usage for librbd
> host krbd vs host librbd : 4x more cpu usage for fio-rbd
>
>
>
>
>
>
> RESULTS
> -------
>
> host + fio - krbd
> -------------------
> read
> -----
> fio iops : 142.9MB/Ss , 36000 iops
> ceph : 134 MB/s rd, 34319 op/s
>
> fio : 70,4% kworker : 93,9% cpu = 164% cpu
>
> 100%cpu : 21000iops
>
> randread
> --------
> fio: 151MB/S,38000 iops
> ceph :148 MB/s rd, 37932 op/s
>
> fio : 80%cpu kwoker : 110,3%cpu = 180% cpu
>
> 100%cpu : 21000 iops
>
>
>
>
> host + fio-rbdengin :
> ---------------------
> randread (cpu bound)
> --------------------
> fio iops : 25000 ops
> ceph iops : 99636 kB/s rd, 24909 op/s
>
> fio : 460%cpu
>
> 100%cpu : 5415iops
>
> read (cpu bound)
> -----------------
> fio iops : 25000 ops
> ceph ios : 94212 kB/s rd, 23553 op/s
>
> fio : 480%cpu
>
> 100%cpu : 5323iops
>
>
>
>
>
> qemu + krbd + virtio + iothread
> ---------------------------------
> read
> ----
> fio :107MB/S : 27000 iops >>>SEEM THAT QEMU AGGREGATE READS ops
> ceph : 93942 kB/s rd, 12430 op/s
>
> kvm: 130% cpu - kworker : 41,2% = 171,2% cpu
>
> 100%cpu ratio : 7260iops
>
> randread
> --------
> fio : 60MBS - 15000 iops
> ceph : 54400 kB/s rd, 13600 op/s
>
> kvm: 95,0% cpu - kworker : 42,1 % cpu = 137,1%cpu
>
> 100%cpu ratio : 9919 iops
>
>
>
>
>
> qemu + krbd + virtio
> ----------------------
> read
> -----
> fio : 70mbs/ , 19000iops
> ceph:75705 kB/s rd, 18926 op/s
> kvm : 164% cpu - kworker : 48,5% cpu = 212,5%cpu
>
> 100%cpu ratio : 8906 iops
>
> randread
> --------
> fio : 54mbs/ , 14000iops
> ceph : 54800 kB/s rd, 13700 op/s
> kvm: 103% cpu - kworker 41,2% cpu = 144,2%cpu
>
> 100%cpu ratio : 9513 iops
>
>
>
> qemu + krbd + virtio-scsi (num_queue 8)
> --------------------------------------
> read:
> ----
> fio : 200MB / 50000 iops >>>SEEM THAT QEMU AGGREGATE READS ops
> ceph : 205 MB/s rd, 7648 op/s
>
> kvm: 145% kworker : 46,5% = 191,5%cpu
>
> 100%cpu ratio : 3993 iops
>
> randread:
> ----------
> fio : 30MB/S / 7500 iops
> ceph : 29318 kB/s rd, 7329 op/s
> kvm : 150% kworker : 21,4% cpu = 171,4% cpu
>
> 100%cpu ratio : 4275 iops
>
>
>
>
> qemu + librbd + virtio + iothread
> ----------------------------------
> read
> ----
> fio : 60MBS/s , 15000iops
> ceph: 56199 kB/s rd, 14052 op/s
>
> kvm: 300% cpu
>
> 100%cpu : 4666iops
>
>
> randread
> ---------
> fio : 56MBS/s, 14000iops
> ceph : 55916 kB/s rd, 13979 op/s
>
> kvm: 300% cpu
>
> 100%cpu : 4659 iops
>
>
>
> qemu + librbd + virtio
> -------------------------
> read
> -----
> fio : 60MBS/s, 15000iops
> ceph : 63021 kB/s rd, 15755 op/s
>
> kvm: 300% cpu
>
> 100%cpu : 5233 iops
>
> randread
> --------
> fio : 60MBS/s, 15000iops
> ceph : 55916 kB/s rd, 13979 op/s
>
> kvm : 300%cpu
>
> 100%cpu : 4659 iops
>
>
> qemu + librbd + virtio-scsi (num_queue 8)
> ----------------------------------------
> read
> ----
> fio : 256 MB/S , 48000iops >>>SEEM THAT QEMU AGGREGATE READS ops
> ceph : 244 MB/s rd, 12002 op/s
>
> kvm : 300% cpu
>
> 100%cpu : 4000 iops
>
> randread
> --------
> fio: 12000iops
> ceph iops : 47511 kB/s rd, 11877 op/s
>
> kvm: 300% cpu
>
> 100%cpu: 3959 iops
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: fio read and randread cpu usage results for qemu and host machine
2014-10-23 15:20 ` Alexandre DERUMIER
@ 2014-10-23 21:38 ` Alexandre DERUMIER
0 siblings, 0 replies; 4+ messages in thread
From: Alexandre DERUMIER @ 2014-10-23 21:38 UTC (permalink / raw)
To: Mark Nelson; +Cc: Ceph Devel
[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]
I have redone the perf reports with tree and missing debug symbols.
Here my analysis:
fio + aioengine + krbd : 35000iops randread 4K
-----------------------------------------------
TOP
---
87,1% idle : 104% cpu total usage
2132 root 20 0 97740 4000 3612 S 47,9 0,0 0:14.50fio
2134 root 20 0 0 0 0 S 42,6 0,0 0:12.8 kworker/0:2
48624 root 20 0 0 0 0 S 6,3 0,0 0:54.9 kworker/2:1
48396 root 20 0 0 0 0 S 5,3 0,0 0:13.14kworker/4:0
3 root 20 0 0 0 0 S 4,0 0,0 2:34.53ksoftirqd/0
48387 root 20 0 0 0 0 S 1,3 0,0 0:07.82kworker/6:1
2130 root 20 0 67788 38m 38m S 0,3 0,1 0:00.09
perf
----
+ 24,23% kworker/0:2 [kernel.kallsyms]
+ 21,47% swapper [kernel.kallsyms]
+ 21,10% fio [kernel.kallsyms]
+ 5,37% fio fio
+ 5,36% fio [libceph]
+ 4,93% kworker/2:1 [kernel.kallsyms]
+ 4,89% kworker/0:2 [libceph]
+ 2,08% fio [rbd]
+ 1,80% kworker/0:2 [rbd]
+ 1,69% swapper [tg3]
+ 0,78% kworker/0:2 [tg3]
+ 0,74% kworker/4:0 [kernel.kallsyms]
+ 0,73% ksoftirqd/0 [kernel.kallsyms]
+ 0,66% kworker/2:1 [libceph]
+ 0,60% kworker/3:1 [kernel.kallsyms]
...
I think that kworker/0:2 is main rbd kernel management : 42% cpi
FIO + RBD ENGINE : 35000iops randread 4K
------------------------------------------
TOP
---
28%idle : 576% cpu total usage
1231 root 20 0 922m 38m 35m S 576,1 0,1 1:28.96 fio
3 root 20 0 0 0 0 S 1,3 0,0 2:27.24 ksoftirqd/0
perf
----
+ 25,00% fio [kernel.kallsyms]
+ 24,84% fio libc-2.13.so -----> malloc,free,... from fio rbdengine
+ 16,92% fio librados.so.2.0.0
+ 12,68% swapper [kernel.kallsyms]
+ 9,87% fio librbd.so.1.0.0
+ 4,64% fio libpthread-2.13.so
+ 2,33% fio libstdc++.so.6.0.17
+ 1,88% fio fio
librados+ librd = 26,79% of 576% = 154% cpu.
So, seem that librbd+librados use 3x more cpu than krbd. Is it normal ????
For fio-rbd engine, seem that it's missing a lot of optimisations.
malloc,free take around 25% of 576% : 144%cpu
other seem to be related to fio code too.
Alexandre
[-- Attachment #2: krbd-reportree.txt.gz --]
[-- Type: application/x-gzip, Size: 7340 bytes --]
[-- Attachment #3: rbd-reporttree.txt.gz --]
[-- Type: application/x-gzip, Size: 55439 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-10-23 21:38 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <ff0167fd-ba60-48d2-b735-63569a5a6dd4@mailpro>
2014-10-23 12:30 ` fio read and randread cpu usage results for qemu and host machine Alexandre DERUMIER
2014-10-23 13:07 ` Mark Nelson
2014-10-23 15:20 ` Alexandre DERUMIER
2014-10-23 21:38 ` Alexandre DERUMIER
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.