Re: speedup ceph / scaling / find the bottleneck

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Some more testing / results:

===== lowering CPU cores =====
1.) disabling CPUs via
echo 0 >/sys/devices/system/cpu/cpuX/online
for core 4-7 does not change anything

2.) When only 50% of the CPUs are available each ceph-osd process takes 
only half of the CPU load they use when all are useable.

3.) Even then iops stay at 14k

===== changing replication level =====

1.) Even changing the replication level from 2 to 1 results in still 14k 
iops

===== change random ios to sequential ios =====

1.) when i change i do a write test with 4k blocks instead of randwrite 
i get jumping values from 13k to 30k average is 18k

2.) the interesting thing is here that the ceph-osd processes take just 
1% CPU load

===== direct io to disk =====

1.) when i directly write to the OSD disk from the system itself i 
archieve around 25000 iops

2.) As with ceph the load should spread to several disks i should see 
higher and not lower iops even when the network is involved

Stefan

Am 29.06.2012 12:46, schrieb Stefan Priebe - Profihost AG:
> Hello list,
>
> i've made some further testing and have the problem that ceph doesn't
> scale for me. I added a 4th osd server to my existing 3 node osd
> cluster. I also reformated all to be able to start with a clean system.
>
> While doing random 4k writes from two VMs i see about 8% idle on the osd
> servers (Single Intel Xeon E5 8 cores 3,6Ghz). I believe that this is
> the limiting factor and also the reason why i don't see any improvement
> by adding osd servers.
>
> 3 nodes: 2VMS: 7000 IOp/s 4k writes osds: 7-15% idle
> 4 nodes: 2VMS: 7500 IOp/s 4k writes osds: 7-15% idle
>
> Even the cpu is not the limiting factor i think it would be really
> important to lower the CPU usage while doing 4k writes. The CPU is only
> used by the ceph-osd process. I see nearly no usage by other processes
> (only 5% by kworker and 5% flush).
>
> Could somebody recommand me a way to debug this? So we know where all
> this CPU usage goes?
>
> Stefan