Re: speedup ceph / scaling / find the bottleneck

[Stefan Priebe <s.priebe@profihost.ag>]

Am 02.07.2012 18:51, schrieb Gregory Farnum:
> On Sun, Jul 1, 2012 at 11:12 PM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> @sage / mark
>> How does the aggregation work? Does it work 4MB blockwise or target node
>> based?
> Aggregation is based on the 4MB blocks, and if you've got caching
> enabled then it's also not going to flush them out to disk very often
> if you're continuously updating the block — I don't remember all the
> conditions, but essentially, you'll run into dirty limits and it will
> asynchronously flush out the data based on a combination of how old it
> is, and how long it's been since some version of it was stable on
> disk.
Is there any way to check if rbd caching works correctly? For me the I/O 
values do not change if i switch writeback on or of and it also doesn't 
matter how large i set the cache size.

...

>> Ceph:
>> 2 VMs:
>>    write: io=2234MB, bw=25405KB/s, iops=6351, runt= 90041msec
>>    read : io=4760MB, bw=54156KB/s, iops=13538, runt= 90007msec
>>    write: io=56372MB, bw=638402KB/s, iops=155, runt= 90421msec
>>    read : io=86572MB, bw=981225KB/s, iops=239, runt= 90346msec
>>
>>    write: io=2222MB, bw=25275KB/s, iops=6318, runt= 90011msec
>>    read : io=4747MB, bw=54000KB/s, iops=13500, runt= 90008msec
>>    write: io=55300MB, bw=626733KB/s, iops=153, runt= 90353msec
>>    read : io=84992MB, bw=965283KB/s, iops=235, runt= 90162msec
>
> I can't quite tell what's going on here, can you describe the test in
> more detail?

I've network booted my VM and then run the following command:
export DISK=/dev/vda; (fio --filename=$DISK --direct=1 --rw=randwrite 
--bs=4k --size=200G --numjobs=50 --runtime=90 --group_reporting 
--name=file1;fio --filename=$DISK --direct=1 --rw=randread --bs=4k 
--size=200G --numjobs=50 --runtime=90 --group_reporting --name=file1;fio 
--filename=$DISK --direct=1 --rw=write --bs=4M --size=200G --numjobs=50 
--runtime=90 --group_reporting --name=file1;fio --filename=$DISK 
--direct=1 --rw=read --bs=4M --size=200G --numjobs=50 --runtime=90 
--group_reporting --name=file1 )|egrep " read| write"

- write random 4k I/O
- read random 4k I/O
- write seq 4M I/O
- read seq 4M I/O

Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html