Ceph write performance

Ceph write performance

[George Shuklin]

Good day.

I've start to play with Ceph... And I found some kinda strange 
performance issues. I'm not sure if this is due ceph limitation or my 
bad setup.

Setup:

osd - xfs on ramdisk (only one osd)
mds - raid0 on 10 disks
mon - second raid0 on 10 disks

I've mount ceph share at localhost and run FIO (randwrite, 4k, iodepth=32)

What I've got: 1900 IOPS on writing (4k block, 1Gb span).

Normally fio shows about 200kIOPS writing on ramdisk.

Why it was so slow? I've  done setup exactly like described here: 
http://ceph.com/docs/master/start/quick-start/#start-the-ceph-cluster 
(but one osd).

Thanks.

Re: Ceph write performance

[George Shuklin]

On 20.07.2012 14:41, Dieter Kasper (KD) wrote:

Good day.

Thank you for attention.

ramdisk size ~70Gb (modprobe brd rd_size=70000000)
journal seems be on same device as storage
size of OSD was unchanged (... means I create it by manual and do not 
make any specific changes)

During test I watch IO load closely, IO on MDS/MON was insignificant 
(most of the time zero, sometimes few very mild peaks).

Just in case, configs:

ceph.conf:

[osd]
         osd journal size = 1000
         filestore xattr use omap = true

[mon.a]
         host = srv1
         mon addr = 192.168.0.1:6789

[osd.0]
         host = srv1

[mds.a]
         host = srv1

fio.ini:
[test]
blocksize=4k
filename=/media/test
size=16g
fallocate=posix
rw=randread
direct=1
buffered=0
ioengine=libaio
iodepth=32


Thanks for advising, I'll recheck with new settings.

> George,
>
> please share more details of your config:
> - RAM size of your system
> - location of the journal
> - size of your OSD
>
> Can you try (just for the 1st test) to
> .. put the journal on RAM disk	
> .. put the MDS on RAM disk
> .. put the MON on RAM disk
> .. use btrfs for OSD
>
> As an alternative to isolate the bottleneck you can try to
> - run without a journal
> - use RBD instead Ceph-FS
>    + create a File System on top of the /dev/rbd0
>
> Regards,
> Dieter Kasper
>
>
> On Fri, Jul 20, 2012 at 12:24:15PM +0200, George Shuklin wrote:
>> Good day.
>>
>> I've start to play with Ceph... And I found some kinda strange
>> performance issues. I'm not sure if this is due ceph limitation or my
>> bad setup.
>>
>> Setup:
>>
>> osd - xfs on ramdisk (only one osd)
>> mds - raid0 on 10 disks
>> mon - second raid0 on 10 disks
>>
>> I've mount ceph share at localhost and run FIO (randwrite, 4k, iodepth=32)
>>
>> What I've got: 1900 IOPS on writing (4k block, 1Gb span).
>>
>> Normally fio shows about 200kIOPS writing on ramdisk.
>>
>> Why it was so slow? I've  done setup exactly like described here:
>> http://ceph.com/docs/master/start/quick-start/#start-the-ceph-cluster
>> (but one osd).
>>
>> Thanks.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph write performance

[Mark Nelson]

Hi George,

I think you may find that the limitation is in the the filestore.  It's 
one of the things I've been working on trying to track down as I've seen 
low performance on SSDs with small request sizes as well.  You can use 
the test_filestore_workloadgen to specifically test the filestore code 
with small requests if you'd like.  I'm not sure if it is included with 
the binary distribution but it can be compiled if you download the src. 
  I think it's "make test_filestore_workloadgen" in the src directory.

Mark

On 7/20/12 5:48 AM, George Shuklin wrote:
> On 20.07.2012 14:41, Dieter Kasper (KD) wrote:
>
> Good day.
>
> Thank you for attention.
>
> ramdisk size ~70Gb (modprobe brd rd_size=70000000)
> journal seems be on same device as storage
> size of OSD was unchanged (... means I create it by manual and do not
> make any specific changes)
>
> During test I watch IO load closely, IO on MDS/MON was insignificant
> (most of the time zero, sometimes few very mild peaks).
>
> Just in case, configs:
>
> ceph.conf:
>
> [osd]
>          osd journal size = 1000
>          filestore xattr use omap = true
>
> [mon.a]
>          host = srv1
>          mon addr = 192.168.0.1:6789
>
> [osd.0]
>          host = srv1
>
> [mds.a]
>          host = srv1
>
> fio.ini:
> [test]
> blocksize=4k
> filename=/media/test
> size=16g
> fallocate=posix
> rw=randread
> direct=1
> buffered=0
> ioengine=libaio
> iodepth=32
>
>
> Thanks for advising, I'll recheck with new settings.
>
>> George,
>>
>> please share more details of your config:
>> - RAM size of your system
>> - location of the journal
>> - size of your OSD
>>
>> Can you try (just for the 1st test) to
>> .. put the journal on RAM disk
>> .. put the MDS on RAM disk
>> .. put the MON on RAM disk
>> .. use btrfs for OSD
>>
>> As an alternative to isolate the bottleneck you can try to
>> - run without a journal
>> - use RBD instead Ceph-FS
>>    + create a File System on top of the /dev/rbd0
>>
>> Regards,
>> Dieter Kasper
>>
>>
>> On Fri, Jul 20, 2012 at 12:24:15PM +0200, George Shuklin wrote:
>>> Good day.
>>>
>>> I've start to play with Ceph... And I found some kinda strange
>>> performance issues. I'm not sure if this is due ceph limitation or my
>>> bad setup.
>>>
>>> Setup:
>>>
>>> osd - xfs on ramdisk (only one osd)
>>> mds - raid0 on 10 disks
>>> mon - second raid0 on 10 disks
>>>
>>> I've mount ceph share at localhost and run FIO (randwrite, 4k,
>>> iodepth=32)
>>>
>>> What I've got: 1900 IOPS on writing (4k block, 1Gb span).
>>>
>>> Normally fio shows about 200kIOPS writing on ramdisk.
>>>
>>> Why it was so slow? I've  done setup exactly like described here:
>>> http://ceph.com/docs/master/start/quick-start/#start-the-ceph-cluster
>>> (but one osd).
>>>
>>> Thanks.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph write performance

[Matthew Richardson]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On 20/07/12 11:24, George Shuklin wrote:
> Good day.
> 
> I've start to play with Ceph... And I found some kinda strange
> performance issues. I'm not sure if this is due ceph limitation or my
> bad setup.

I'm seeing a similar problem which looks like a potential bug, which
someone else seems to have already reported

(http://www.spinics.net/lists/ceph-devel/msg07335.html and
http://www.spinics.net/lists/ceph-devel/msg07691.html)

The problem only seems to hit for me when I do random writes - can you
try fio with sequential writes (rw=write) and see if your problem also
disappears?  It might help confirm this as an issue.

Thanks,

Matthew



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ceph write performance

[Gregory Farnum]

On Fri, Jul 20, 2012 at 3:24 AM, George Shuklin <shuklin@selectel.ru> wrote:
> Good day.
>
> I've start to play with Ceph... And I found some kinda strange performance
> issues. I'm not sure if this is due ceph limitation or my bad setup.
>
> Setup:
>
> osd - xfs on ramdisk (only one osd)
> mds - raid0 on 10 disks
> mon - second raid0 on 10 disks

I'm not going to butt in on the performance discussion, but just FYI,
the MDS does not use any local storage — it puts everything on the
OSDs. :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph write performance on RAM-DISK

[Dieter Kasper]

[-- Attachment #1: Type: text/plain, Size: 5220 bytes --]

Hi Mark, George,

I can observe a similar (poor) Performance on my system with fio on /dev/rbd1

#--- seq. write RBD
RX37-0:~ # dd if=/dev/zero of=/dev/rbd1 bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 41.1819 s, 255 MB/s

#--- seq. read RBD
RX37-0:~ # dd of=/dev/zero if=/dev/rbd1 bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 40.9595 s, 256 MB/s

#--- seq. read /dev/ramX
RX37-0:~ # dd of=/dev/zero if=/dev/ram0 bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 4.68389 s, 2.2 GB/s

Does ceph-osd/filestore 'eat' 90% of my resources/bandwidth/latency ?


RX37-0:~ # fio --filename=/dev/rbd1 --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=64 --runtime=30 --group_reporting --name=file1
(...)
  write: io=461592KB, bw=15371KB/s, iops=3842 , runt= 30030msec
  write: io=5120.0MB, bw=893927KB/s, iops=223481 , runt=  5865msec (on /dev/ram0)


RX37-0:~ # fio --filename=/dev/rbd1 --direct=1 --rw=randread --bs=4k --size=5G --numjobs=64 --runtime=30 --group_reporting --name=file1
(...)
  read : io=698356KB, bw=23240KB/s, iops=5809 , runt= 30050msec
  read : io=5120.0MB, bw=1631.1MB/s, iops=417559 , runt=  3139msec (on /dev/ram0)


RX37-0:~ # fio --filename=/dev/rbd1 --direct=1 --rw=randwrite --bs=1m --size=5G --numjobs=4 --runtime=10 --group_reporting --name=file1
(...)
  write: io=6377.0MB, bw=217125KB/s, iops=212 , runt= 30075msec
  write: io=5120.0MB, bw=2114.9MB/s, iops=2114 , runt=  2421msec (on /dev/ram0)


Where is the bottleneck ?
What is filestore doing ?
How can I disable the journal and write only to the btrfs OSDs ? (like as they would be SSDs)
How can I get better performance ?


Regards,
Dieter 

P.S. I will try to get the "test_filestore_workloadgen" 


On Fri, Jul 20, 2012 at 06:49:30AM -0500, Mark Nelson wrote:
> Hi George,
> 
> I think you may find that the limitation is in the the filestore.
> It's one of the things I've been working on trying to track down as
> I've seen low performance on SSDs with small request sizes as well.
> You can use the test_filestore_workloadgen to specifically test the
> filestore code with small requests if you'd like.  I'm not sure if
> it is included with the binary distribution but it can be compiled
> if you download the src.  I think it's "make
> test_filestore_workloadgen" in the src directory.
> 
> Mark
> 
> On 7/20/12 5:48 AM, George Shuklin wrote:
> >On 20.07.2012 14:41, Dieter Kasper (KD) wrote:
> >
> >Good day.
> >
> >Thank you for attention.
> >
> >ramdisk size ~70Gb (modprobe brd rd_size=70000000)
> >journal seems be on same device as storage
> >size of OSD was unchanged (... means I create it by manual and do not
> >make any specific changes)
> >
> >During test I watch IO load closely, IO on MDS/MON was insignificant
> >(most of the time zero, sometimes few very mild peaks).
> >
> >Just in case, configs:
> >
> >ceph.conf:
> >
> >[osd]
> >         osd journal size = 1000
> >         filestore xattr use omap = true
> >
> >[mon.a]
> >         host = srv1
> >         mon addr = 192.168.0.1:6789
> >
> >[osd.0]
> >         host = srv1
> >
> >[mds.a]
> >         host = srv1
> >
> >fio.ini:
> >[test]
> >blocksize=4k
> >filename=/media/test
> >size=16g
> >fallocate=posix
> >rw=randread
> >direct=1
> >buffered=0
> >ioengine=libaio
> >iodepth=32
> >
> >
> >Thanks for advising, I'll recheck with new settings.
> >
> >>George,
> >>
> >>please share more details of your config:
> >>- RAM size of your system
> >>- location of the journal
> >>- size of your OSD
> >>
> >>Can you try (just for the 1st test) to
> >>.. put the journal on RAM disk
> >>.. put the MDS on RAM disk
> >>.. put the MON on RAM disk
> >>.. use btrfs for OSD
> >>
> >>As an alternative to isolate the bottleneck you can try to
> >>- run without a journal
> >>- use RBD instead Ceph-FS
> >>   + create a File System on top of the /dev/rbd0
> >>
> >>Regards,
> >>Dieter Kasper
> >>
> >>
> >>On Fri, Jul 20, 2012 at 12:24:15PM +0200, George Shuklin wrote:
> >>>Good day.
> >>>
> >>>I've start to play with Ceph... And I found some kinda strange
> >>>performance issues. I'm not sure if this is due ceph limitation or my
> >>>bad setup.
> >>>
> >>>Setup:
> >>>
> >>>osd - xfs on ramdisk (only one osd)
> >>>mds - raid0 on 10 disks
> >>>mon - second raid0 on 10 disks
> >>>
> >>>I've mount ceph share at localhost and run FIO (randwrite, 4k,
> >>>iodepth=32)
> >>>
> >>>What I've got: 1900 IOPS on writing (4k block, 1Gb span).
> >>>
> >>>Normally fio shows about 200kIOPS writing on ramdisk.
> >>>
> >>>Why it was so slow? I've  done setup exactly like described here:
> >>>http://ceph.com/docs/master/start/quick-start/#start-the-ceph-cluster
> >>>(but one osd).
> >>>
> >>>Thanks.
> >>>--
> >>>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>>the body of a message to majordomo@vger.kernel.org
> >>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >--
> >To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

[-- Attachment #2: ceph.conf --]
[-- Type: text/plain, Size: 864 bytes --]

[global]
        pid file = /var/run/ceph/$name.pid
        debug ms = 0
        auth supported = cephx
        keyring = /etc/ceph/keyring.client
[mon]
        mon data = /tmp/mon$id
[mon.a]
	host = localhost
	mon addr = 127.0.0.1:6789

[osd]
        journal dio = false
        osd data = /data/$name
	osd journal = /mnt/osd.journal/$name/journal
        osd journal size = 1000

        keyring = /etc/ceph/keyring.$name
        # debug osd = 20
        # debug ms = 1         ; message traffic
        # debug filestore = 20 ; local object storage
        # debug journal = 20   ; local journaling
        # debug monc = 5      ; monitor interaction, startup

[osd.0]
	host = localhost
        btrfs devs = /dev/ram0

[osd.1]
	host = localhost
        btrfs devs = /dev/ram1

[osd.2]
	host = localhost
        btrfs devs = /dev/ram2

[mds.a]
	host = localhost

Re: Ceph write performance on RAM-DISK

[Mark Nelson]

On 07/20/2012 03:36 PM, Dieter Kasper wrote:
> Hi Mark, George,
>
> I can observe a similar (poor) Performance on my system with fio on /dev/rbd1
>
> #--- seq. write RBD
> RX37-0:~ # dd if=/dev/zero of=/dev/rbd1 bs=1024k count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 41.1819 s, 255 MB/s
>
> #--- seq. read RBD
> RX37-0:~ # dd of=/dev/zero if=/dev/rbd1 bs=1024k count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 40.9595 s, 256 MB/s
>
> #--- seq. read /dev/ramX
> RX37-0:~ # dd of=/dev/zero if=/dev/ram0 bs=1024k count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 4.68389 s, 2.2 GB/s
>
> Does ceph-osd/filestore 'eat' 90% of my resources/bandwidth/latency ?
>

Well, there are multiple layers involved here, so it's possible that 
some of the code for RBD is playing a part in this too.  I have 
specifically seen slow performance with smaller requests with the 
filestore though, so that is where I'm focusing my energy right now.

>
> RX37-0:~ # fio --filename=/dev/rbd1 --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=64 --runtime=30 --group_reporting --name=file1
> (...)
>    write: io=461592KB, bw=15371KB/s, iops=3842 , runt= 30030msec
>    write: io=5120.0MB, bw=893927KB/s, iops=223481 , runt=  5865msec (on /dev/ram0)
>
>
> RX37-0:~ # fio --filename=/dev/rbd1 --direct=1 --rw=randread --bs=4k --size=5G --numjobs=64 --runtime=30 --group_reporting --name=file1
> (...)
>    read : io=698356KB, bw=23240KB/s, iops=5809 , runt= 30050msec
>    read : io=5120.0MB, bw=1631.1MB/s, iops=417559 , runt=  3139msec (on /dev/ram0)
>
>
> RX37-0:~ # fio --filename=/dev/rbd1 --direct=1 --rw=randwrite --bs=1m --size=5G --numjobs=4 --runtime=10 --group_reporting --name=file1
> (...)
>    write: io=6377.0MB, bw=217125KB/s, iops=212 , runt= 30075msec
>    write: io=5120.0MB, bw=2114.9MB/s, iops=2114 , runt=  2421msec (on /dev/ram0)
>
>
> Where is the bottleneck ?
> What is filestore doing ?
> How can I disable the journal and write only to the btrfs OSDs ? (like as they would be SSDs)
> How can I get better performance ?

Not yet sure where the bottleneck is, but we are actively looking into 
it.  Sadly the process has been complicated by potential bottleneck in 
our test hardware that could be masking real issues in the code.

>
>
> Regards,
> Dieter
>
> P.S. I will try to get the "test_filestore_workloadgen"
>
>
> On Fri, Jul 20, 2012 at 06:49:30AM -0500, Mark Nelson wrote:
>> Hi George,
>>
>> I think you may find that the limitation is in the the filestore.
>> It's one of the things I've been working on trying to track down as
>> I've seen low performance on SSDs with small request sizes as well.
>> You can use the test_filestore_workloadgen to specifically test the
>> filestore code with small requests if you'd like.  I'm not sure if
>> it is included with the binary distribution but it can be compiled
>> if you download the src.  I think it's "make
>> test_filestore_workloadgen" in the src directory.
>>
>> Mark
>>
>> On 7/20/12 5:48 AM, George Shuklin wrote:
>>> On 20.07.2012 14:41, Dieter Kasper (KD) wrote:
>>>
>>> Good day.
>>>
>>> Thank you for attention.
>>>
>>> ramdisk size ~70Gb (modprobe brd rd_size=70000000)
>>> journal seems be on same device as storage
>>> size of OSD was unchanged (... means I create it by manual and do not
>>> make any specific changes)
>>>
>>> During test I watch IO load closely, IO on MDS/MON was insignificant
>>> (most of the time zero, sometimes few very mild peaks).
>>>
>>> Just in case, configs:
>>>
>>> ceph.conf:
>>>
>>> [osd]
>>>          osd journal size = 1000
>>>          filestore xattr use omap = true
>>>
>>> [mon.a]
>>>          host = srv1
>>>          mon addr = 192.168.0.1:6789
>>>
>>> [osd.0]
>>>          host = srv1
>>>
>>> [mds.a]
>>>          host = srv1
>>>
>>> fio.ini:
>>> [test]
>>> blocksize=4k
>>> filename=/media/test
>>> size=16g
>>> fallocate=posix
>>> rw=randread
>>> direct=1
>>> buffered=0
>>> ioengine=libaio
>>> iodepth=32
>>>
>>>
>>> Thanks for advising, I'll recheck with new settings.
>>>
>>>> George,
>>>>
>>>> please share more details of your config:
>>>> - RAM size of your system
>>>> - location of the journal
>>>> - size of your OSD
>>>>
>>>> Can you try (just for the 1st test) to
>>>> .. put the journal on RAM disk
>>>> .. put the MDS on RAM disk
>>>> .. put the MON on RAM disk
>>>> .. use btrfs for OSD
>>>>
>>>> As an alternative to isolate the bottleneck you can try to
>>>> - run without a journal
>>>> - use RBD instead Ceph-FS
>>>>    + create a File System on top of the /dev/rbd0
>>>>
>>>> Regards,
>>>> Dieter Kasper
>>>>
>>>>
>>>> On Fri, Jul 20, 2012 at 12:24:15PM +0200, George Shuklin wrote:
>>>>> Good day.
>>>>>
>>>>> I've start to play with Ceph... And I found some kinda strange
>>>>> performance issues. I'm not sure if this is due ceph limitation or my
>>>>> bad setup.
>>>>>
>>>>> Setup:
>>>>>
>>>>> osd - xfs on ramdisk (only one osd)
>>>>> mds - raid0 on 10 disks
>>>>> mon - second raid0 on 10 disks
>>>>>
>>>>> I've mount ceph share at localhost and run FIO (randwrite, 4k,
>>>>> iodepth=32)
>>>>>
>>>>> What I've got: 1900 IOPS on writing (4k block, 1Gb span).
>>>>>
>>>>> Normally fio shows about 200kIOPS writing on ramdisk.
>>>>>
>>>>> Why it was so slow? I've  done setup exactly like described here:
>>>>> http://ceph.com/docs/master/start/quick-start/#start-the-ceph-cluster
>>>>> (but one osd).
>>>>>
>>>>> Thanks.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>


-- 
Mark Nelson
Performance Engineer
Inktank

RBD performance - tuning hints

[Dieter Kasper]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

Hi,

on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
I can observe a pretty nice rados bench performance 
(see bench-rados.txt for details):

Bandwidth (MB/sec):     961.710 
Max bandwidth (MB/sec): 1040
Min bandwidth (MB/sec): 772


Also the bandwidth performance generated with
  fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}

.... is acceptable, e.g.
fio_write_4m_16		795 MB/s
fio_randwrite_8m_128	717 MB/s
fio_randwrite_8m_16	714 MB/s
fio_randwrite_2m_32	692 MB/s


But, the write IOPS seems to be limited around 19k ...
RBD                     4M      64k (= optimal_io_size)
fio_randread_512_128    53286   55925
fio_randread_4k_128     51110   44382
fio_randread_8k_128     30854   29938
fio_randwrite_512_128   18888    2386
fio_randwrite_512_64    18844    2582
fio_randwrite_8k_64     17350    2445
(...)
fio_read_4k_128         10073   53151
fio_read_4k_64           9500   39757
fio_read_4k_32           9220   23650
(...)
fio_read_4k_16           9122   14322
fio_write_4k_128         2190   14306
fio_read_8k_32            706   13894
fio_write_4k_64          2197   12297
fio_write_8k_64          3563   11705
fio_write_8k_128         3444   11219


Any hints for tuning the IOPS (read and/or write) would be appreciated.

How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)


Kind Regards,
-Dieter

[-- Attachment #2: bench-rados.txt --]
[-- Type: text/plain, Size: 1746 bytes --]

rados bench -p pbench 60 write
 Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds.
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
     0       0         0         0         0         0         -         0
     1      16       228       212   847.857       848  0.042984 0.0684383
     2      16       451       435    869.88       892  0.084162 0.0700566
     3      16       695       679   905.223       976  0.057677 0.0695337
     4      16       942       926   925.894       988  0.038117 0.0685357
     5      16      1162      1146     916.7       880  0.042098 0.0693864
     6      16      1400      1384   922.569       952  0.063983 0.0689167
     7      16      1644      1628   930.189       976  0.065745 0.0684646
     8      16      1895      1879   939.404      1004  0.051277 0.0677953
     9      16      2145      2129   946.127      1000  0.055165  0.067354
(...)
    57      16     13704     13688    960.47       996  0.082716 0.0665862
    58      16     13954     13938    961.15      1000  0.041879 0.0665307
    59      16     14194     14178   961.129       960  0.046657 0.0664642
2012-08-28 17:32:18.620060min lat: 0.030234 max lat: 3.17834 avg lat: 0.0664676
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    60      16     14446     14430   961.909      1008  0.051635 0.0664676
 Total time run:         60.084612
Total writes made:      14446
Write size:             4194304
Bandwidth (MB/sec):     961.710 

Stddev Bandwidth:       54.0809
Max bandwidth (MB/sec): 1040
Min bandwidth (MB/sec): 772
Average Latency:        0.0665337
Stddev Latency:         0.0800225
Max latency:            3.17834
Min latency:            0.030234

[-- Attachment #3: bench-config.txt --]
[-- Type: text/plain, Size: 26557 bytes --]

--- RX37-3c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-3 3.0.41-5.1-default #1 SMP Wed Aug 22 00:54:03 UTC 2012 (9c63123) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856332 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdm 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdn 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdo 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdp 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdq 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdr 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sds 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdt 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     37 C
  Blocks sent to initiator = 198232151949312
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     39 C
  Blocks sent to initiator = 188127268306944
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     42 C
  Blocks sent to initiator = 241646771896320
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     33 C
  Blocks sent to initiator = 202151376715776
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     34 C
  Blocks sent to initiator = 186279543177216
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 200414079221760
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     40 C
  Blocks sent to initiator = 301595287879680
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     30 C
  Blocks sent to initiator = 190686448058368
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdm on /data/osd.30 type btrfs (rw,noatime)
/dev/sdn on /data/osd.31 type btrfs (rw,noatime)
/dev/sdo on /data/osd.32 type btrfs (rw,noatime)
/dev/sdp on /data/osd.33 type btrfs (rw,noatime)
/dev/sdq on /data/osd.34 type btrfs (rw,noatime)
/dev/sdr on /data/osd.35 type btrfs (rw,noatime)
/dev/sds on /data/osd.36 type btrfs (rw,noatime)
/dev/sdt on /data/osd.37 type btrfs (rw,noatime)
--- RX37-4c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-4 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856432 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdd 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sde 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdf 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdg 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdh 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdi 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdj 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdk 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     33 C
  Blocks sent to initiator = 326270260871168
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     29 C
  Blocks sent to initiator = 230247207272448
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     34 C
  Blocks sent to initiator = 168513041858560
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     37 C
  Blocks sent to initiator = 171904673513472
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     30 C
  Blocks sent to initiator = 175995797635072
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 206814587125760
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     26 C
  Blocks sent to initiator = 239652363567104
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     32 C
  Blocks sent to initiator = 221954917269504
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdd on /data/osd.40 type btrfs (rw,noatime)
/dev/sde on /data/osd.41 type btrfs (rw,noatime)
/dev/sdf on /data/osd.42 type btrfs (rw,noatime)
/dev/sdg on /data/osd.43 type btrfs (rw,noatime)
/dev/sdh on /data/osd.44 type btrfs (rw,noatime)
/dev/sdi on /data/osd.45 type btrfs (rw,noatime)
/dev/sdj on /data/osd.46 type btrfs (rw,noatime)
/dev/sdk on /data/osd.47 type btrfs (rw,noatime)
--- RX37-5c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-5 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       74226012 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdo 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdp 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdq 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdr 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sds 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdt 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdu 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdv 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 195550280417280
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     37 C
  Blocks sent to initiator = 177656960122880
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     41 C
  Blocks sent to initiator = 238550402465792
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     31 C
  Blocks sent to initiator = 226579741409280
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     33 C
  Blocks sent to initiator = 186652383248384
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     34 C
  Blocks sent to initiator = 219684389519360
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     39 C
  Blocks sent to initiator = 223471107833856
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     29 C
  Blocks sent to initiator = 190300723085312
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdo on /data/osd.50 type btrfs (rw,noatime)
/dev/sdp on /data/osd.51 type btrfs (rw,noatime)
/dev/sdq on /data/osd.52 type btrfs (rw,noatime)
/dev/sdr on /data/osd.53 type btrfs (rw,noatime)
/dev/sds on /data/osd.54 type btrfs (rw,noatime)
/dev/sdt on /data/osd.55 type btrfs (rw,noatime)
/dev/sdu on /data/osd.56 type btrfs (rw,noatime)
/dev/sdv on /data/osd.57 type btrfs (rw,noatime)
--- RX37-6c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-6 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856344 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdn 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdo 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdp 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdq 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdr 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sds 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdt 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdu 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     41 C
  Blocks sent to initiator = 195597608943616
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 197325225984000
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     42 C
  Blocks sent to initiator = 182463498289152
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     45 C
  Blocks sent to initiator = 250870398713856
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     37 C
  Blocks sent to initiator = 209343584665600
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     33 C
  Blocks sent to initiator = 226728102330368
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     43 C
  Blocks sent to initiator = 213839006138368
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     38 C
  Blocks sent to initiator = 179503745728512
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdn on /data/osd.60 type btrfs (rw,noatime)
/dev/sdo on /data/osd.61 type btrfs (rw,noatime)
/dev/sdp on /data/osd.62 type btrfs (rw,noatime)
/dev/sdq on /data/osd.63 type btrfs (rw,noatime)
/dev/sdr on /data/osd.64 type btrfs (rw,noatime)
/dev/sds on /data/osd.65 type btrfs (rw,noatime)
/dev/sdt on /data/osd.66 type btrfs (rw,noatime)
/dev/sdu on /data/osd.67 type btrfs (rw,noatime)
--- RX37-7c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-7 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856344 kB
optimal_io_size: 4194304
65536
scheduler:       [noop] deadline cfq 
noop deadline [cfq] 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
--- RX37-8c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-8 3.0.36-16-default #1 SMP Wed Jul 18 00:18:54 UTC 2012 (544e41f) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       65952088 kB
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
--------------------------------------------------------------------------------

dumped osdmap epoch 15
epoch 15
fsid 7ab4662b-0575-4875-b59d-3bef85bb918d
created 2012-08-26 15:10:43.529294
modifed 2012-08-26 15:11:09.537529
flags 

pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 4352 pgp_num 4352 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 4352 pgp_num 4352 last_change 1 owner 0
pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 4352 pgp_num 4352 last_change 1 owner 0

max_osd 68
osd.30 up   in  weight 1 up_from 2 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6800/7884 192.168.114.52:6800/7884 192.168.114.52:6801/7884 exists,up f1912b6b-2abf-4eef-83e0-8657d78e48f8
osd.31 up   in  weight 1 up_from 4 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6801/8057 192.168.114.52:6802/8057 192.168.114.52:6803/8057 exists,up 2a254612-5242-4ae8-8ba7-3fe2eaa3eec5
osd.32 up   in  weight 1 up_from 3 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6802/8225 192.168.114.52:6804/8225 192.168.114.52:6805/8225 exists,up d41508ee-131c-47b8-9218-8f81bc7f7716
osd.33 up   in  weight 1 up_from 3 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6803/8415 192.168.114.52:6806/8415 192.168.114.52:6807/8415 exists,up 2e5a96be-ca3a-4c7d-8895-b61c07d858ac
osd.34 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6804/8588 192.168.114.52:6808/8588 192.168.114.52:6809/8588 exists,up 214d8253-ad9b-4268-ba67-365ae9bc612a
osd.35 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6805/8777 192.168.114.52:6810/8777 192.168.114.52:6811/8777 exists,up 9d328117-581a-4fdb-bee8-e373e74ee013
osd.36 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6806/8966 192.168.114.52:6812/8966 192.168.114.52:6813/8966 exists,up 0d046c45-ddd3-4c24-814c-36ace0632167
osd.37 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.52:6807/9155 192.168.114.52:6814/9155 192.168.114.52:6815/9155 exists,up 2265a65a-624c-4729-bf64-47850270b4a9
osd.40 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6800/14455 192.168.114.53:6800/14455 192.168.114.53:6801/14455 exists,up e782364f-c5ee-4181-98ba-8e8009a789db
osd.41 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6801/14639 192.168.114.53:6802/14639 192.168.114.53:6803/14639 exists,up 3154b1e5-e49a-417a-9b80-d64995afb2c8
osd.42 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6802/14816 192.168.114.53:6804/14816 192.168.114.53:6805/14816 exists,up a7cab833-70b2-4067-83a3-a8a7b7ccb1c2
osd.43 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6803/15013 192.168.114.53:6806/15013 192.168.114.53:6807/15013 exists,up 5afeea03-5a5d-4643-bbde-aaadda1bde01
osd.44 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6804/15190 192.168.114.53:6808/15190 192.168.114.53:6809/15190 exists,up 5b1a90a2-596d-40d4-b33d-cf74142f7e96
osd.45 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6805/15420 192.168.114.53:6810/15420 192.168.114.53:6811/15420 exists,up e4d85019-c8d4-4dc8-bec3-ceaddab60b99
osd.46 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6806/15623 192.168.114.53:6812/15623 192.168.114.53:6813/15623 exists,up 0a1b6a02-1b70-457f-9602-8f02e00d7ae1
osd.47 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.53:6807/15826 192.168.114.53:6814/15826 192.168.114.53:6815/15826 exists,up 7be9d381-8c38-440c-ae22-fc29a9349351
osd.50 up   in  weight 1 up_from 5 up_thru 12 down_at 0 last_clean_interval [0,0) 192.168.113.54:6800/1915 192.168.114.54:6800/1915 192.168.114.54:6801/1915 exists,up 7653343d-5602-4a6e-ac69-a278dab28c8c
osd.51 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.54:6801/2155 192.168.114.54:6802/2155 192.168.114.54:6803/2155 exists,up a58bfbfb-8f21-4939-8ca1-b8209be68a30
osd.52 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.54:6802/2322 192.168.114.54:6804/2322 192.168.114.54:6805/2322 exists,up 81daeb73-23f4-4f68-b56b-7d5a1b95e7e0
osd.53 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.54:6803/2515 192.168.114.54:6806/2515 192.168.114.54:6807/2515 exists,up b3978c52-f689-45e8-9ee2-681e3bdeeeb2
osd.54 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.54:6804/2702 192.168.114.54:6808/2702 192.168.114.54:6809/2702 exists,up 205b59d3-176a-4048-84c5-81dd181a8e71
osd.55 up   in  weight 1 up_from 5 up_thru 11 down_at 0 last_clean_interval [0,0) 192.168.113.54:6805/2889 192.168.114.54:6810/2889 192.168.114.54:6811/2889 exists,up cd4d82de-0da8-48b0-a54f-d1372b611958
osd.56 up   in  weight 1 up_from 6 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.54:6806/3082 192.168.114.54:6812/3082 192.168.114.54:6813/3082 exists,up b82b38a6-64ad-487a-899b-6c62ebe6bb13
osd.57 up   in  weight 1 up_from 6 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.54:6807/3269 192.168.114.54:6814/3269 192.168.114.54:6815/3269 exists,up c155cf46-d287-4439-a39e-ff80c22e0caa
osd.60 up   in  weight 1 up_from 7 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6800/30607 192.168.114.55:6800/30607 192.168.114.55:6801/30607 exists,up ab8370bf-c722-4eab-9842-498b6dfef765
osd.61 up   in  weight 1 up_from 7 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6801/30801 192.168.114.55:6802/30801 192.168.114.55:6803/30801 exists,up a189a254-efcd-4129-867e-384cd0765d19
osd.62 up   in  weight 1 up_from 8 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6802/30946 192.168.114.55:6804/30946 192.168.114.55:6805/30946 exists,up 2ddc9000-a5be-4c7f-9362-2c525b93db7f
osd.63 up   in  weight 1 up_from 9 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6803/31139 192.168.114.55:6806/31139 192.168.114.55:6807/31139 exists,up 5c4661fb-4c6c-411d-bf46-b4ead15a019a
osd.64 up   in  weight 1 up_from 9 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6804/31332 192.168.114.55:6808/31332 192.168.114.55:6809/31332 exists,up b67f9e9b-d0f6-41b9-ac7f-0c355950316f
osd.65 up   in  weight 1 up_from 10 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6805/31525 192.168.114.55:6810/31525 192.168.114.55:6811/31525 exists,up 9e179b5f-b0ca-4799-8b02-13fc3a78eda5
osd.66 up   in  weight 1 up_from 10 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6806/31814 192.168.114.55:6812/31814 192.168.114.55:6813/31814 exists,up e300060b-ac96-4ed0-9670-ffe3d7547a18
osd.67 up   in  weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) 192.168.113.55:6807/32063 192.168.114.55:6814/32063 192.168.114.55:6815/32063 exists,up f87f78b3-61ba-403a-b012-ddd055ced47f



ceph.conf
---content---
# global
[global]
	# enable secure authentication
	auth supported = none

        # allow ourselves to open a lot of files
        #max open files = 1100000
        max open files = 131072

        # set log file
        log file = /ceph/log/$name.log
        # log_to_syslog = true        # uncomment this line to log to syslog

        # set up pid files
        pid file = /var/run/ceph/$name.pid

        # If you want to run a IPv6 cluster, set this to true. Dual-stack isn't possible
        #ms bind ipv6 = true
	public network = 192.168.113.0/24
	cluster network = 192.168.114.0/24

# monitors
#  You need at least one.  You need at least three if you want to
#  tolerate any node failures.  Always create an odd number.
[mon]
        mon data = /ceph/$name

        # If you are using for example the RADOS Gateway and want to have your newly created
        # pools a higher replication level, you can set a default
        #osd pool default size = 3

        # You can also specify a CRUSH rule for new pools
        # Wiki: http://ceph.newdream.net/wiki/Custom_data_placement_with_CRUSH
        #osd pool default crush rule = 0

        # Timing is critical for monitors, but if you want to allow the clocks to drift a
        # bit more, you can specify the max drift.
        #mon clock drift allowed = 1

        # Tell the monitor to backoff from this warning for 30 seconds
        #mon clock drift warn backoff = 30

	# logging, for debugging monitor crashes, in order of
	# their likelihood of being helpful :)
	#debug ms = 1
	#debug mon = 20
	#debug paxos = 20
	#debug auth = 20
	debug optracker = 0

[mon.0]
	host = RX37-3c
	mon addr = 192.168.113.52:6789
[mon.1]
	host = RX37-7c
	mon addr = 192.168.113.56:6789
[mon.2]
	host = RX37-8c
	mon addr = 192.168.113.57:6789
	
# mds
#  You need at least one.  Define two to get a standby.
[mds]
#        mds data = /ceph/$name
	# where the mds keeps it's secret encryption keys
	#keyring = /data/keyring.$name

	# mds logging to debug issues.
	#debug ms = 1
	#debug mds = 20
	debug optracker = 0

[mds.0]
        host = RX37-8c

# osd
#  You need at least one.  Two if you want data to be replicated.
#  Define as many as you like.
[osd]
	# This is where the btrfs volume will be mounted.
	osd data = /data/$name

#        journal dio = true
#        osd op threads = 24
#        osd disk threads = 24
#        filestore op threads = 6
#        filestore queue max ops = 24

	# Ideally, make this a separate disk or partition.  A few
 	# hundred MB should be enough; more if you have fast or many
 	# disks.  You can use a file under the osd data dir if need be
 	# (e.g. /data/$name/journal), but it will be slower than a
 	# separate disk or partition.

        # This is an example of a file-based journal.
	# osd journal = /ceph/$name/journal
	# osd journal size = 2048 
	# journal size, in megabytes

        # If you want to run the journal on a tmpfs, disable DirectIO
        #journal dio = false

        # You can change the number of recovery operations to speed up recovery
        # or slow it down if your machines can't handle it
        # osd recovery max active = 3

	# osd logging to debug osd issues, in order of likelihood of being
	# helpful
	#debug ms = 1
	#debug osd = 20
	#debug filestore = 20
	#debug journal = 20
	debug optracker = 0
	fstype = btrfs

[osd.30]
	host = RX37-3c
	devs = /dev/sdm
	osd journal = /dev/ram0
[osd.31]
	host = RX37-3c
	devs = /dev/sdn
	osd journal = /dev/ram1
[osd.32]
	host = RX37-3c
	devs = /dev/sdo
	osd journal = /dev/ram2
[osd.33]
	host = RX37-3c
	devs = /dev/sdp
	osd journal = /dev/ram3
[osd.34]
	host = RX37-3c
	devs = /dev/sdq
	osd journal = /dev/ram4
[osd.35]
	host = RX37-3c
	devs = /dev/sdr
	osd journal = /dev/ram5
[osd.36]
	host = RX37-3c
	devs = /dev/sds
	osd journal = /dev/ram6
[osd.37]
	host = RX37-3c
	devs = /dev/sdt
	osd journal = /dev/ram7
[osd.40]
	host = RX37-4c
	devs = /dev/sdd
	osd journal = /dev/ram0
[osd.41]
	host = RX37-4c
	devs = /dev/sde
	osd journal = /dev/ram1
[osd.42]
	host = RX37-4c
	devs = /dev/sdf
	osd journal = /dev/ram2
[osd.43]
	host = RX37-4c
	devs = /dev/sdg
	osd journal = /dev/ram3
[osd.44]
	host = RX37-4c
	devs = /dev/sdh
	osd journal = /dev/ram4
[osd.45]
	host = RX37-4c
	devs = /dev/sdi
	osd journal = /dev/ram5
[osd.46]
	host = RX37-4c
	devs = /dev/sdj
	osd journal = /dev/ram6
[osd.47]
	host = RX37-4c
	devs = /dev/sdk
	osd journal = /dev/ram7
[osd.50]
	host = RX37-5c
	devs = /dev/sdo
	osd journal = /dev/ram0
[osd.51]
	host = RX37-5c
	devs = /dev/sdp
	osd journal = /dev/ram1
[osd.52]
	host = RX37-5c
	devs = /dev/sdq
	osd journal = /dev/ram2
[osd.53]
	host = RX37-5c
	devs = /dev/sdr
	osd journal = /dev/ram3
[osd.54]
	host = RX37-5c
	devs = /dev/sds
	osd journal = /dev/ram4
[osd.55]
	host = RX37-5c
	devs = /dev/sdt
	osd journal = /dev/ram5
[osd.56]
	host = RX37-5c
	devs = /dev/sdu
	osd journal = /dev/ram6
[osd.57]
	host = RX37-5c
	devs = /dev/sdv
	osd journal = /dev/ram7
[osd.60]
	host = RX37-6c
	devs = /dev/sdn
	osd journal = /dev/ram0
[osd.61]
	host = RX37-6c
	devs = /dev/sdo
	osd journal = /dev/ram1
[osd.62]
	host = RX37-6c
	devs = /dev/sdp
	osd journal = /dev/ram2
[osd.63]
	host = RX37-6c
	devs = /dev/sdq
	osd journal = /dev/ram3
[osd.64]
	host = RX37-6c
	devs = /dev/sdr
	osd journal = /dev/ram4
[osd.65]
	host = RX37-6c
	devs = /dev/sds
	osd journal = /dev/ram5
[osd.66]
	host = RX37-6c
	devs = /dev/sdt
	osd journal = /dev/ram6
[osd.67]
	host = RX37-6c
	devs = /dev/sdu
	osd journal = /dev/ram7
	devs = /dev/sdc

[client.01]
	client hostname = RX37-7c

Re: RBD performance - tuning hints

[Smart Weblications GmbH - Florian Wiessner]

Am 28.08.2012 19:48, schrieb Dieter Kasper:
> Hi,
> 
> on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
> I can observe a pretty nice rados bench performance 
> (see bench-rados.txt for details):

i'd like to know which 10GE Switch you have used? Do you use 10GE-Base-T?




-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Dieter Kasper]

On Tue, Aug 28, 2012 at 08:53:46PM +0200, Smart Weblications GmbH - Florian Wiessner wrote:
> Am 28.08.2012 19:48, schrieb Dieter Kasper:
> > Hi,
> > 
> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
> > I can observe a pretty nice rados bench performance 
> > (see bench-rados.txt for details):
> 
> i'd like to know which 10GE Switch you have used? Do you use 10GE-Base-T?
http://www.brocade.com/products/all/switches/product-details/turboiron-24x-switch/index.page

Mit freundlichen Grüßen
Dieter Kasper

> 
> 
> 
> 
> -- 
> 
> Mit freundlichen Grüßen,
> 
> Florian Wiessner
> 
> Smart Weblications GmbH
> Martinsberger Str. 1
> D-95119 Naila
> 
> fon.: +49 9282 9638 200
> fax.: +49 9282 9638 205
> 24/7: +49 900 144 000 00 - 0,99 EUR/Min*
> http://www.smart-weblications.de
> 
> --
> Sitz der Gesellschaft: Naila
> Geschäftsführer: Florian Wiessner
> HRB-Nr.: HRB 3840 Amtsgericht Hof
> *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Alexandre DERUMIER]

Nice results !
(can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)

>>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
I think you can try to tune these values

filestore max sync interval = 30
filestore min sync interval = 29
filestore flusher = false
filestore queue max ops = 10000



----- Mail original ----- 

De: "Dieter Kasper" <d.kasper@kabelmail.de> 
À: ceph-devel@vger.kernel.org 
Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
Envoyé: Mardi 28 Août 2012 19:48:42 
Objet: RBD performance - tuning hints 

Hi, 

on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
I can observe a pretty nice rados bench performance 
(see bench-rados.txt for details): 

Bandwidth (MB/sec): 961.710 
Max bandwidth (MB/sec): 1040 
Min bandwidth (MB/sec): 772 


Also the bandwidth performance generated with 
fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 

.... is acceptable, e.g. 
fio_write_4m_16 795 MB/s 
fio_randwrite_8m_128 717 MB/s 
fio_randwrite_8m_16 714 MB/s 
fio_randwrite_2m_32 692 MB/s 


But, the write IOPS seems to be limited around 19k ... 
RBD 4M 64k (= optimal_io_size) 
fio_randread_512_128 53286 55925 
fio_randread_4k_128 51110 44382 
fio_randread_8k_128 30854 29938 
fio_randwrite_512_128 18888 2386 
fio_randwrite_512_64 18844 2582 
fio_randwrite_8k_64 17350 2445 
(...) 
fio_read_4k_128 10073 53151 
fio_read_4k_64 9500 39757 
fio_read_4k_32 9220 23650 
(...) 
fio_read_4k_16 9122 14322 
fio_write_4k_128 2190 14306 
fio_read_8k_32 706 13894 
fio_write_4k_64 2197 12297 
fio_write_8k_64 3563 11705 
fio_write_8k_128 3444 11219 


Any hints for tuning the IOPS (read and/or write) would be appreciated. 

How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 


Kind Regards, 
-Dieter 



-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Josh Durgin]

On 08/29/2012 01:50 AM, Alexandre DERUMIER wrote:
> Nice results !
> (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
> I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
>
>>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> I think you can try to tune these values
>
> filestore max sync interval = 30
> filestore min sync interval = 29
> filestore flusher = false
> filestore queue max ops = 10000

Increasing filestore_op_threads might help as well.

> ----- Mail original -----
>
> De: "Dieter Kasper" <d.kasper@kabelmail.de>
> À: ceph-devel@vger.kernel.org
> Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
> Envoyé: Mardi 28 Août 2012 19:48:42
> Objet: RBD performance - tuning hints
>
> Hi,
>
> on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
> I can observe a pretty nice rados bench performance
> (see bench-rados.txt for details):
>
> Bandwidth (MB/sec): 961.710
> Max bandwidth (MB/sec): 1040
> Min bandwidth (MB/sec): 772
>
>
> Also the bandwidth performance generated with
> fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
>
> .... is acceptable, e.g.
> fio_write_4m_16 795 MB/s
> fio_randwrite_8m_128 717 MB/s
> fio_randwrite_8m_16 714 MB/s
> fio_randwrite_2m_32 692 MB/s
>
>
> But, the write IOPS seems to be limited around 19k ...
> RBD 4M 64k (= optimal_io_size)
> fio_randread_512_128 53286 55925
> fio_randread_4k_128 51110 44382
> fio_randread_8k_128 30854 29938
> fio_randwrite_512_128 18888 2386
> fio_randwrite_512_64 18844 2582
> fio_randwrite_8k_64 17350 2445
> (...)
> fio_read_4k_128 10073 53151
> fio_read_4k_64 9500 39757
> fio_read_4k_32 9220 23650
> (...)
> fio_read_4k_16 9122 14322
> fio_write_4k_128 2190 14306
> fio_read_8k_32 706 13894
> fio_write_4k_64 2197 12297
> fio_write_8k_64 3563 11705
> fio_write_8k_128 3444 11219
>
>
> Any hints for tuning the IOPS (read and/or write) would be appreciated.
>
> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>
>
> Kind Regards,
> -Dieter
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints / parameter doc

[Dieter Kasper]

Hi Josh,

thanks for the hint.
Can you please spend a view words about the meaing of these parameters ?
- filestore min/max sync interval = 	int/float ?	seconds ? of what ?
- filestore flusher = false
- filestore queue max ops = 10000	
	what is 'one op' ?	queue in front of what ?
- filestore op threads =	
	what are useful values here ?

- journal dio = true/false
- osd op threads = 
- osd disk threads = 


Kind Regards,
-Dieter


On Wed, Aug 29, 2012 at 07:37:36PM +0200, Josh Durgin wrote:
> On 08/29/2012 01:50 AM, Alexandre DERUMIER wrote:
> > Nice results !
> > (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
> > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
> >
> >>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> > I think you can try to tune these values
> >
> > filestore max sync interval = 30
> > filestore min sync interval = 29
> > filestore flusher = false
> > filestore queue max ops = 10000
> 
> Increasing filestore_op_threads might help as well.
> 
> > ----- Mail original -----
> >
> > De: "Dieter Kasper" <d.kasper@kabelmail.de>
> > À: ceph-devel@vger.kernel.org
> > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
> > Envoyé: Mardi 28 Août 2012 19:48:42
> > Objet: RBD performance - tuning hints
> >
> > Hi,
> >
> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
> > I can observe a pretty nice rados bench performance
> > (see bench-rados.txt for details):
> >
> > Bandwidth (MB/sec): 961.710
> > Max bandwidth (MB/sec): 1040
> > Min bandwidth (MB/sec): 772
> >
> >
> > Also the bandwidth performance generated with
> > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
> >
> > .... is acceptable, e.g.
> > fio_write_4m_16 795 MB/s
> > fio_randwrite_8m_128 717 MB/s
> > fio_randwrite_8m_16 714 MB/s
> > fio_randwrite_2m_32 692 MB/s
> >
> >
> > But, the write IOPS seems to be limited around 19k ...
> > RBD 4M 64k (= optimal_io_size)
> > fio_randread_512_128 53286 55925
> > fio_randread_4k_128 51110 44382
> > fio_randread_8k_128 30854 29938
> > fio_randwrite_512_128 18888 2386
> > fio_randwrite_512_64 18844 2582
> > fio_randwrite_8k_64 17350 2445
> > (...)
> > fio_read_4k_128 10073 53151
> > fio_read_4k_64 9500 39757
> > fio_read_4k_32 9220 23650
> > (...)
> > fio_read_4k_16 9122 14322
> > fio_write_4k_128 2190 14306
> > fio_read_8k_32 706 13894
> > fio_write_4k_64 2197 12297
> > fio_write_8k_64 3563 11705
> > fio_write_8k_128 3444 11219
> >
> >
> > Any hints for tuning the IOPS (read and/or write) would be appreciated.
> >
> > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> >
> >
> > Kind Regards,
> > -Dieter
> >
> >
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints / parameter doc

[Samuel Just]

filestore [min|max] sync interval:

Periodically, the filestore needs to quiesce writes and do a syncfs in
order to create
a consistent commit point up to which it can free journal entries.  Syncing more
frequently tends to reduce the time required to do the sync, and
reduces the amount
of data that needs to remain in the journal.  Less frequent syncs
would allow the
backing filesystem to better coalesce small writes and metadata
updates hopefully
resulting in more efficient syncs.  'filestore max sync interval'
defines the maximum
time period between syncs, 'filestore min sync interval' defines the
minimum time
period between syncs.

filestore flusher:

The filestore flusher forces data from large writes to be written out
using sync_file_range
before the sync in order to (hopefully) reduce the cost of the
eventual sync.  In practice,
disabling 'filestore flusher' seems to improve performance in some cases.

filestore queue max ops:

'filestore queue max ops' defines the number of in progress ops the
filestore will accept
before blocking on queueing new ones.  This mostly shouldn't have much
of an effect
on performance and should probably be ignored.

filestore op threads:

'filestore op threads' defines the number of threads used to submit
filesystem operations
in parallel.

journal dio:

'journal dio' enables using O_DIRECT for writing to the journal.  This
should usually
be enabled.  If possible, 'journal aio' should also be enabled to
allow use of libaio
to do asynchronous writes.

osd op threads:

'osd op threads' defines the size of the thread pool used to service
OSD operations
such as client requests.  Increasing this may increase the rate of
request processing.

osd disk threads:

'osd disk threads' defines the number of threads used to perform background disk
intensive osd operations such as scrubbing and snap trimming.

On Wed, Aug 29, 2012 at 12:29 PM, Dieter Kasper <d.kasper@kabelmail.de> wrote:
> Hi Josh,
>
> thanks for the hint.
> Can you please spend a view words about the meaing of these parameters ?
> - filestore min/max sync interval =     int/float ?     seconds ? of what ?
> - filestore flusher = false
> - filestore queue max ops = 10000
>         what is 'one op' ?      queue in front of what ?
> - filestore op threads =
>         what are useful values here ?
>
> - journal dio = true/false
> - osd op threads =
> - osd disk threads =
>
>
> Kind Regards,
> -Dieter
>
>
> On Wed, Aug 29, 2012 at 07:37:36PM +0200, Josh Durgin wrote:
>> On 08/29/2012 01:50 AM, Alexandre DERUMIER wrote:
>> > Nice results !
>> > (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
>> > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
>> >
>> >>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>> > I think you can try to tune these values
>> >
>> > filestore max sync interval = 30
>> > filestore min sync interval = 29
>> > filestore flusher = false
>> > filestore queue max ops = 10000
>>
>> Increasing filestore_op_threads might help as well.
>>
>> > ----- Mail original -----
>> >
>> > De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> > À: ceph-devel@vger.kernel.org
>> > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
>> > Envoyé: Mardi 28 Août 2012 19:48:42
>> > Objet: RBD performance - tuning hints
>> >
>> > Hi,
>> >
>> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
>> > I can observe a pretty nice rados bench performance
>> > (see bench-rados.txt for details):
>> >
>> > Bandwidth (MB/sec): 961.710
>> > Max bandwidth (MB/sec): 1040
>> > Min bandwidth (MB/sec): 772
>> >
>> >
>> > Also the bandwidth performance generated with
>> > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
>> >
>> > .... is acceptable, e.g.
>> > fio_write_4m_16 795 MB/s
>> > fio_randwrite_8m_128 717 MB/s
>> > fio_randwrite_8m_16 714 MB/s
>> > fio_randwrite_2m_32 692 MB/s
>> >
>> >
>> > But, the write IOPS seems to be limited around 19k ...
>> > RBD 4M 64k (= optimal_io_size)
>> > fio_randread_512_128 53286 55925
>> > fio_randread_4k_128 51110 44382
>> > fio_randread_8k_128 30854 29938
>> > fio_randwrite_512_128 18888 2386
>> > fio_randwrite_512_64 18844 2582
>> > fio_randwrite_8k_64 17350 2445
>> > (...)
>> > fio_read_4k_128 10073 53151
>> > fio_read_4k_64 9500 39757
>> > fio_read_4k_32 9220 23650
>> > (...)
>> > fio_read_4k_16 9122 14322
>> > fio_write_4k_128 2190 14306
>> > fio_read_8k_32 706 13894
>> > fio_write_4k_64 2197 12297
>> > fio_write_8k_64 3563 11705
>> > fio_write_8k_128 3444 11219
>> >
>> >
>> > Any hints for tuning the IOPS (read and/or write) would be appreciated.
>> >
>> > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>> >
>> >
>> > Kind Regards,
>> > -Dieter
>> >
>> >
>> >
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Dieter Kasper]

Hi Alexandre,

with the 4 filestore parameter below some fio values could be increased:
  filestore max sync interval = 30
  filestore min sync interval = 29
  filestore flusher = false
  filestore queue max ops = 10000

###### IOPS 
fio_read_4k_64:              9373 
fio_read_4k_128:             9939 
fio_randwrite_8k_16:        12376 
fio_randwrite_4k_16:        13315 
fio_randwrite_512_32:       13660 
fio_randwrite_8k_32:        17318 
fio_randwrite_4k_32:        18057 
fio_randwrite_8k_64:        19693 
fio_randwrite_512_64:       20015 	<<<
fio_randwrite_4k_64:        20024 	<<<
fio_randwrite_8k_128:       20547 	<<<
fio_randwrite_4k_128:       20839 	<<<
fio_randwrite_512_128:      21417 	<<<
fio_randread_8k_128:        48872 
fio_randread_4k_128:        50002 
fio_randread_512_128:       51202 

###### MB/s 
fio_randread_2m_32:           628 
fio_read_4m_64:               630 
fio_randread_8m_32:           633 
fio_read_2m_32:               637 
fio_read_4m_16:               640 
fio_randread_4m_16:           652 
fio_write_2m_32:              660 
fio_randread_4m_32:           677 
fio_read_4m_32:               678 
(...)
fio_write_4m_64:              771 
fio_randwrite_2m_64:          789 
fio_write_8m_128:             796 
fio_write_4m_32:              802 
fio_randwrite_4m_128:         807 	<<<
fio_randwrite_2m_32:          811 	<<<
fio_write_2m_128:             833 	<<<
fio_write_8m_64:              901 	<<<

Best Regards,
-Dieter


On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote:
> Nice results !
> (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
> I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
> 
> >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> I think you can try to tune these values
> 
> filestore max sync interval = 30
> filestore min sync interval = 29
> filestore flusher = false
> filestore queue max ops = 10000
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: ceph-devel@vger.kernel.org 
> Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
> Envoyé: Mardi 28 Août 2012 19:48:42 
> Objet: RBD performance - tuning hints 
> 
> Hi, 
> 
> on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
> I can observe a pretty nice rados bench performance 
> (see bench-rados.txt for details): 
> 
> Bandwidth (MB/sec): 961.710 
> Max bandwidth (MB/sec): 1040 
> Min bandwidth (MB/sec): 772 
> 
> 
> Also the bandwidth performance generated with 
> fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
> 
> .... is acceptable, e.g. 
> fio_write_4m_16 795 MB/s 
> fio_randwrite_8m_128 717 MB/s 
> fio_randwrite_8m_16 714 MB/s 
> fio_randwrite_2m_32 692 MB/s 
> 
> 
> But, the write IOPS seems to be limited around 19k ... 
> RBD 4M 64k (= optimal_io_size) 
> fio_randread_512_128 53286 55925 
> fio_randread_4k_128 51110 44382 
> fio_randread_8k_128 30854 29938 
> fio_randwrite_512_128 18888 2386 
> fio_randwrite_512_64 18844 2582 
> fio_randwrite_8k_64 17350 2445 
> (...) 
> fio_read_4k_128 10073 53151 
> fio_read_4k_64 9500 39757 
> fio_read_4k_32 9220 23650 
> (...) 
> fio_read_4k_16 9122 14322 
> fio_write_4k_128 2190 14306 
> fio_read_8k_32 706 13894 
> fio_write_4k_64 2197 12297 
> fio_write_8k_64 3563 11705 
> fio_write_8k_128 3444 11219 
> 
> 
> Any hints for tuning the IOPS (read and/or write) would be appreciated. 
> 
> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> 
> 
> Kind Regards, 
> -Dieter 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 	
> 
> Alexandre D e rumier 
> 
> Ingénieur Systèmes et Réseaux 
> 
> 
> Fixe : 03 20 68 88 85 
> 
> Fax : 03 20 68 90 88 
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints / parameter doc

[Dieter Kasper]

Samuel,

thank you very much for this explicitely description!

As far as I understand the journal acts as a ringbuffer in front of the OSD.
Using time as a parameter to trigger sync might not be the best for 
a dynamic Storage subsystem. On a high workload e.g. 10/20 for min/max 
might be optimal for for 4 nodes with 10 OSDs each, 
but not after adding 4 additional nodes.

Are there parameters to trigger the syncs to OSD
in relation to the fill grade of the journal ?
e.g.
filestore [min|max] sync percent:

Do not sync before min-% full; sync after max-% full

What would happen if I set "filestore [min|max] sync interval" to 999999 ?
Will the journal sync start at 100% full or at X% ?
What is 'X' by defaut ?
How can I set 'X' ?

Best Regards,
-Dieter


On Thu, Aug 30, 2012 at 12:34:43AM +0200, Samuel Just wrote:
> filestore [min|max] sync interval:
> 
> Periodically, the filestore needs to quiesce writes and do a syncfs in
> order to create
> a consistent commit point up to which it can free journal entries.  Syncing more
> frequently tends to reduce the time required to do the sync, and
> reduces the amount
> of data that needs to remain in the journal.  Less frequent syncs
> would allow the
> backing filesystem to better coalesce small writes and metadata
> updates hopefully
> resulting in more efficient syncs.  'filestore max sync interval'
> defines the maximum
> time period between syncs, 'filestore min sync interval' defines the
> minimum time
> period between syncs.
> 
> filestore flusher:
> 
> The filestore flusher forces data from large writes to be written out
> using sync_file_range
> before the sync in order to (hopefully) reduce the cost of the
> eventual sync.  In practice,
> disabling 'filestore flusher' seems to improve performance in some cases.
> 
> filestore queue max ops:
> 
> 'filestore queue max ops' defines the number of in progress ops the
> filestore will accept
> before blocking on queueing new ones.  This mostly shouldn't have much
> of an effect
> on performance and should probably be ignored.
> 
> filestore op threads:
> 
> 'filestore op threads' defines the number of threads used to submit
> filesystem operations
> in parallel.
> 
> journal dio:
> 
> 'journal dio' enables using O_DIRECT for writing to the journal.  This
> should usually
> be enabled.  If possible, 'journal aio' should also be enabled to
> allow use of libaio
> to do asynchronous writes.
> 
> osd op threads:
> 
> 'osd op threads' defines the size of the thread pool used to service
> OSD operations
> such as client requests.  Increasing this may increase the rate of
> request processing.
> 
> osd disk threads:
> 
> 'osd disk threads' defines the number of threads used to perform background disk
> intensive osd operations such as scrubbing and snap trimming.
> 
> On Wed, Aug 29, 2012 at 12:29 PM, Dieter Kasper <d.kasper@kabelmail.de> wrote:
> > Hi Josh,
> >
> > thanks for the hint.
> > Can you please spend a view words about the meaing of these parameters ?
> > - filestore min/max sync interval =     int/float ?     seconds ? of what ?
> > - filestore flusher = false
> > - filestore queue max ops = 10000
> >         what is 'one op' ?      queue in front of what ?
> > - filestore op threads =
> >         what are useful values here ?
> >
> > - journal dio = true/false
> > - osd op threads =
> > - osd disk threads =
> >
> >
> > Kind Regards,
> > -Dieter
> >
> >
> > On Wed, Aug 29, 2012 at 07:37:36PM +0200, Josh Durgin wrote:
> >> On 08/29/2012 01:50 AM, Alexandre DERUMIER wrote:
> >> > Nice results !
> >> > (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
> >> > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
> >> >
> >> >>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> >> > I think you can try to tune these values
> >> >
> >> > filestore max sync interval = 30
> >> > filestore min sync interval = 29
> >> > filestore flusher = false
> >> > filestore queue max ops = 10000
> >>
> >> Increasing filestore_op_threads might help as well.
> >>
> >> > ----- Mail original -----
> >> >
> >> > De: "Dieter Kasper" <d.kasper@kabelmail.de>
> >> > À: ceph-devel@vger.kernel.org
> >> > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
> >> > Envoyé: Mardi 28 Août 2012 19:48:42
> >> > Objet: RBD performance - tuning hints
> >> >
> >> > Hi,
> >> >
> >> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
> >> > I can observe a pretty nice rados bench performance
> >> > (see bench-rados.txt for details):
> >> >
> >> > Bandwidth (MB/sec): 961.710
> >> > Max bandwidth (MB/sec): 1040
> >> > Min bandwidth (MB/sec): 772
> >> >
> >> >
> >> > Also the bandwidth performance generated with
> >> > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
> >> >
> >> > .... is acceptable, e.g.
> >> > fio_write_4m_16 795 MB/s
> >> > fio_randwrite_8m_128 717 MB/s
> >> > fio_randwrite_8m_16 714 MB/s
> >> > fio_randwrite_2m_32 692 MB/s
> >> >
> >> >
> >> > But, the write IOPS seems to be limited around 19k ...
> >> > RBD 4M 64k (= optimal_io_size)
> >> > fio_randread_512_128 53286 55925
> >> > fio_randread_4k_128 51110 44382
> >> > fio_randread_8k_128 30854 29938
> >> > fio_randwrite_512_128 18888 2386
> >> > fio_randwrite_512_64 18844 2582
> >> > fio_randwrite_8k_64 17350 2445
> >> > (...)
> >> > fio_read_4k_128 10073 53151
> >> > fio_read_4k_64 9500 39757
> >> > fio_read_4k_32 9220 23650
> >> > (...)
> >> > fio_read_4k_16 9122 14322
> >> > fio_write_4k_128 2190 14306
> >> > fio_read_8k_32 706 13894
> >> > fio_write_4k_64 2197 12297
> >> > fio_write_8k_64 3563 11705
> >> > fio_write_8k_128 3444 11219
> >> >
> >> >
> >> > Any hints for tuning the IOPS (read and/or write) would be appreciated.
> >> >
> >> > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> >> >
> >> >
> >> > Kind Regards,
> >> > -Dieter
> >> >
> >> >
> >> >
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Alexandre DERUMIER]

Thanks for the report !

vs your first benchmark, it's with RBD 4M or 64K ?

(how much ssd by node?)



----- Mail original ----- 

De: "Dieter Kasper" <d.kasper@kabelmail.de> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Jeudi 30 Août 2012 16:56:34 
Objet: Re: RBD performance - tuning hints 

Hi Alexandre, 

with the 4 filestore parameter below some fio values could be increased: 
filestore max sync interval = 30 
filestore min sync interval = 29 
filestore flusher = false 
filestore queue max ops = 10000 

###### IOPS 
fio_read_4k_64: 9373 
fio_read_4k_128: 9939 
fio_randwrite_8k_16: 12376 
fio_randwrite_4k_16: 13315 
fio_randwrite_512_32: 13660 
fio_randwrite_8k_32: 17318 
fio_randwrite_4k_32: 18057 
fio_randwrite_8k_64: 19693 
fio_randwrite_512_64: 20015 <<< 
fio_randwrite_4k_64: 20024 <<< 
fio_randwrite_8k_128: 20547 <<< 
fio_randwrite_4k_128: 20839 <<< 
fio_randwrite_512_128: 21417 <<< 
fio_randread_8k_128: 48872 
fio_randread_4k_128: 50002 
fio_randread_512_128: 51202 

###### MB/s 
fio_randread_2m_32: 628 
fio_read_4m_64: 630 
fio_randread_8m_32: 633 
fio_read_2m_32: 637 
fio_read_4m_16: 640 
fio_randread_4m_16: 652 
fio_write_2m_32: 660 
fio_randread_4m_32: 677 
fio_read_4m_32: 678 
(...) 
fio_write_4m_64: 771 
fio_randwrite_2m_64: 789 
fio_write_8m_128: 796 
fio_write_4m_32: 802 
fio_randwrite_4m_128: 807 <<< 
fio_randwrite_2m_32: 811 <<< 
fio_write_2m_128: 833 <<< 
fio_write_8m_64: 901 <<< 

Best Regards, 
-Dieter 


On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote: 
> Nice results ! 
> (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
> I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster) 
> 
> >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> I think you can try to tune these values 
> 
> filestore max sync interval = 30 
> filestore min sync interval = 29 
> filestore flusher = false 
> filestore queue max ops = 10000 
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: ceph-devel@vger.kernel.org 
> Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
> Envoyé: Mardi 28 Août 2012 19:48:42 
> Objet: RBD performance - tuning hints 
> 
> Hi, 
> 
> on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
> I can observe a pretty nice rados bench performance 
> (see bench-rados.txt for details): 
> 
> Bandwidth (MB/sec): 961.710 
> Max bandwidth (MB/sec): 1040 
> Min bandwidth (MB/sec): 772 
> 
> 
> Also the bandwidth performance generated with 
> fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
> 
> .... is acceptable, e.g. 
> fio_write_4m_16 795 MB/s 
> fio_randwrite_8m_128 717 MB/s 
> fio_randwrite_8m_16 714 MB/s 
> fio_randwrite_2m_32 692 MB/s 
> 
> 
> But, the write IOPS seems to be limited around 19k ... 
> RBD 4M 64k (= optimal_io_size) 
> fio_randread_512_128 53286 55925 
> fio_randread_4k_128 51110 44382 
> fio_randread_8k_128 30854 29938 
> fio_randwrite_512_128 18888 2386 
> fio_randwrite_512_64 18844 2582 
> fio_randwrite_8k_64 17350 2445 
> (...) 
> fio_read_4k_128 10073 53151 
> fio_read_4k_64 9500 39757 
> fio_read_4k_32 9220 23650 
> (...) 
> fio_read_4k_16 9122 14322 
> fio_write_4k_128 2190 14306 
> fio_read_8k_32 706 13894 
> fio_write_4k_64 2197 12297 
> fio_write_8k_64 3563 11705 
> fio_write_8k_128 3444 11219 
> 
> 
> Any hints for tuning the IOPS (read and/or write) would be appreciated. 
> 
> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> 
> 
> Kind Regards, 
> -Dieter 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> 
> Alexandre D e rumier 
> 
> Ingénieur Systèmes et Réseaux 
> 
> 
> Fixe : 03 20 68 88 85 
> 
> Fax : 03 20 68 90 88 
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 




-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Dieter Kasper]

[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]

On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote:
> Thanks for the report !
> 
> vs your first benchmark, it's with RBD 4M or 64K ?
with 4MB (see attached config info)

Cheers,
-Dieter

> 
> (how much ssd by node?)
8x SSD, 200GB each

> 
> 
> 
> ----- Mail original -----
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de>
> À: "Alexandre DERUMIER" <aderumier@odiso.com>
> Cc: ceph-devel@vger.kernel.org
> Envoyé: Jeudi 30 Août 2012 16:56:34
> Objet: Re: RBD performance - tuning hints
> 
> Hi Alexandre,
> 
> with the 4 filestore parameter below some fio values could be increased:
> filestore max sync interval = 30
> filestore min sync interval = 29
> filestore flusher = false
> filestore queue max ops = 10000
> 
> ###### IOPS
> fio_read_4k_64: 9373
> fio_read_4k_128: 9939
> fio_randwrite_8k_16: 12376
> fio_randwrite_4k_16: 13315
> fio_randwrite_512_32: 13660
> fio_randwrite_8k_32: 17318
> fio_randwrite_4k_32: 18057
> fio_randwrite_8k_64: 19693
> fio_randwrite_512_64: 20015 <<<
> fio_randwrite_4k_64: 20024 <<<
> fio_randwrite_8k_128: 20547 <<<
> fio_randwrite_4k_128: 20839 <<<
> fio_randwrite_512_128: 21417 <<<
> fio_randread_8k_128: 48872
> fio_randread_4k_128: 50002
> fio_randread_512_128: 51202
> 
> ###### MB/s
> fio_randread_2m_32: 628
> fio_read_4m_64: 630
> fio_randread_8m_32: 633
> fio_read_2m_32: 637
> fio_read_4m_16: 640
> fio_randread_4m_16: 652
> fio_write_2m_32: 660
> fio_randread_4m_32: 677
> fio_read_4m_32: 678
> (...)
> fio_write_4m_64: 771
> fio_randwrite_2m_64: 789
> fio_write_8m_128: 796
> fio_write_4m_32: 802
> fio_randwrite_4m_128: 807 <<<
> fio_randwrite_2m_32: 811 <<<
> fio_write_2m_128: 833 <<<
> fio_write_8m_64: 901 <<<
> 
> Best Regards,
> -Dieter
> 
> 
> On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote:
> > Nice results !
> > (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
> > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
> >
> > >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> > I think you can try to tune these values
> >
> > filestore max sync interval = 30
> > filestore min sync interval = 29
> > filestore flusher = false
> > filestore queue max ops = 10000
> >
> >
> >
> > ----- Mail original -----
> >
> > De: "Dieter Kasper" <d.kasper@kabelmail.de>
> > À: ceph-devel@vger.kernel.org
> > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
> > Envoyé: Mardi 28 Août 2012 19:48:42
> > Objet: RBD performance - tuning hints
> >
> > Hi,
> >
> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
> > I can observe a pretty nice rados bench performance
> > (see bench-rados.txt for details):
> >
> > Bandwidth (MB/sec): 961.710
> > Max bandwidth (MB/sec): 1040
> > Min bandwidth (MB/sec): 772
> >
> >
> > Also the bandwidth performance generated with
> > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
> >
> > .... is acceptable, e.g.
> > fio_write_4m_16 795 MB/s
> > fio_randwrite_8m_128 717 MB/s
> > fio_randwrite_8m_16 714 MB/s
> > fio_randwrite_2m_32 692 MB/s
> >
> >
> > But, the write IOPS seems to be limited around 19k ...
> > RBD 4M 64k (= optimal_io_size)
> > fio_randread_512_128 53286 55925
> > fio_randread_4k_128 51110 44382
> > fio_randread_8k_128 30854 29938
> > fio_randwrite_512_128 18888 2386
> > fio_randwrite_512_64 18844 2582
> > fio_randwrite_8k_64 17350 2445
> > (...)
> > fio_read_4k_128 10073 53151
> > fio_read_4k_64 9500 39757
> > fio_read_4k_32 9220 23650
> > (...)
> > fio_read_4k_16 9122 14322
> > fio_write_4k_128 2190 14306
> > fio_read_8k_32 706 13894
> > fio_write_4k_64 2197 12297
> > fio_write_8k_64 3563 11705
> > fio_write_8k_128 3444 11219
> >
> >
> > Any hints for tuning the IOPS (read and/or write) would be appreciated.
> >
> > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
> >
> >
> > Kind Regards,
> > -Dieter
> >
> >
> >
> > --
> >
> > --
> >
> >
> >
> >
> >
> > Alexandre D e rumier
> >
> > Ingénieur Systèmes et Réseaux
> >
> >
> > Fixe : 03 20 68 88 85
> >
> > Fax : 03 20 68 90 88
> >
> >
> > 45 Bvd du Général Leclerc 59100 Roubaix
> > 12 rue Marivaux 75002 Paris
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> 
> --
> 
> --
> 
> 
> 
> 
> 
> Alexandre D e rumier
> 
> Ingénieur Systèmes et Réseaux
> 
> 
> Fixe : 03 20 68 88 85
> 
> Fax : 03 20 68 90 88
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> 

[-- Attachment #2: hwconf.txt --]
[-- Type: text/plain, Size: 26784 bytes --]

--- RX37-3c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-3 3.0.41-5.1-default #1 SMP Wed Aug 22 00:54:03 UTC 2012 (9c63123) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856332 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdm 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdn 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdo 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdp 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdq 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdr 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sds 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdt 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     38 C
  Blocks sent to initiator = 257379169992704
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     40 C
  Blocks sent to initiator = 238453816033280
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     43 C
  Blocks sent to initiator = 297650494636032
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     34 C
  Blocks sent to initiator = 254438979665920
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     35 C
  Blocks sent to initiator = 238876987752448
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     37 C
  Blocks sent to initiator = 259011676995584
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     41 C
  Blocks sent to initiator = 359638046343168
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     31 C
  Blocks sent to initiator = 247008082264064
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdm on /data/osd.30 type xfs (rw,noatime)
/dev/sdn on /data/osd.31 type xfs (rw,noatime)
/dev/sdo on /data/osd.32 type xfs (rw,noatime)
/dev/sdp on /data/osd.33 type xfs (rw,noatime)
/dev/sdq on /data/osd.34 type xfs (rw,noatime)
/dev/sdr on /data/osd.35 type xfs (rw,noatime)
/dev/sds on /data/osd.36 type xfs (rw,noatime)
/dev/sdt on /data/osd.37 type xfs (rw,noatime)
--- RX37-4c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-4 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856432 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdd 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sde 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdf 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdg 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdh 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdi 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdj 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdk 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     34 C
  Blocks sent to initiator = 389173798240256
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     30 C
  Blocks sent to initiator = 286249688498176
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     35 C
  Blocks sent to initiator = 220455000604672
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     38 C
  Blocks sent to initiator = 223169319272448
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     31 C
  Blocks sent to initiator = 232096593346560
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 264802534424576
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     27 C
  Blocks sent to initiator = 288896512425984
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     32 C
  Blocks sent to initiator = 282331621359616
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdd on /data/osd.40 type xfs (rw,noatime)
/dev/sde on /data/osd.41 type xfs (rw,noatime)
/dev/sdf on /data/osd.42 type xfs (rw,noatime)
/dev/sdg on /data/osd.43 type xfs (rw,noatime)
/dev/sdh on /data/osd.44 type xfs (rw,noatime)
/dev/sdi on /data/osd.45 type xfs (rw,noatime)
/dev/sdj on /data/osd.46 type xfs (rw,noatime)
/dev/sdk on /data/osd.47 type xfs (rw,noatime)
--- RX37-5c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-5 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       74226012 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdo 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdp 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdq 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdr 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sds 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdt 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdu 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdv 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 247461838848000
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     38 C
  Blocks sent to initiator = 231320898764800
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     41 C
  Blocks sent to initiator = 290086906232832
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     32 C
  Blocks sent to initiator = 287719053852672
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     33 C
  Blocks sent to initiator = 243922265702400
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     35 C
  Blocks sent to initiator = 272285122428928
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     40 C
  Blocks sent to initiator = 279561266790400
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     29 C
  Blocks sent to initiator = 247978778427392
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdo on /data/osd.50 type xfs (rw,noatime)
/dev/sdp on /data/osd.51 type xfs (rw,noatime)
/dev/sdq on /data/osd.52 type xfs (rw,noatime)
/dev/sdr on /data/osd.53 type xfs (rw,noatime)
/dev/sds on /data/osd.54 type xfs (rw,noatime)
/dev/sdt on /data/osd.55 type xfs (rw,noatime)
/dev/sdu on /data/osd.56 type xfs (rw,noatime)
/dev/sdv on /data/osd.57 type xfs (rw,noatime)
--- RX37-6c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-6 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       32856344 kB
Disk /dev/ram0: 2048 MB, 2048000000 bytes
Disk /dev/ram1: 2048 MB, 2048000000 bytes
Disk /dev/ram2: 2048 MB, 2048000000 bytes
Disk /dev/ram3: 2048 MB, 2048000000 bytes
Disk /dev/ram4: 2048 MB, 2048000000 bytes
Disk /dev/ram5: 2048 MB, 2048000000 bytes
Disk /dev/ram6: 2048 MB, 2048000000 bytes
Disk /dev/ram7: 2048 MB, 2048000000 bytes
[10:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdn 
[10:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdo 
[10:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdp 
[10:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdq 
[11:0:0:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdr 
[11:0:1:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sds 
[11:0:2:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdt 
[11:0:3:0]   disk    INTEL(R)  SSD 910 200GB   a411  /dev/sdu 
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     41 C
  Blocks sent to initiator = 259148495192064
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     36 C
  Blocks sent to initiator = 250183472381952
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     43 C
  Blocks sent to initiator = 232864704626688
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     46 C
  Blocks sent to initiator = 313614921629696
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     37 C
  Blocks sent to initiator = 269851218149376
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     34 C
  Blocks sent to initiator = 278551060283392
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     43 C
  Blocks sent to initiator = 267839076302848
Device: INTEL(R)  SSD 910 200GB   Version: a411
Current Drive Temperature:     39 C
  Blocks sent to initiator = 233988811653120
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
/dev/sdn on /data/osd.60 type xfs (rw,noatime)
/dev/sdo on /data/osd.61 type xfs (rw,noatime)
/dev/sdp on /data/osd.62 type xfs (rw,noatime)
/dev/sdq on /data/osd.63 type xfs (rw,noatime)
/dev/sdr on /data/osd.64 type xfs (rw,noatime)
/dev/sds on /data/osd.65 type xfs (rw,noatime)
/dev/sdt on /data/osd.66 type xfs (rw,noatime)
/dev/sdu on /data/osd.67 type xfs (rw,noatime)
--- RX37-7c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-7 3.0.36-10-default #1 SMP Mon Jul 9 14:42:03 UTC 2012 (595894d) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 1.20 GHz (asserted by call to hardware).
MemTotal:       32856344 kB
optimal_io_size: 4194304
4194304
4194304
scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
noop deadline [cfq] 
--- RX37-8c --------------------------------------------------------------------
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
Linux RX37-8 3.0.36-16-default #1 SMP Wed Jul 18 00:18:54 UTC 2012 (544e41f) x86_64 x86_64 x86_64 GNU/Linux

model name	: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Logial CPUs: 12
  current CPU frequency is 2.30 GHz (asserted by call to hardware).
MemTotal:       65952088 kB
optimal_io_size: scheduler:       [noop] deadline cfq 
[noop] deadline cfq 
[noop] deadline cfq 
--------------------------------------------------------------------------------

dumped osdmap epoch 19
epoch 19
fsid 31dc8e8c-45cb-4b94-b581-a9258964f1a6
created 2012-08-29 22:08:58.870313
modifed 2012-08-29 22:09:50.084564
flags 

pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 4352 pgp_num 4352 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 4352 pgp_num 4352 last_change 1 owner 0
pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 4352 pgp_num 4352 last_change 1 owner 0
pool 3 'pbench' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 768 pgp_num 768 last_change 18 owner 0

max_osd 68
osd.30 up   in  weight 1 up_from 3 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6800/24876 192.168.114.52:6800/24876 192.168.114.52:6801/24876 exists,up 0a9a6db3-1c0d-4d66-ac99-bd900076c42c
osd.31 up   in  weight 1 up_from 3 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6801/25090 192.168.114.52:6802/25090 192.168.114.52:6803/25090 exists,up 0adab61b-c1c3-479f-b58e-42bec92bd5b0
osd.32 up   in  weight 1 up_from 3 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6802/25276 192.168.114.52:6804/25276 192.168.114.52:6805/25276 exists,up 331bf096-d785-4ae8-b790-d746a0abb694
osd.33 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6803/25464 192.168.114.52:6806/25464 192.168.114.52:6807/25464 exists,up a1f9ea5b-e0db-474c-b7bc-6cb3d3a213a4
osd.34 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6804/25650 192.168.114.52:6808/25650 192.168.114.52:6809/25650 exists,up dcbe68e7-fef3-430d-a857-560db28de27f
osd.35 up   in  weight 1 up_from 2 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6805/25838 192.168.114.52:6810/25838 192.168.114.52:6811/25838 exists,up ab1589d0-e725-4484-8f5d-f65bc5c64643
osd.36 up   in  weight 1 up_from 3 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6806/26026 192.168.114.52:6812/26026 192.168.114.52:6813/26026 exists,up 2eea079f-bcfe-48a4-abb5-a15c7daf80ba
osd.37 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.52:6807/26218 192.168.114.52:6814/26218 192.168.114.52:6815/26218 exists,up 9822d872-79a6-4cd3-898f-2e905fbce44a
osd.40 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6800/18525 192.168.114.53:6800/18525 192.168.114.53:6801/18525 exists,up 0f0c61ea-4d78-429c-9928-b3422ad2dec7
osd.41 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6801/18750 192.168.114.53:6802/18750 192.168.114.53:6803/18750 exists,up 3935c6a7-61ff-4c97-88b9-472051ba8b6c
osd.42 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6802/18946 192.168.114.53:6804/18946 192.168.114.53:6805/18946 exists,up 3efc6383-5097-4e95-9af2-e0e7bc9ddc10
osd.43 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6803/19154 192.168.114.53:6806/19154 192.168.114.53:6807/19154 exists,up cdb8cf82-077b-40c2-adbc-fae29ba41645
osd.44 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6804/19350 192.168.114.53:6808/19350 192.168.114.53:6809/19350 exists,up 5ab69e45-a73a-4cd4-9837-2d54fb4ea4ec
osd.45 up   in  weight 1 up_from 4 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6805/19546 192.168.114.53:6810/19546 192.168.114.53:6811/19546 exists,up ec3d2118-6f46-4ef8-a431-553710f33a18
osd.46 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6806/19766 192.168.114.53:6812/19766 192.168.114.53:6813/19766 exists,up dcd94df3-b679-46a6-b670-5269a29913c1
osd.47 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.53:6807/19968 192.168.114.53:6814/19968 192.168.114.53:6815/19968 exists,up 41019d97-c4f3-4c8d-9189-bae642c31678
osd.50 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6800/3848 192.168.114.54:6800/3848 192.168.114.54:6801/3848 exists,up 0b9ebe8e-9cb8-440d-948e-d4c8aa16b407
osd.51 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6801/4061 192.168.114.54:6802/4061 192.168.114.54:6803/4061 exists,up 3c2e8031-d01d-4bf9-965e-1b77563d5f8f
osd.52 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6802/4248 192.168.114.54:6804/4248 192.168.114.54:6805/4248 exists,up 4d641c3c-0a7a-4b20-b047-9042b61685bb
osd.53 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6803/4446 192.168.114.54:6806/4446 192.168.114.54:6807/4446 exists,up e335a6e9-9c32-48c6-8f15-11aa84a6287d
osd.54 up   in  weight 1 up_from 5 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6804/4632 192.168.114.54:6808/4632 192.168.114.54:6809/4632 exists,up 16f3955c-9eee-442b-86d8-cbbc5938efbf
osd.55 up   in  weight 1 up_from 6 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6805/4836 192.168.114.54:6810/4836 192.168.114.54:6811/4836 exists,up 83e59145-9ff8-4c0b-b066-2b2e4e9c9953
osd.56 up   in  weight 1 up_from 6 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6806/5029 192.168.114.54:6812/5029 192.168.114.54:6813/5029 exists,up dfdeb186-5c96-4466-b4d3-5f32fa712792
osd.57 up   in  weight 1 up_from 7 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.54:6807/5351 192.168.114.54:6814/5351 192.168.114.54:6815/5351 exists,up adf7a484-b0f1-4bf7-a8e7-2c1e64dfb77f
osd.60 up   in  weight 1 up_from 7 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6800/31038 192.168.114.55:6800/31038 192.168.114.55:6801/31038 exists,up e9b949c8-1b47-4749-9408-1e9f7b89b0e6
osd.61 up   in  weight 1 up_from 8 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6801/31257 192.168.114.55:6802/31257 192.168.114.55:6803/31257 exists,up 19fcad53-d951-4645-a6d5-7dad1deba6fb
osd.62 up   in  weight 1 up_from 8 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6802/31449 192.168.114.55:6804/31449 192.168.114.55:6805/31449 exists,up 7e98db0e-2ae2-473d-9b03-798ec472b29b
osd.63 up   in  weight 1 up_from 9 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6803/31641 192.168.114.55:6806/31641 192.168.114.55:6807/31641 exists,up 9abc714c-06e4-40ba-8afe-8465209e0272
osd.64 up   in  weight 1 up_from 9 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6804/31937 192.168.114.55:6808/31937 192.168.114.55:6809/31937 exists,up 6a20e4b1-d1e9-4f69-b903-b403136ddb1d
osd.65 up   in  weight 1 up_from 10 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6805/32175 192.168.114.55:6810/32175 192.168.114.55:6811/32175 exists,up e95ad5b2-6866-4161-8060-781a31d7ece2
osd.66 up   in  weight 1 up_from 10 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6806/32487 192.168.114.55:6812/32487 192.168.114.55:6813/32487 exists,up f3126979-ecd6-45de-b0bf-54cb2b0af042
osd.67 up   in  weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) 192.168.113.55:6807/32679 192.168.114.55:6814/32679 192.168.114.55:6815/32679 exists,up 37d3f121-b6f4-4c6f-ac9b-30533e8fa60a



ceph.conf
---content---
# global
[global]
	# enable secure authentication
	auth supported = none

        # allow ourselves to open a lot of files
        #max open files = 1100000
        max open files = 131072

        # set log file
        log file = /ceph/log/$name.log
        # log_to_syslog = true        # uncomment this line to log to syslog

        # set up pid files
        pid file = /var/run/ceph/$name.pid

        # If you want to run a IPv6 cluster, set this to true. Dual-stack isn't possible
        #ms bind ipv6 = true
	public network = 192.168.113.0/24
	cluster network = 192.168.114.0/24

# monitors
#  You need at least one.  You need at least three if you want to
#  tolerate any node failures.  Always create an odd number.
[mon]
        mon data = /ceph/$name

        # If you are using for example the RADOS Gateway and want to have your newly created
        # pools a higher replication level, you can set a default
        #osd pool default size = 3

        # You can also specify a CRUSH rule for new pools
        # Wiki: http://ceph.newdream.net/wiki/Custom_data_placement_with_CRUSH
        #osd pool default crush rule = 0

        # Timing is critical for monitors, but if you want to allow the clocks to drift a
        # bit more, you can specify the max drift.
        #mon clock drift allowed = 1

        # Tell the monitor to backoff from this warning for 30 seconds
        #mon clock drift warn backoff = 30

	# logging, for debugging monitor crashes, in order of
	# their likelihood of being helpful :)
	#debug ms = 1
	#debug mon = 20
	#debug paxos = 20
	#debug auth = 20
	debug optracker = 0

[mon.0]
	host = RX37-3c
	mon addr = 192.168.113.52:6789
[mon.1]
	host = RX37-7c
	mon addr = 192.168.113.56:6789
[mon.2]
	host = RX37-8c
	mon addr = 192.168.113.57:6789
	
# mds
#  You need at least one.  Define two to get a standby.
[mds]
#        mds data = /ceph/$name
	# where the mds keeps it's secret encryption keys
	#keyring = /data/keyring.$name

	# mds logging to debug issues.
	#debug ms = 1
	#debug mds = 20
	debug optracker = 0

[mds.0]
        host = RX37-8c

# osd
#  You need at least one.  Two if you want data to be replicated.
#  Define as many as you like.
[osd]
	# This is where the btrfs volume will be mounted.
	osd data = /data/$name

#        journal dio = true
#        osd op threads = 24
#        osd disk threads = 24
#        filestore op threads = 6
#        filestore queue max ops = 24
	filestore max sync interval = 30
	filestore min sync interval = 29
	filestore flusher = false
	filestore queue max ops = 10000

	# Ideally, make this a separate disk or partition.  A few
 	# hundred MB should be enough; more if you have fast or many
 	# disks.  You can use a file under the osd data dir if need be
 	# (e.g. /data/$name/journal), but it will be slower than a
 	# separate disk or partition.

        # This is an example of a file-based journal.
	# osd journal = /ceph/$name/journal
	# osd journal size = 2048 
	# journal size, in megabytes

        # If you want to run the journal on a tmpfs, disable DirectIO
        #journal dio = false

        # You can change the number of recovery operations to speed up recovery
        # or slow it down if your machines can't handle it
        # osd recovery max active = 3

	# osd logging to debug osd issues, in order of likelihood of being
	# helpful
	#debug ms = 1
	#debug osd = 20
	#debug filestore = 20
	#debug journal = 20
	debug optracker = 0
	fstype = xfs

[osd.30]
	host = RX37-3c
	devs = /dev/sdm
	osd journal = /dev/ram0
[osd.31]
	host = RX37-3c
	devs = /dev/sdn
	osd journal = /dev/ram1
[osd.32]
	host = RX37-3c
	devs = /dev/sdo
	osd journal = /dev/ram2
[osd.33]
	host = RX37-3c
	devs = /dev/sdp
	osd journal = /dev/ram3
[osd.34]
	host = RX37-3c
	devs = /dev/sdq
	osd journal = /dev/ram4
[osd.35]
	host = RX37-3c
	devs = /dev/sdr
	osd journal = /dev/ram5
[osd.36]
	host = RX37-3c
	devs = /dev/sds
	osd journal = /dev/ram6
[osd.37]
	host = RX37-3c
	devs = /dev/sdt
	osd journal = /dev/ram7
[osd.40]
	host = RX37-4c
	devs = /dev/sdd
	osd journal = /dev/ram0
[osd.41]
	host = RX37-4c
	devs = /dev/sde
	osd journal = /dev/ram1
[osd.42]
	host = RX37-4c
	devs = /dev/sdf
	osd journal = /dev/ram2
[osd.43]
	host = RX37-4c
	devs = /dev/sdg
	osd journal = /dev/ram3
[osd.44]
	host = RX37-4c
	devs = /dev/sdh
	osd journal = /dev/ram4
[osd.45]
	host = RX37-4c
	devs = /dev/sdi
	osd journal = /dev/ram5
[osd.46]
	host = RX37-4c
	devs = /dev/sdj
	osd journal = /dev/ram6
[osd.47]
	host = RX37-4c
	devs = /dev/sdk
	osd journal = /dev/ram7
[osd.50]
	host = RX37-5c
	devs = /dev/sdo
	osd journal = /dev/ram0
[osd.51]
	host = RX37-5c
	devs = /dev/sdp
	osd journal = /dev/ram1
[osd.52]
	host = RX37-5c
	devs = /dev/sdq
	osd journal = /dev/ram2
[osd.53]
	host = RX37-5c
	devs = /dev/sdr
	osd journal = /dev/ram3
[osd.54]
	host = RX37-5c
	devs = /dev/sds
	osd journal = /dev/ram4
[osd.55]
	host = RX37-5c
	devs = /dev/sdt
	osd journal = /dev/ram5
[osd.56]
	host = RX37-5c
	devs = /dev/sdu
	osd journal = /dev/ram6
[osd.57]
	host = RX37-5c
	devs = /dev/sdv
	osd journal = /dev/ram7
[osd.60]
	host = RX37-6c
	devs = /dev/sdn
	osd journal = /dev/ram0
[osd.61]
	host = RX37-6c
	devs = /dev/sdo
	osd journal = /dev/ram1
[osd.62]
	host = RX37-6c
	devs = /dev/sdp
	osd journal = /dev/ram2
[osd.63]
	host = RX37-6c
	devs = /dev/sdq
	osd journal = /dev/ram3
[osd.64]
	host = RX37-6c
	devs = /dev/sdr
	osd journal = /dev/ram4
[osd.65]
	host = RX37-6c
	devs = /dev/sds
	osd journal = /dev/ram5
[osd.66]
	host = RX37-6c
	devs = /dev/sdt
	osd journal = /dev/ram6
[osd.67]
	host = RX37-6c
	devs = /dev/sdu
	osd journal = /dev/ram7
	devs = /dev/sdc

[client.01]
	client hostname = RX37-7c

Re: RBD performance - tuning hints

[Alexandre DERUMIER]

Thanks

>> 8x SSD, 200GB each 

20000 iops seem pretty low,no ?


for @intank:

Is their a bottleneck somewhere in ceph ?

I said that, because I would like to know if it's scale by adding new nodes.

Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list)


----- Mail original ----- 

De: "Dieter Kasper" <d.kasper@kabelmail.de> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Jeudi 30 Août 2012 17:33:42 
Objet: Re: RBD performance - tuning hints 

On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote: 
> Thanks for the report ! 
> 
> vs your first benchmark, it's with RBD 4M or 64K ? 
with 4MB (see attached config info) 

Cheers, 
-Dieter 

> 
> (how much ssd by node?) 
8x SSD, 200GB each 

> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Jeudi 30 Août 2012 16:56:34 
> Objet: Re: RBD performance - tuning hints 
> 
> Hi Alexandre, 
> 
> with the 4 filestore parameter below some fio values could be increased: 
> filestore max sync interval = 30 
> filestore min sync interval = 29 
> filestore flusher = false 
> filestore queue max ops = 10000 
> 
> ###### IOPS 
> fio_read_4k_64: 9373 
> fio_read_4k_128: 9939 
> fio_randwrite_8k_16: 12376 
> fio_randwrite_4k_16: 13315 
> fio_randwrite_512_32: 13660 
> fio_randwrite_8k_32: 17318 
> fio_randwrite_4k_32: 18057 
> fio_randwrite_8k_64: 19693 
> fio_randwrite_512_64: 20015 <<< 
> fio_randwrite_4k_64: 20024 <<< 
> fio_randwrite_8k_128: 20547 <<< 
> fio_randwrite_4k_128: 20839 <<< 
> fio_randwrite_512_128: 21417 <<< 
> fio_randread_8k_128: 48872 
> fio_randread_4k_128: 50002 
> fio_randread_512_128: 51202 
> 
> ###### MB/s 
> fio_randread_2m_32: 628 
> fio_read_4m_64: 630 
> fio_randread_8m_32: 633 
> fio_read_2m_32: 637 
> fio_read_4m_16: 640 
> fio_randread_4m_16: 652 
> fio_write_2m_32: 660 
> fio_randread_4m_32: 677 
> fio_read_4m_32: 678 
> (...) 
> fio_write_4m_64: 771 
> fio_randwrite_2m_64: 789 
> fio_write_8m_128: 796 
> fio_write_4m_32: 802 
> fio_randwrite_4m_128: 807 <<< 
> fio_randwrite_2m_32: 811 <<< 
> fio_write_2m_128: 833 <<< 
> fio_write_8m_64: 901 <<< 
> 
> Best Regards, 
> -Dieter 
> 
> 
> On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote: 
> > Nice results ! 
> > (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
> > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster) 
> > 
> > >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > I think you can try to tune these values 
> > 
> > filestore max sync interval = 30 
> > filestore min sync interval = 29 
> > filestore flusher = false 
> > filestore queue max ops = 10000 
> > 
> > 
> > 
> > ----- Mail original ----- 
> > 
> > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > À: ceph-devel@vger.kernel.org 
> > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
> > Envoyé: Mardi 28 Août 2012 19:48:42 
> > Objet: RBD performance - tuning hints 
> > 
> > Hi, 
> > 
> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
> > I can observe a pretty nice rados bench performance 
> > (see bench-rados.txt for details): 
> > 
> > Bandwidth (MB/sec): 961.710 
> > Max bandwidth (MB/sec): 1040 
> > Min bandwidth (MB/sec): 772 
> > 
> > 
> > Also the bandwidth performance generated with 
> > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
> > 
> > .... is acceptable, e.g. 
> > fio_write_4m_16 795 MB/s 
> > fio_randwrite_8m_128 717 MB/s 
> > fio_randwrite_8m_16 714 MB/s 
> > fio_randwrite_2m_32 692 MB/s 
> > 
> > 
> > But, the write IOPS seems to be limited around 19k ... 
> > RBD 4M 64k (= optimal_io_size) 
> > fio_randread_512_128 53286 55925 
> > fio_randread_4k_128 51110 44382 
> > fio_randread_8k_128 30854 29938 
> > fio_randwrite_512_128 18888 2386 
> > fio_randwrite_512_64 18844 2582 
> > fio_randwrite_8k_64 17350 2445 
> > (...) 
> > fio_read_4k_128 10073 53151 
> > fio_read_4k_64 9500 39757 
> > fio_read_4k_32 9220 23650 
> > (...) 
> > fio_read_4k_16 9122 14322 
> > fio_write_4k_128 2190 14306 
> > fio_read_8k_32 706 13894 
> > fio_write_4k_64 2197 12297 
> > fio_write_8k_64 3563 11705 
> > fio_write_8k_128 3444 11219 
> > 
> > 
> > Any hints for tuning the IOPS (read and/or write) would be appreciated. 
> > 
> > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > 
> > 
> > Kind Regards, 
> > -Dieter 
> > 
> > 
> > 
> > -- 
> > 
> > -- 
> > 
> > 
> > 
> > 
> > 
> > Alexandre D e rumier 
> > 
> > Ingénieur Systèmes et Réseaux 
> > 
> > 
> > Fixe : 03 20 68 88 85 
> > 
> > Fax : 03 20 68 90 88 
> > 
> > 
> > 45 Bvd du Général Leclerc 59100 Roubaix 
> > 12 rue Marivaux 75002 Paris 
> > -- 
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> > the body of a message to majordomo@vger.kernel.org 
> > More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> 
> Alexandre D e rumier 
> 
> Ingénieur Systèmes et Réseaux 
> 
> 
> Fixe : 03 20 68 88 85 
> 
> Fax : 03 20 68 90 88 
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris 
> 



-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Dieter Kasper]

On Thu, Aug 30, 2012 at 05:46:35PM +0200, Alexandre DERUMIER wrote:
> Thanks
> 
> >> 8x SSD, 200GB each 
> 
> 20000 iops seem pretty low,no ?
well, you have to compare
- pure a SSD (via PCIe or SAS-6G)	vs.
- Ceph-Journal, which goes 2x over 10GbE with IP
  Client -> primary-copy -> 2nd-copy
  (= redundancy over Ethernet distance)

I'm curious about the answer from Inktank,

-Dieter

> 
> 
> for @intank:
> 
> Is their a bottleneck somewhere in ceph ?
Maybe "SimpleMessenger dispatching: cause of performance problems?"
from Thu, 16 Aug 2012 18:08:39 +0200
by <andreas.bluemle@itxperts.de>
can be an answer.
Especially if a small number of OSDs is used.

> 
> I said that, because I would like to know if it's scale by adding new nodes.
> 
> Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list)
> 
> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Jeudi 30 Août 2012 17:33:42 
> Objet: Re: RBD performance - tuning hints 
> 
> On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote: 
> > Thanks for the report ! 
> > 
> > vs your first benchmark, it's with RBD 4M or 64K ? 
> with 4MB (see attached config info) 
> 
> Cheers, 
> -Dieter 
> 
> > 
> > (how much ssd by node?) 
> 8x SSD, 200GB each 
> 
> > 
> > 
> > 
> > ----- Mail original ----- 
> > 
> > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> > Cc: ceph-devel@vger.kernel.org 
> > Envoyé: Jeudi 30 Août 2012 16:56:34 
> > Objet: Re: RBD performance - tuning hints 
> > 
> > Hi Alexandre, 
> > 
> > with the 4 filestore parameter below some fio values could be increased: 
> > filestore max sync interval = 30 
> > filestore min sync interval = 29 
> > filestore flusher = false 
> > filestore queue max ops = 10000 
> > 
> > ###### IOPS 
> > fio_read_4k_64: 9373 
> > fio_read_4k_128: 9939 
> > fio_randwrite_8k_16: 12376 
> > fio_randwrite_4k_16: 13315 
> > fio_randwrite_512_32: 13660 
> > fio_randwrite_8k_32: 17318 
> > fio_randwrite_4k_32: 18057 
> > fio_randwrite_8k_64: 19693 
> > fio_randwrite_512_64: 20015 <<< 
> > fio_randwrite_4k_64: 20024 <<< 
> > fio_randwrite_8k_128: 20547 <<< 
> > fio_randwrite_4k_128: 20839 <<< 
> > fio_randwrite_512_128: 21417 <<< 
> > fio_randread_8k_128: 48872 
> > fio_randread_4k_128: 50002 
> > fio_randread_512_128: 51202 
> > 
> > ###### MB/s 
> > fio_randread_2m_32: 628 
> > fio_read_4m_64: 630 
> > fio_randread_8m_32: 633 
> > fio_read_2m_32: 637 
> > fio_read_4m_16: 640 
> > fio_randread_4m_16: 652 
> > fio_write_2m_32: 660 
> > fio_randread_4m_32: 677 
> > fio_read_4m_32: 678 
> > (...) 
> > fio_write_4m_64: 771 
> > fio_randwrite_2m_64: 789 
> > fio_write_8m_128: 796 
> > fio_write_4m_32: 802 
> > fio_randwrite_4m_128: 807 <<< 
> > fio_randwrite_2m_32: 811 <<< 
> > fio_write_2m_128: 833 <<< 
> > fio_write_8m_64: 901 <<< 
> > 
> > Best Regards, 
> > -Dieter 
> > 
> > 
> > On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote: 
> > > Nice results ! 
> > > (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
> > > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster) 
> > > 
> > > >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > > I think you can try to tune these values 
> > > 
> > > filestore max sync interval = 30 
> > > filestore min sync interval = 29 
> > > filestore flusher = false 
> > > filestore queue max ops = 10000 
> > > 
> > > 
> > > 
> > > ----- Mail original ----- 
> > > 
> > > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > > À: ceph-devel@vger.kernel.org 
> > > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
> > > Envoyé: Mardi 28 Août 2012 19:48:42 
> > > Objet: RBD performance - tuning hints 
> > > 
> > > Hi, 
> > > 
> > > on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
> > > I can observe a pretty nice rados bench performance 
> > > (see bench-rados.txt for details): 
> > > 
> > > Bandwidth (MB/sec): 961.710 
> > > Max bandwidth (MB/sec): 1040 
> > > Min bandwidth (MB/sec): 772 
> > > 
> > > 
> > > Also the bandwidth performance generated with 
> > > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
> > > 
> > > .... is acceptable, e.g. 
> > > fio_write_4m_16 795 MB/s 
> > > fio_randwrite_8m_128 717 MB/s 
> > > fio_randwrite_8m_16 714 MB/s 
> > > fio_randwrite_2m_32 692 MB/s 
> > > 
> > > 
> > > But, the write IOPS seems to be limited around 19k ... 
> > > RBD 4M 64k (= optimal_io_size) 
> > > fio_randread_512_128 53286 55925 
> > > fio_randread_4k_128 51110 44382 
> > > fio_randread_8k_128 30854 29938 
> > > fio_randwrite_512_128 18888 2386 
> > > fio_randwrite_512_64 18844 2582 
> > > fio_randwrite_8k_64 17350 2445 
> > > (...) 
> > > fio_read_4k_128 10073 53151 
> > > fio_read_4k_64 9500 39757 
> > > fio_read_4k_32 9220 23650 
> > > (...) 
> > > fio_read_4k_16 9122 14322 
> > > fio_write_4k_128 2190 14306 
> > > fio_read_8k_32 706 13894 
> > > fio_write_4k_64 2197 12297 
> > > fio_write_8k_64 3563 11705 
> > > fio_write_8k_128 3444 11219 
> > > 
> > > 
> > > Any hints for tuning the IOPS (read and/or write) would be appreciated. 
> > > 
> > > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > > 
> > > 
> > > Kind Regards, 
> > > -Dieter 
> > > 
> > > 
> > > 
> > > -- 
> > > 
> > > -- 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Alexandre D e rumier 
> > > 
> > > Ingénieur Systèmes et Réseaux 
> > > 
> > > 
> > > Fixe : 03 20 68 88 85 
> > > 
> > > Fax : 03 20 68 90 88 
> > > 
> > > 
> > > 45 Bvd du Général Leclerc 59100 Roubaix 
> > > 12 rue Marivaux 75002 Paris 
> > > -- 
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> > > the body of a message to majordomo@vger.kernel.org 
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html 
> > 
> > 
> > 
> > 
> > -- 
> > 
> > -- 
> > 
> > 
> > 
> > 
> > 
> > Alexandre D e rumier 
> > 
> > Ingénieur Systèmes et Réseaux 
> > 
> > 
> > Fixe : 03 20 68 88 85 
> > 
> > Fax : 03 20 68 90 88 
> > 
> > 
> > 45 Bvd du Général Leclerc 59100 Roubaix 
> > 12 rue Marivaux 75002 Paris 
> > 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 	
> 
> Alexandre D e rumier 
> 
> Ingénieur Systèmes et Réseaux 
> 
> 
> Fixe : 03 20 68 88 85 
> 
> Fax : 03 20 68 90 88 
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Alexandre DERUMIER]

>>well, you have to compare
>>- pure a SSD (via PCIe or SAS-6G)        vs.
>>- Ceph-Journal, which goes 2x over 10GbE with IP
>>  Client -> primary-copy -> 2nd-copy
>>  (= redundancy over Ethernet distance)

Sure but the first osd ack to the client,before replicating to the others osd.

Client -> primary-copy -> 2nd-copy
       <-ack
         primary-copy -> 2nd-copy
                      -> 3st-copy

Or I'm wrong ?


----- Mail original ----- 

De: "Dieter Kasper" <d.kasper@kabelmail.de> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org, "Andreas Bluemle" <andreas.bluemle@itxperts.de> 
Envoyé: Jeudi 30 Août 2012 18:02:05 
Objet: Re: RBD performance - tuning hints 

On Thu, Aug 30, 2012 at 05:46:35PM +0200, Alexandre DERUMIER wrote: 
> Thanks 
> 
> >> 8x SSD, 200GB each 
> 
> 20000 iops seem pretty low,no ? 
well, you have to compare 
- pure a SSD (via PCIe or SAS-6G) vs. 
- Ceph-Journal, which goes 2x over 10GbE with IP 
Client -> primary-copy -> 2nd-copy 
(= redundancy over Ethernet distance) 

I'm curious about the answer from Inktank, 

-Dieter 

> 
> 
> for @intank: 
> 
> Is their a bottleneck somewhere in ceph ? 
Maybe "SimpleMessenger dispatching: cause of performance problems?" 
from Thu, 16 Aug 2012 18:08:39 +0200 
by <andreas.bluemle@itxperts.de> 
can be an answer. 
Especially if a small number of OSDs is used. 

> 
> I said that, because I would like to know if it's scale by adding new nodes. 
> 
> Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list) 
> 
> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Jeudi 30 Août 2012 17:33:42 
> Objet: Re: RBD performance - tuning hints 
> 
> On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote: 
> > Thanks for the report ! 
> > 
> > vs your first benchmark, it's with RBD 4M or 64K ? 
> with 4MB (see attached config info) 
> 
> Cheers, 
> -Dieter 
> 
> > 
> > (how much ssd by node?) 
> 8x SSD, 200GB each 
> 
> > 
> > 
> > 
> > ----- Mail original ----- 
> > 
> > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> > Cc: ceph-devel@vger.kernel.org 
> > Envoyé: Jeudi 30 Août 2012 16:56:34 
> > Objet: Re: RBD performance - tuning hints 
> > 
> > Hi Alexandre, 
> > 
> > with the 4 filestore parameter below some fio values could be increased: 
> > filestore max sync interval = 30 
> > filestore min sync interval = 29 
> > filestore flusher = false 
> > filestore queue max ops = 10000 
> > 
> > ###### IOPS 
> > fio_read_4k_64: 9373 
> > fio_read_4k_128: 9939 
> > fio_randwrite_8k_16: 12376 
> > fio_randwrite_4k_16: 13315 
> > fio_randwrite_512_32: 13660 
> > fio_randwrite_8k_32: 17318 
> > fio_randwrite_4k_32: 18057 
> > fio_randwrite_8k_64: 19693 
> > fio_randwrite_512_64: 20015 <<< 
> > fio_randwrite_4k_64: 20024 <<< 
> > fio_randwrite_8k_128: 20547 <<< 
> > fio_randwrite_4k_128: 20839 <<< 
> > fio_randwrite_512_128: 21417 <<< 
> > fio_randread_8k_128: 48872 
> > fio_randread_4k_128: 50002 
> > fio_randread_512_128: 51202 
> > 
> > ###### MB/s 
> > fio_randread_2m_32: 628 
> > fio_read_4m_64: 630 
> > fio_randread_8m_32: 633 
> > fio_read_2m_32: 637 
> > fio_read_4m_16: 640 
> > fio_randread_4m_16: 652 
> > fio_write_2m_32: 660 
> > fio_randread_4m_32: 677 
> > fio_read_4m_32: 678 
> > (...) 
> > fio_write_4m_64: 771 
> > fio_randwrite_2m_64: 789 
> > fio_write_8m_128: 796 
> > fio_write_4m_32: 802 
> > fio_randwrite_4m_128: 807 <<< 
> > fio_randwrite_2m_32: 811 <<< 
> > fio_write_2m_128: 833 <<< 
> > fio_write_8m_64: 901 <<< 
> > 
> > Best Regards, 
> > -Dieter 
> > 
> > 
> > On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote: 
> > > Nice results ! 
> > > (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
> > > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster) 
> > > 
> > > >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > > I think you can try to tune these values 
> > > 
> > > filestore max sync interval = 30 
> > > filestore min sync interval = 29 
> > > filestore flusher = false 
> > > filestore queue max ops = 10000 
> > > 
> > > 
> > > 
> > > ----- Mail original ----- 
> > > 
> > > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > > À: ceph-devel@vger.kernel.org 
> > > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
> > > Envoyé: Mardi 28 Août 2012 19:48:42 
> > > Objet: RBD performance - tuning hints 
> > > 
> > > Hi, 
> > > 
> > > on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
> > > I can observe a pretty nice rados bench performance 
> > > (see bench-rados.txt for details): 
> > > 
> > > Bandwidth (MB/sec): 961.710 
> > > Max bandwidth (MB/sec): 1040 
> > > Min bandwidth (MB/sec): 772 
> > > 
> > > 
> > > Also the bandwidth performance generated with 
> > > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
> > > 
> > > .... is acceptable, e.g. 
> > > fio_write_4m_16 795 MB/s 
> > > fio_randwrite_8m_128 717 MB/s 
> > > fio_randwrite_8m_16 714 MB/s 
> > > fio_randwrite_2m_32 692 MB/s 
> > > 
> > > 
> > > But, the write IOPS seems to be limited around 19k ... 
> > > RBD 4M 64k (= optimal_io_size) 
> > > fio_randread_512_128 53286 55925 
> > > fio_randread_4k_128 51110 44382 
> > > fio_randread_8k_128 30854 29938 
> > > fio_randwrite_512_128 18888 2386 
> > > fio_randwrite_512_64 18844 2582 
> > > fio_randwrite_8k_64 17350 2445 
> > > (...) 
> > > fio_read_4k_128 10073 53151 
> > > fio_read_4k_64 9500 39757 
> > > fio_read_4k_32 9220 23650 
> > > (...) 
> > > fio_read_4k_16 9122 14322 
> > > fio_write_4k_128 2190 14306 
> > > fio_read_8k_32 706 13894 
> > > fio_write_4k_64 2197 12297 
> > > fio_write_8k_64 3563 11705 
> > > fio_write_8k_128 3444 11219 
> > > 
> > > 
> > > Any hints for tuning the IOPS (read and/or write) would be appreciated. 
> > > 
> > > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > > 
> > > 
> > > Kind Regards, 
> > > -Dieter 
> > > 
> > > 
> > > 
> > > -- 
> > > 
> > > -- 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Alexandre D e rumier 
> > > 
> > > Ingénieur Systèmes et Réseaux 
> > > 
> > > 
> > > Fixe : 03 20 68 88 85 
> > > 
> > > Fax : 03 20 68 90 88 
> > > 
> > > 
> > > 45 Bvd du Général Leclerc 59100 Roubaix 
> > > 12 rue Marivaux 75002 Paris 
> > > -- 
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> > > the body of a message to majordomo@vger.kernel.org 
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html 
> > 
> > 
> > 
> > 
> > -- 
> > 
> > -- 
> > 
> > 
> > 
> > 
> > 
> > Alexandre D e rumier 
> > 
> > Ingénieur Systèmes et Réseaux 
> > 
> > 
> > Fixe : 03 20 68 88 85 
> > 
> > Fax : 03 20 68 90 88 
> > 
> > 
> > 45 Bvd du Général Leclerc 59100 Roubaix 
> > 12 rue Marivaux 75002 Paris 
> > 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> 
> Alexandre D e rumier 
> 
> Ingénieur Systèmes et Réseaux 
> 
> 
> Fixe : 03 20 68 88 85 
> 
> Fax : 03 20 68 90 88 
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 




-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Josh Durgin]

On 08/30/2012 09:12 AM, Alexandre DERUMIER wrote:
>>> well, you have to compare
>>> - pure a SSD (via PCIe or SAS-6G)        vs.
>>> - Ceph-Journal, which goes 2x over 10GbE with IP
>>>   Client -> primary-copy -> 2nd-copy
>>>   (= redundancy over Ethernet distance)
>
> Sure but the first osd ack to the client,before replicating to the others osd.
>
> Client -> primary-copy -> 2nd-copy
>         <-ack
>           primary-copy -> 2nd-copy
>                        -> 3st-copy
>
> Or I'm wrong ?

RBD waits for the data to be on disk on all replicas. It's pretty easy
to relax this to in memory on all replicas, but there's no option for
that right now.

Josh

>
> ----- Mail original -----
>
> De: "Dieter Kasper" <d.kasper@kabelmail.de>
> À: "Alexandre DERUMIER" <aderumier@odiso.com>
> Cc: ceph-devel@vger.kernel.org, "Andreas Bluemle" <andreas.bluemle@itxperts.de>
> Envoyé: Jeudi 30 Août 2012 18:02:05
> Objet: Re: RBD performance - tuning hints
>
> On Thu, Aug 30, 2012 at 05:46:35PM +0200, Alexandre DERUMIER wrote:
>> Thanks
>>
>>>> 8x SSD, 200GB each
>>
>> 20000 iops seem pretty low,no ?
> well, you have to compare
> - pure a SSD (via PCIe or SAS-6G) vs.
> - Ceph-Journal, which goes 2x over 10GbE with IP
> Client -> primary-copy -> 2nd-copy
> (= redundancy over Ethernet distance)
>
> I'm curious about the answer from Inktank,
>
> -Dieter
>
>>
>>
>> for @intank:
>>
>> Is their a bottleneck somewhere in ceph ?
> Maybe "SimpleMessenger dispatching: cause of performance problems?"
> from Thu, 16 Aug 2012 18:08:39 +0200
> by <andreas.bluemle@itxperts.de>
> can be an answer.
> Especially if a small number of OSDs is used.
>
>>
>> I said that, because I would like to know if it's scale by adding new nodes.
>>
>> Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list)
>>
>>
>> ----- Mail original -----
>>
>> De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>> Cc: ceph-devel@vger.kernel.org
>> Envoyé: Jeudi 30 Août 2012 17:33:42
>> Objet: Re: RBD performance - tuning hints
>>
>> On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote:
>>> Thanks for the report !
>>>
>>> vs your first benchmark, it's with RBD 4M or 64K ?
>> with 4MB (see attached config info)
>>
>> Cheers,
>> -Dieter
>>
>>>
>>> (how much ssd by node?)
>> 8x SSD, 200GB each
>>
>>>
>>>
>>>
>>> ----- Mail original -----
>>>
>>> De: "Dieter Kasper" <d.kasper@kabelmail.de>
>>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>>> Cc: ceph-devel@vger.kernel.org
>>> Envoyé: Jeudi 30 Août 2012 16:56:34
>>> Objet: Re: RBD performance - tuning hints
>>>
>>> Hi Alexandre,
>>>
>>> with the 4 filestore parameter below some fio values could be increased:
>>> filestore max sync interval = 30
>>> filestore min sync interval = 29
>>> filestore flusher = false
>>> filestore queue max ops = 10000
>>>
>>> ###### IOPS
>>> fio_read_4k_64: 9373
>>> fio_read_4k_128: 9939
>>> fio_randwrite_8k_16: 12376
>>> fio_randwrite_4k_16: 13315
>>> fio_randwrite_512_32: 13660
>>> fio_randwrite_8k_32: 17318
>>> fio_randwrite_4k_32: 18057
>>> fio_randwrite_8k_64: 19693
>>> fio_randwrite_512_64: 20015 <<<
>>> fio_randwrite_4k_64: 20024 <<<
>>> fio_randwrite_8k_128: 20547 <<<
>>> fio_randwrite_4k_128: 20839 <<<
>>> fio_randwrite_512_128: 21417 <<<
>>> fio_randread_8k_128: 48872
>>> fio_randread_4k_128: 50002
>>> fio_randread_512_128: 51202
>>>
>>> ###### MB/s
>>> fio_randread_2m_32: 628
>>> fio_read_4m_64: 630
>>> fio_randread_8m_32: 633
>>> fio_read_2m_32: 637
>>> fio_read_4m_16: 640
>>> fio_randread_4m_16: 652
>>> fio_write_2m_32: 660
>>> fio_randread_4m_32: 677
>>> fio_read_4m_32: 678
>>> (...)
>>> fio_write_4m_64: 771
>>> fio_randwrite_2m_64: 789
>>> fio_write_8m_128: 796
>>> fio_write_4m_32: 802
>>> fio_randwrite_4m_128: 807 <<<
>>> fio_randwrite_2m_32: 811 <<<
>>> fio_write_2m_128: 833 <<<
>>> fio_write_8m_64: 901 <<<
>>>
>>> Best Regards,
>>> -Dieter
>>>
>>>
>>> On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote:
>>>> Nice results !
>>>> (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
>>>> I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
>>>>
>>>>>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>>>> I think you can try to tune these values
>>>>
>>>> filestore max sync interval = 30
>>>> filestore min sync interval = 29
>>>> filestore flusher = false
>>>> filestore queue max ops = 10000
>>>>
>>>>
>>>>
>>>> ----- Mail original -----
>>>>
>>>> De: "Dieter Kasper" <d.kasper@kabelmail.de>
>>>> À: ceph-devel@vger.kernel.org
>>>> Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
>>>> Envoyé: Mardi 28 Août 2012 19:48:42
>>>> Objet: RBD performance - tuning hints
>>>>
>>>> Hi,
>>>>
>>>> on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
>>>> I can observe a pretty nice rados bench performance
>>>> (see bench-rados.txt for details):
>>>>
>>>> Bandwidth (MB/sec): 961.710
>>>> Max bandwidth (MB/sec): 1040
>>>> Min bandwidth (MB/sec): 772
>>>>
>>>>
>>>> Also the bandwidth performance generated with
>>>> fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
>>>>
>>>> .... is acceptable, e.g.
>>>> fio_write_4m_16 795 MB/s
>>>> fio_randwrite_8m_128 717 MB/s
>>>> fio_randwrite_8m_16 714 MB/s
>>>> fio_randwrite_2m_32 692 MB/s
>>>>
>>>>
>>>> But, the write IOPS seems to be limited around 19k ...
>>>> RBD 4M 64k (= optimal_io_size)
>>>> fio_randread_512_128 53286 55925
>>>> fio_randread_4k_128 51110 44382
>>>> fio_randread_8k_128 30854 29938
>>>> fio_randwrite_512_128 18888 2386
>>>> fio_randwrite_512_64 18844 2582
>>>> fio_randwrite_8k_64 17350 2445
>>>> (...)
>>>> fio_read_4k_128 10073 53151
>>>> fio_read_4k_64 9500 39757
>>>> fio_read_4k_32 9220 23650
>>>> (...)
>>>> fio_read_4k_16 9122 14322
>>>> fio_write_4k_128 2190 14306
>>>> fio_read_8k_32 706 13894
>>>> fio_write_4k_64 2197 12297
>>>> fio_write_8k_64 3563 11705
>>>> fio_write_8k_128 3444 11219
>>>>
>>>>
>>>> Any hints for tuning the IOPS (read and/or write) would be appreciated.
>>>>
>>>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>>>>
>>>>
>>>> Kind Regards,
>>>> -Dieter
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Alexandre D e rumier
>>>>
>>>> Ingénieur Systèmes et Réseaux
>>>>
>>>>
>>>> Fixe : 03 20 68 88 85
>>>>
>>>> Fax : 03 20 68 90 88
>>>>
>>>>
>>>> 45 Bvd du Général Leclerc 59100 Roubaix
>>>> 12 rue Marivaux 75002 Paris
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>>
>>>
>>>
>>>
>>>
>>> Alexandre D e rumier
>>>
>>> Ingénieur Systèmes et Réseaux
>>>
>>>
>>> Fixe : 03 20 68 88 85
>>>
>>> Fax : 03 20 68 90 88
>>>
>>>
>>> 45 Bvd du Général Leclerc 59100 Roubaix
>>> 12 rue Marivaux 75002 Paris
>>>
>>
>>
>>
>> --
>>
>> --
>>
>>
>>
>>
>>
>> Alexandre D e rumier
>>
>> Ingénieur Systèmes et Réseaux
>>
>>
>> Fixe : 03 20 68 88 85
>>
>> Fax : 03 20 68 90 88
>>
>>
>> 45 Bvd du Général Leclerc 59100 Roubaix
>> 12 rue Marivaux 75002 Paris
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Dieter Kasper]

[-- Attachment #1: Type: text/plain, Size: 10043 bytes --]

On Thu, Aug 30, 2012 at 06:12:11PM +0200, Alexandre DERUMIER wrote:
> >>well, you have to compare
> >>- pure a SSD (via PCIe or SAS-6G)        vs.
> >>- Ceph-Journal, which goes 2x over 10GbE with IP
> >>  Client -> primary-copy -> 2nd-copy
> >>  (= redundancy over Ethernet distance)
> 
> Sure but the first osd ack to the client,before replicating to the others osd.
no 

> 
> Client -> primary-copy -> 2nd-copy
>        <-ack
>          primary-copy -> 2nd-copy
>                       -> 3st-copy
> 
> Or I'm wrong ?
yes,
please have a look at the attached file: ceph-replication-acks.png
The client usually will continue on 'ACK' and not wait for the 'commit'.

BTW. all my journals are in RAM (/dev/ramX)
32x 2GB = 32GB of data with replica 2x

If "filestore min/max sync interval" is set to 99999999
data should 'never' be written to OSD
('never' at least during the tests if the written data is < 32GB)

In such a configuration only the Ceph-Code and the Interconnect (10GbE/IP) would be the brakeman.

Cheers,
-Dieter


> 
> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org, "Andreas Bluemle" <andreas.bluemle@itxperts.de> 
> Envoyé: Jeudi 30 Août 2012 18:02:05 
> Objet: Re: RBD performance - tuning hints 
> 
> On Thu, Aug 30, 2012 at 05:46:35PM +0200, Alexandre DERUMIER wrote: 
> > Thanks 
> > 
> > >> 8x SSD, 200GB each 
> > 
> > 20000 iops seem pretty low,no ? 
> well, you have to compare 
> - pure a SSD (via PCIe or SAS-6G) vs. 
> - Ceph-Journal, which goes 2x over 10GbE with IP 
> Client -> primary-copy -> 2nd-copy 
> (= redundancy over Ethernet distance) 
> 
> I'm curious about the answer from Inktank, 
> 
> -Dieter 
> 
> > 
> > 
> > for @intank: 
> > 
> > Is their a bottleneck somewhere in ceph ? 
> Maybe "SimpleMessenger dispatching: cause of performance problems?" 
> from Thu, 16 Aug 2012 18:08:39 +0200 
> by <andreas.bluemle@itxperts.de> 
> can be an answer. 
> Especially if a small number of OSDs is used. 
> 
> > 
> > I said that, because I would like to know if it's scale by adding new nodes. 
> > 
> > Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list) 
> > 
> > 
> > ----- Mail original ----- 
> > 
> > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> > Cc: ceph-devel@vger.kernel.org 
> > Envoyé: Jeudi 30 Août 2012 17:33:42 
> > Objet: Re: RBD performance - tuning hints 
> > 
> > On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote: 
> > > Thanks for the report ! 
> > > 
> > > vs your first benchmark, it's with RBD 4M or 64K ? 
> > with 4MB (see attached config info) 
> > 
> > Cheers, 
> > -Dieter 
> > 
> > > 
> > > (how much ssd by node?) 
> > 8x SSD, 200GB each 
> > 
> > > 
> > > 
> > > 
> > > ----- Mail original ----- 
> > > 
> > > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > > À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> > > Cc: ceph-devel@vger.kernel.org 
> > > Envoyé: Jeudi 30 Août 2012 16:56:34 
> > > Objet: Re: RBD performance - tuning hints 
> > > 
> > > Hi Alexandre, 
> > > 
> > > with the 4 filestore parameter below some fio values could be increased: 
> > > filestore max sync interval = 30 
> > > filestore min sync interval = 29 
> > > filestore flusher = false 
> > > filestore queue max ops = 10000 
> > > 
> > > ###### IOPS 
> > > fio_read_4k_64: 9373 
> > > fio_read_4k_128: 9939 
> > > fio_randwrite_8k_16: 12376 
> > > fio_randwrite_4k_16: 13315 
> > > fio_randwrite_512_32: 13660 
> > > fio_randwrite_8k_32: 17318 
> > > fio_randwrite_4k_32: 18057 
> > > fio_randwrite_8k_64: 19693 
> > > fio_randwrite_512_64: 20015 <<< 
> > > fio_randwrite_4k_64: 20024 <<< 
> > > fio_randwrite_8k_128: 20547 <<< 
> > > fio_randwrite_4k_128: 20839 <<< 
> > > fio_randwrite_512_128: 21417 <<< 
> > > fio_randread_8k_128: 48872 
> > > fio_randread_4k_128: 50002 
> > > fio_randread_512_128: 51202 
> > > 
> > > ###### MB/s 
> > > fio_randread_2m_32: 628 
> > > fio_read_4m_64: 630 
> > > fio_randread_8m_32: 633 
> > > fio_read_2m_32: 637 
> > > fio_read_4m_16: 640 
> > > fio_randread_4m_16: 652 
> > > fio_write_2m_32: 660 
> > > fio_randread_4m_32: 677 
> > > fio_read_4m_32: 678 
> > > (...) 
> > > fio_write_4m_64: 771 
> > > fio_randwrite_2m_64: 789 
> > > fio_write_8m_128: 796 
> > > fio_write_4m_32: 802 
> > > fio_randwrite_4m_128: 807 <<< 
> > > fio_randwrite_2m_32: 811 <<< 
> > > fio_write_2m_128: 833 <<< 
> > > fio_write_8m_64: 901 <<< 
> > > 
> > > Best Regards, 
> > > -Dieter 
> > > 
> > > 
> > > On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote: 
> > > > Nice results ! 
> > > > (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
> > > > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster) 
> > > > 
> > > > >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > > > I think you can try to tune these values 
> > > > 
> > > > filestore max sync interval = 30 
> > > > filestore min sync interval = 29 
> > > > filestore flusher = false 
> > > > filestore queue max ops = 10000 
> > > > 
> > > > 
> > > > 
> > > > ----- Mail original ----- 
> > > > 
> > > > De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> > > > À: ceph-devel@vger.kernel.org 
> > > > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
> > > > Envoyé: Mardi 28 Août 2012 19:48:42 
> > > > Objet: RBD performance - tuning hints 
> > > > 
> > > > Hi, 
> > > > 
> > > > on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
> > > > I can observe a pretty nice rados bench performance 
> > > > (see bench-rados.txt for details): 
> > > > 
> > > > Bandwidth (MB/sec): 961.710 
> > > > Max bandwidth (MB/sec): 1040 
> > > > Min bandwidth (MB/sec): 772 
> > > > 
> > > > 
> > > > Also the bandwidth performance generated with 
> > > > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
> > > > 
> > > > .... is acceptable, e.g. 
> > > > fio_write_4m_16 795 MB/s 
> > > > fio_randwrite_8m_128 717 MB/s 
> > > > fio_randwrite_8m_16 714 MB/s 
> > > > fio_randwrite_2m_32 692 MB/s 
> > > > 
> > > > 
> > > > But, the write IOPS seems to be limited around 19k ... 
> > > > RBD 4M 64k (= optimal_io_size) 
> > > > fio_randread_512_128 53286 55925 
> > > > fio_randread_4k_128 51110 44382 
> > > > fio_randread_8k_128 30854 29938 
> > > > fio_randwrite_512_128 18888 2386 
> > > > fio_randwrite_512_64 18844 2582 
> > > > fio_randwrite_8k_64 17350 2445 
> > > > (...) 
> > > > fio_read_4k_128 10073 53151 
> > > > fio_read_4k_64 9500 39757 
> > > > fio_read_4k_32 9220 23650 
> > > > (...) 
> > > > fio_read_4k_16 9122 14322 
> > > > fio_write_4k_128 2190 14306 
> > > > fio_read_8k_32 706 13894 
> > > > fio_write_4k_64 2197 12297 
> > > > fio_write_8k_64 3563 11705 
> > > > fio_write_8k_128 3444 11219 
> > > > 
> > > > 
> > > > Any hints for tuning the IOPS (read and/or write) would be appreciated. 
> > > > 
> > > > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
> > > > 
> > > > 
> > > > Kind Regards, 
> > > > -Dieter 
> > > > 
> > > > 
> > > > 
> > > > -- 
> > > > 
> > > > -- 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Alexandre D e rumier 
> > > > 
> > > > Ingénieur Systèmes et Réseaux 
> > > > 
> > > > 
> > > > Fixe : 03 20 68 88 85 
> > > > 
> > > > Fax : 03 20 68 90 88 
> > > > 
> > > > 
> > > > 45 Bvd du Général Leclerc 59100 Roubaix 
> > > > 12 rue Marivaux 75002 Paris 
> > > > -- 
> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> > > > the body of a message to majordomo@vger.kernel.org 
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html 
> > > 
> > > 
> > > 
> > > 
> > > -- 
> > > 
> > > -- 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Alexandre D e rumier 
> > > 
> > > Ingénieur Systèmes et Réseaux 
> > > 
> > > 
> > > Fixe : 03 20 68 88 85 
> > > 
> > > Fax : 03 20 68 90 88 
> > > 
> > > 
> > > 45 Bvd du Général Leclerc 59100 Roubaix 
> > > 12 rue Marivaux 75002 Paris 
> > > 
> > 
> > 
> > 
> > -- 
> > 
> > -- 
> > 
> > 
> > 
> > 
> > 
> > Alexandre D e rumier 
> > 
> > Ingénieur Systèmes et Réseaux 
> > 
> > 
> > Fixe : 03 20 68 88 85 
> > 
> > Fax : 03 20 68 90 88 
> > 
> > 
> > 45 Bvd du Général Leclerc 59100 Roubaix 
> > 12 rue Marivaux 75002 Paris 
> > -- 
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> > the body of a message to majordomo@vger.kernel.org 
> > More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 	
> 
> Alexandre D e rumier 
> 
> Ingénieur Systèmes et Réseaux 
> 
> 
> Fixe : 03 20 68 88 85 
> 
> Fax : 03 20 68 90 88 
> 
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Principal Consultant, Data Center Storage Architecture and Technology
FTS CTO
FUJITSU TECHNOLOGY SOLUTIONS GMBH
Mies-van-der-Rohe-Straße 8 / 4F
80807 München
Germany

Telephone:      +49 89 62060     1898
Telefax:	+49 89 62060 329 1898
Mobile: 	+49 170 8563173
Email:  	dieter.kasper@ts.fujitsu.com
Internet:       http://ts.fujitsu.com
Company Details: http://ts.fujitsu.com/imprint.html

[-- Attachment #2: ceph-replication-acks.png --]
[-- Type: image/png, Size: 18144 bytes --]

Re: RBD performance - tuning hints

[Gregory Farnum]

On Thu, Aug 30, 2012 at 9:48 AM, Dieter Kasper <d.kasper@kabelmail.de> wrote:
> On Thu, Aug 30, 2012 at 06:12:11PM +0200, Alexandre DERUMIER wrote:
>> >>well, you have to compare
>> >>- pure a SSD (via PCIe or SAS-6G)        vs.
>> >>- Ceph-Journal, which goes 2x over 10GbE with IP
>> >>  Client -> primary-copy -> 2nd-copy
>> >>  (= redundancy over Ethernet distance)
>>
>> Sure but the first osd ack to the client,before replicating to the others osd.
> no
>
>>
>> Client -> primary-copy -> 2nd-copy
>>        <-ack
>>          primary-copy -> 2nd-copy
>>                       -> 3st-copy
>>
>> Or I'm wrong ?
> yes,
> please have a look at the attached file: ceph-replication-acks.png
> The client usually will continue on 'ACK' and not wait for the 'commit'.
>
> BTW. all my journals are in RAM (/dev/ramX)
> 32x 2GB = 32GB of data with replica 2x
>
> If "filestore min/max sync interval" is set to 99999999
> data should 'never' be written to OSD
> ('never' at least during the tests if the written data is < 32GB)

I believe it actually will start syncing to disk when the journal is
half full (right, Sam?) — and even if it doesn't sync, there's a
reasonable chance that some of the data will be written out to disk in
the background (though that shouldn't slow anything down, of course).
:)
-Greg


>
> In such a configuration only the Ceph-Code and the Interconnect (10GbE/IP) would be the brakeman.
>
> Cheers,
> -Dieter
>
>
>>
>>
>> ----- Mail original -----
>>
>> De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>> Cc: ceph-devel@vger.kernel.org, "Andreas Bluemle" <andreas.bluemle@itxperts.de>
>> Envoyé: Jeudi 30 Août 2012 18:02:05
>> Objet: Re: RBD performance - tuning hints
>>
>> On Thu, Aug 30, 2012 at 05:46:35PM +0200, Alexandre DERUMIER wrote:
>> > Thanks
>> >
>> > >> 8x SSD, 200GB each
>> >
>> > 20000 iops seem pretty low,no ?
>> well, you have to compare
>> - pure a SSD (via PCIe or SAS-6G) vs.
>> - Ceph-Journal, which goes 2x over 10GbE with IP
>> Client -> primary-copy -> 2nd-copy
>> (= redundancy over Ethernet distance)
>>
>> I'm curious about the answer from Inktank,
>>
>> -Dieter
>>
>> >
>> >
>> > for @intank:
>> >
>> > Is their a bottleneck somewhere in ceph ?
>> Maybe "SimpleMessenger dispatching: cause of performance problems?"
>> from Thu, 16 Aug 2012 18:08:39 +0200
>> by <andreas.bluemle@itxperts.de>
>> can be an answer.
>> Especially if a small number of OSDs is used.
>>
>> >
>> > I said that, because I would like to know if it's scale by adding new nodes.
>> >
>> > Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list)
>> >
>> >
>> > ----- Mail original -----
>> >
>> > De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> > À: "Alexandre DERUMIER" <aderumier@odiso.com>
>> > Cc: ceph-devel@vger.kernel.org
>> > Envoyé: Jeudi 30 Août 2012 17:33:42
>> > Objet: Re: RBD performance - tuning hints
>> >
>> > On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote:
>> > > Thanks for the report !
>> > >
>> > > vs your first benchmark, it's with RBD 4M or 64K ?
>> > with 4MB (see attached config info)
>> >
>> > Cheers,
>> > -Dieter
>> >
>> > >
>> > > (how much ssd by node?)
>> > 8x SSD, 200GB each
>> >
>> > >
>> > >
>> > >
>> > > ----- Mail original -----
>> > >
>> > > De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> > > À: "Alexandre DERUMIER" <aderumier@odiso.com>
>> > > Cc: ceph-devel@vger.kernel.org
>> > > Envoyé: Jeudi 30 Août 2012 16:56:34
>> > > Objet: Re: RBD performance - tuning hints
>> > >
>> > > Hi Alexandre,
>> > >
>> > > with the 4 filestore parameter below some fio values could be increased:
>> > > filestore max sync interval = 30
>> > > filestore min sync interval = 29
>> > > filestore flusher = false
>> > > filestore queue max ops = 10000
>> > >
>> > > ###### IOPS
>> > > fio_read_4k_64: 9373
>> > > fio_read_4k_128: 9939
>> > > fio_randwrite_8k_16: 12376
>> > > fio_randwrite_4k_16: 13315
>> > > fio_randwrite_512_32: 13660
>> > > fio_randwrite_8k_32: 17318
>> > > fio_randwrite_4k_32: 18057
>> > > fio_randwrite_8k_64: 19693
>> > > fio_randwrite_512_64: 20015 <<<
>> > > fio_randwrite_4k_64: 20024 <<<
>> > > fio_randwrite_8k_128: 20547 <<<
>> > > fio_randwrite_4k_128: 20839 <<<
>> > > fio_randwrite_512_128: 21417 <<<
>> > > fio_randread_8k_128: 48872
>> > > fio_randread_4k_128: 50002
>> > > fio_randread_512_128: 51202
>> > >
>> > > ###### MB/s
>> > > fio_randread_2m_32: 628
>> > > fio_read_4m_64: 630
>> > > fio_randread_8m_32: 633
>> > > fio_read_2m_32: 637
>> > > fio_read_4m_16: 640
>> > > fio_randread_4m_16: 652
>> > > fio_write_2m_32: 660
>> > > fio_randread_4m_32: 677
>> > > fio_read_4m_32: 678
>> > > (...)
>> > > fio_write_4m_64: 771
>> > > fio_randwrite_2m_64: 789
>> > > fio_write_8m_128: 796
>> > > fio_write_4m_32: 802
>> > > fio_randwrite_4m_128: 807 <<<
>> > > fio_randwrite_2m_32: 811 <<<
>> > > fio_write_2m_128: 833 <<<
>> > > fio_write_8m_64: 901 <<<
>> > >
>> > > Best Regards,
>> > > -Dieter
>> > >
>> > >
>> > > On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote:
>> > > > Nice results !
>> > > > (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
>> > > > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
>> > > >
>> > > > >>How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>> > > > I think you can try to tune these values
>> > > >
>> > > > filestore max sync interval = 30
>> > > > filestore min sync interval = 29
>> > > > filestore flusher = false
>> > > > filestore queue max ops = 10000
>> > > >
>> > > >
>> > > >
>> > > > ----- Mail original -----
>> > > >
>> > > > De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> > > > À: ceph-devel@vger.kernel.org
>> > > > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
>> > > > Envoyé: Mardi 28 Août 2012 19:48:42
>> > > > Objet: RBD performance - tuning hints
>> > > >
>> > > > Hi,
>> > > >
>> > > > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
>> > > > I can observe a pretty nice rados bench performance
>> > > > (see bench-rados.txt for details):
>> > > >
>> > > > Bandwidth (MB/sec): 961.710
>> > > > Max bandwidth (MB/sec): 1040
>> > > > Min bandwidth (MB/sec): 772
>> > > >
>> > > >
>> > > > Also the bandwidth performance generated with
>> > > > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
>> > > >
>> > > > .... is acceptable, e.g.
>> > > > fio_write_4m_16 795 MB/s
>> > > > fio_randwrite_8m_128 717 MB/s
>> > > > fio_randwrite_8m_16 714 MB/s
>> > > > fio_randwrite_2m_32 692 MB/s
>> > > >
>> > > >
>> > > > But, the write IOPS seems to be limited around 19k ...
>> > > > RBD 4M 64k (= optimal_io_size)
>> > > > fio_randread_512_128 53286 55925
>> > > > fio_randread_4k_128 51110 44382
>> > > > fio_randread_8k_128 30854 29938
>> > > > fio_randwrite_512_128 18888 2386
>> > > > fio_randwrite_512_64 18844 2582
>> > > > fio_randwrite_8k_64 17350 2445
>> > > > (...)
>> > > > fio_read_4k_128 10073 53151
>> > > > fio_read_4k_64 9500 39757
>> > > > fio_read_4k_32 9220 23650
>> > > > (...)
>> > > > fio_read_4k_16 9122 14322
>> > > > fio_write_4k_128 2190 14306
>> > > > fio_read_8k_32 706 13894
>> > > > fio_write_4k_64 2197 12297
>> > > > fio_write_8k_64 3563 11705
>> > > > fio_write_8k_128 3444 11219
>> > > >
>> > > >
>> > > > Any hints for tuning the IOPS (read and/or write) would be appreciated.
>> > > >
>> > > > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>> > > >
>> > > >
>> > > > Kind Regards,
>> > > > -Dieter
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > --
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Alexandre D e rumier
>> > > >
>> > > > Ingénieur Systèmes et Réseaux
>> > > >
>> > > >
>> > > > Fixe : 03 20 68 88 85
>> > > >
>> > > > Fax : 03 20 68 90 88
>> > > >
>> > > >
>> > > > 45 Bvd du Général Leclerc 59100 Roubaix
>> > > > 12 rue Marivaux 75002 Paris
>> > > > --
>> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > > > the body of a message to majordomo@vger.kernel.org
>> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > --
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Alexandre D e rumier
>> > >
>> > > Ingénieur Systèmes et Réseaux
>> > >
>> > >
>> > > Fixe : 03 20 68 88 85
>> > >
>> > > Fax : 03 20 68 90 88
>> > >
>> > >
>> > > 45 Bvd du Général Leclerc 59100 Roubaix
>> > > 12 rue Marivaux 75002 Paris
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > --
>> >
>> >
>> >
>> >
>> >
>> > Alexandre D e rumier
>> >
>> > Ingénieur Systèmes et Réseaux
>> >
>> >
>> > Fixe : 03 20 68 88 85
>> >
>> > Fax : 03 20 68 90 88
>> >
>> >
>> > 45 Bvd du Général Leclerc 59100 Roubaix
>> > 12 rue Marivaux 75002 Paris
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>>
>> --
>>
>> --
>>
>>
>>
>>
>>
>> Alexandre D e rumier
>>
>> Ingénieur Systèmes et Réseaux
>>
>>
>> Fixe : 03 20 68 88 85
>>
>> Fax : 03 20 68 90 88
>>
>>
>> 45 Bvd du Général Leclerc 59100 Roubaix
>> 12 rue Marivaux 75002 Paris
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> Principal Consultant, Data Center Storage Architecture and Technology
> FTS CTO
> FUJITSU TECHNOLOGY SOLUTIONS GMBH
> Mies-van-der-Rohe-Straße 8 / 4F
> 80807 München
> Germany
>
> Telephone:      +49 89 62060     1898
> Telefax:        +49 89 62060 329 1898
> Mobile:         +49 170 8563173
> Email:          dieter.kasper@ts.fujitsu.com
> Internet:       http://ts.fujitsu.com
> Company Details: http://ts.fujitsu.com/imprint.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints / parameter doc

[Samuel Just]

Ah, those are just min and max.  Sync is also triggered when the
journal hits the half-full mark.  We could make the percentage
configurable in the future.
-Sam

On Thu, Aug 30, 2012 at 8:08 AM, Dieter Kasper <d.kasper@kabelmail.de> wrote:
> Samuel,
>
> thank you very much for this explicitely description!
>
> As far as I understand the journal acts as a ringbuffer in front of the OSD.
> Using time as a parameter to trigger sync might not be the best for
> a dynamic Storage subsystem. On a high workload e.g. 10/20 for min/max
> might be optimal for for 4 nodes with 10 OSDs each,
> but not after adding 4 additional nodes.
>
> Are there parameters to trigger the syncs to OSD
> in relation to the fill grade of the journal ?
> e.g.
> filestore [min|max] sync percent:
>
> Do not sync before min-% full; sync after max-% full
>
> What would happen if I set "filestore [min|max] sync interval" to 999999 ?
> Will the journal sync start at 100% full or at X% ?
> What is 'X' by defaut ?
> How can I set 'X' ?
>
> Best Regards,
> -Dieter
>
>
> On Thu, Aug 30, 2012 at 12:34:43AM +0200, Samuel Just wrote:
>> filestore [min|max] sync interval:
>>
>> Periodically, the filestore needs to quiesce writes and do a syncfs in
>> order to create
>> a consistent commit point up to which it can free journal entries.  Syncing more
>> frequently tends to reduce the time required to do the sync, and
>> reduces the amount
>> of data that needs to remain in the journal.  Less frequent syncs
>> would allow the
>> backing filesystem to better coalesce small writes and metadata
>> updates hopefully
>> resulting in more efficient syncs.  'filestore max sync interval'
>> defines the maximum
>> time period between syncs, 'filestore min sync interval' defines the
>> minimum time
>> period between syncs.
>>
>> filestore flusher:
>>
>> The filestore flusher forces data from large writes to be written out
>> using sync_file_range
>> before the sync in order to (hopefully) reduce the cost of the
>> eventual sync.  In practice,
>> disabling 'filestore flusher' seems to improve performance in some cases.
>>
>> filestore queue max ops:
>>
>> 'filestore queue max ops' defines the number of in progress ops the
>> filestore will accept
>> before blocking on queueing new ones.  This mostly shouldn't have much
>> of an effect
>> on performance and should probably be ignored.
>>
>> filestore op threads:
>>
>> 'filestore op threads' defines the number of threads used to submit
>> filesystem operations
>> in parallel.
>>
>> journal dio:
>>
>> 'journal dio' enables using O_DIRECT for writing to the journal.  This
>> should usually
>> be enabled.  If possible, 'journal aio' should also be enabled to
>> allow use of libaio
>> to do asynchronous writes.
>>
>> osd op threads:
>>
>> 'osd op threads' defines the size of the thread pool used to service
>> OSD operations
>> such as client requests.  Increasing this may increase the rate of
>> request processing.
>>
>> osd disk threads:
>>
>> 'osd disk threads' defines the number of threads used to perform background disk
>> intensive osd operations such as scrubbing and snap trimming.
>>
>> On Wed, Aug 29, 2012 at 12:29 PM, Dieter Kasper <d.kasper@kabelmail.de> wrote:
>> > Hi Josh,
>> >
>> > thanks for the hint.
>> > Can you please spend a view words about the meaing of these parameters ?
>> > - filestore min/max sync interval =     int/float ?     seconds ? of what ?
>> > - filestore flusher = false
>> > - filestore queue max ops = 10000
>> >         what is 'one op' ?      queue in front of what ?
>> > - filestore op threads =
>> >         what are useful values here ?
>> >
>> > - journal dio = true/false
>> > - osd op threads =
>> > - osd disk threads =
>> >
>> >
>> > Kind Regards,
>> > -Dieter
>> >
>> >
>> > On Wed, Aug 29, 2012 at 07:37:36PM +0200, Josh Durgin wrote:
>> >> On 08/29/2012 01:50 AM, Alexandre DERUMIER wrote:
>> >> > Nice results !
>> >> > (can you make same benchmark from a qemu-kvm guest with virtio-driver ?
>> >> > I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster)
>> >> >
>> >> >>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>> >> > I think you can try to tune these values
>> >> >
>> >> > filestore max sync interval = 30
>> >> > filestore min sync interval = 29
>> >> > filestore flusher = false
>> >> > filestore queue max ops = 10000
>> >>
>> >> Increasing filestore_op_threads might help as well.
>> >>
>> >> > ----- Mail original -----
>> >> >
>> >> > De: "Dieter Kasper" <d.kasper@kabelmail.de>
>> >> > À: ceph-devel@vger.kernel.org
>> >> > Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de>
>> >> > Envoyé: Mardi 28 Août 2012 19:48:42
>> >> > Objet: RBD performance - tuning hints
>> >> >
>> >> > Hi,
>> >> >
>> >> > on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
>> >> > I can observe a pretty nice rados bench performance
>> >> > (see bench-rados.txt for details):
>> >> >
>> >> > Bandwidth (MB/sec): 961.710
>> >> > Max bandwidth (MB/sec): 1040
>> >> > Min bandwidth (MB/sec): 772
>> >> >
>> >> >
>> >> > Also the bandwidth performance generated with
>> >> > fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}
>> >> >
>> >> > .... is acceptable, e.g.
>> >> > fio_write_4m_16 795 MB/s
>> >> > fio_randwrite_8m_128 717 MB/s
>> >> > fio_randwrite_8m_16 714 MB/s
>> >> > fio_randwrite_2m_32 692 MB/s
>> >> >
>> >> >
>> >> > But, the write IOPS seems to be limited around 19k ...
>> >> > RBD 4M 64k (= optimal_io_size)
>> >> > fio_randread_512_128 53286 55925
>> >> > fio_randread_4k_128 51110 44382
>> >> > fio_randread_8k_128 30854 29938
>> >> > fio_randwrite_512_128 18888 2386
>> >> > fio_randwrite_512_64 18844 2582
>> >> > fio_randwrite_8k_64 17350 2445
>> >> > (...)
>> >> > fio_read_4k_128 10073 53151
>> >> > fio_read_4k_64 9500 39757
>> >> > fio_read_4k_32 9220 23650
>> >> > (...)
>> >> > fio_read_4k_16 9122 14322
>> >> > fio_write_4k_128 2190 14306
>> >> > fio_read_8k_32 706 13894
>> >> > fio_write_4k_64 2197 12297
>> >> > fio_write_8k_64 3563 11705
>> >> > fio_write_8k_128 3444 11219
>> >> >
>> >> >
>> >> > Any hints for tuning the IOPS (read and/or write) would be appreciated.
>> >> >
>> >> > How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)
>> >> >
>> >> >
>> >> > Kind Regards,
>> >> > -Dieter
>> >> >
>> >> >
>> >> >
>> >>
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints

[Alexandre DERUMIER]

>>RBD waits for the data to be on disk on all replicas. It's pretty easy
>>to relax this to in memory on all replicas, but there's no option for
>>that right now.

Ok, thanks, I miss that.

When you say disk, you mean journal ?



----- Mail original ----- 

De: "Josh Durgin" <josh.durgin@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: "Dieter Kasper" <d.kasper@kabelmail.de>, ceph-devel@vger.kernel.org, "Andreas Bluemle" <andreas.bluemle@itxperts.de> 
Envoyé: Jeudi 30 Août 2012 18:16:47 
Objet: Re: RBD performance - tuning hints 

On 08/30/2012 09:12 AM, Alexandre DERUMIER wrote: 
>>> well, you have to compare 
>>> - pure a SSD (via PCIe or SAS-6G) vs. 
>>> - Ceph-Journal, which goes 2x over 10GbE with IP 
>>> Client -> primary-copy -> 2nd-copy 
>>> (= redundancy over Ethernet distance) 
> 
> Sure but the first osd ack to the client,before replicating to the others osd. 
> 
> Client -> primary-copy -> 2nd-copy 
> <-ack 
> primary-copy -> 2nd-copy 
> -> 3st-copy 
> 
> Or I'm wrong ? 

RBD waits for the data to be on disk on all replicas. It's pretty easy 
to relax this to in memory on all replicas, but there's no option for 
that right now. 

Josh 

> 
> ----- Mail original ----- 
> 
> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org, "Andreas Bluemle" <andreas.bluemle@itxperts.de> 
> Envoyé: Jeudi 30 Août 2012 18:02:05 
> Objet: Re: RBD performance - tuning hints 
> 
> On Thu, Aug 30, 2012 at 05:46:35PM +0200, Alexandre DERUMIER wrote: 
>> Thanks 
>> 
>>>> 8x SSD, 200GB each 
>> 
>> 20000 iops seem pretty low,no ? 
> well, you have to compare 
> - pure a SSD (via PCIe or SAS-6G) vs. 
> - Ceph-Journal, which goes 2x over 10GbE with IP 
> Client -> primary-copy -> 2nd-copy 
> (= redundancy over Ethernet distance) 
> 
> I'm curious about the answer from Inktank, 
> 
> -Dieter 
> 
>> 
>> 
>> for @intank: 
>> 
>> Is their a bottleneck somewhere in ceph ? 
> Maybe "SimpleMessenger dispatching: cause of performance problems?" 
> from Thu, 16 Aug 2012 18:08:39 +0200 
> by <andreas.bluemle@itxperts.de> 
> can be an answer. 
> Especially if a small number of OSDs is used. 
> 
>> 
>> I said that, because I would like to know if it's scale by adding new nodes. 
>> 
>> Does Intank have already done some random iops benchmark ? (I always see sequential throughput bench in the mailing list) 
>> 
>> 
>> ----- Mail original ----- 
>> 
>> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
>> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
>> Cc: ceph-devel@vger.kernel.org 
>> Envoyé: Jeudi 30 Août 2012 17:33:42 
>> Objet: Re: RBD performance - tuning hints 
>> 
>> On Thu, Aug 30, 2012 at 05:28:02PM +0200, Alexandre DERUMIER wrote: 
>>> Thanks for the report ! 
>>> 
>>> vs your first benchmark, it's with RBD 4M or 64K ? 
>> with 4MB (see attached config info) 
>> 
>> Cheers, 
>> -Dieter 
>> 
>>> 
>>> (how much ssd by node?) 
>> 8x SSD, 200GB each 
>> 
>>> 
>>> 
>>> 
>>> ----- Mail original ----- 
>>> 
>>> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
>>> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
>>> Cc: ceph-devel@vger.kernel.org 
>>> Envoyé: Jeudi 30 Août 2012 16:56:34 
>>> Objet: Re: RBD performance - tuning hints 
>>> 
>>> Hi Alexandre, 
>>> 
>>> with the 4 filestore parameter below some fio values could be increased: 
>>> filestore max sync interval = 30 
>>> filestore min sync interval = 29 
>>> filestore flusher = false 
>>> filestore queue max ops = 10000 
>>> 
>>> ###### IOPS 
>>> fio_read_4k_64: 9373 
>>> fio_read_4k_128: 9939 
>>> fio_randwrite_8k_16: 12376 
>>> fio_randwrite_4k_16: 13315 
>>> fio_randwrite_512_32: 13660 
>>> fio_randwrite_8k_32: 17318 
>>> fio_randwrite_4k_32: 18057 
>>> fio_randwrite_8k_64: 19693 
>>> fio_randwrite_512_64: 20015 <<< 
>>> fio_randwrite_4k_64: 20024 <<< 
>>> fio_randwrite_8k_128: 20547 <<< 
>>> fio_randwrite_4k_128: 20839 <<< 
>>> fio_randwrite_512_128: 21417 <<< 
>>> fio_randread_8k_128: 48872 
>>> fio_randread_4k_128: 50002 
>>> fio_randread_512_128: 51202 
>>> 
>>> ###### MB/s 
>>> fio_randread_2m_32: 628 
>>> fio_read_4m_64: 630 
>>> fio_randread_8m_32: 633 
>>> fio_read_2m_32: 637 
>>> fio_read_4m_16: 640 
>>> fio_randread_4m_16: 652 
>>> fio_write_2m_32: 660 
>>> fio_randread_4m_32: 677 
>>> fio_read_4m_32: 678 
>>> (...) 
>>> fio_write_4m_64: 771 
>>> fio_randwrite_2m_64: 789 
>>> fio_write_8m_128: 796 
>>> fio_write_4m_32: 802 
>>> fio_randwrite_4m_128: 807 <<< 
>>> fio_randwrite_2m_32: 811 <<< 
>>> fio_write_2m_128: 833 <<< 
>>> fio_write_8m_64: 901 <<< 
>>> 
>>> Best Regards, 
>>> -Dieter 
>>> 
>>> 
>>> On Wed, Aug 29, 2012 at 10:50:12AM +0200, Alexandre DERUMIER wrote: 
>>>> Nice results ! 
>>>> (can you make same benchmark from a qemu-kvm guest with virtio-driver ? 
>>>> I have made some bench some month ago with stephan priebe, and we never be able to have more than 20000iops, with a full ssd 3nodes cluster) 
>>>> 
>>>>>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
>>>> I think you can try to tune these values 
>>>> 
>>>> filestore max sync interval = 30 
>>>> filestore min sync interval = 29 
>>>> filestore flusher = false 
>>>> filestore queue max ops = 10000 
>>>> 
>>>> 
>>>> 
>>>> ----- Mail original ----- 
>>>> 
>>>> De: "Dieter Kasper" <d.kasper@kabelmail.de> 
>>>> À: ceph-devel@vger.kernel.org 
>>>> Cc: "Dieter Kasper (KD)" <d.kasper@kabelmail.de> 
>>>> Envoyé: Mardi 28 Août 2012 19:48:42 
>>>> Objet: RBD performance - tuning hints 
>>>> 
>>>> Hi, 
>>>> 
>>>> on my 4-node system (SSD + 10GbE, see bench-config.txt for details) 
>>>> I can observe a pretty nice rados bench performance 
>>>> (see bench-rados.txt for details): 
>>>> 
>>>> Bandwidth (MB/sec): 961.710 
>>>> Max bandwidth (MB/sec): 1040 
>>>> Min bandwidth (MB/sec): 772 
>>>> 
>>>> 
>>>> Also the bandwidth performance generated with 
>>>> fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads} 
>>>> 
>>>> .... is acceptable, e.g. 
>>>> fio_write_4m_16 795 MB/s 
>>>> fio_randwrite_8m_128 717 MB/s 
>>>> fio_randwrite_8m_16 714 MB/s 
>>>> fio_randwrite_2m_32 692 MB/s 
>>>> 
>>>> 
>>>> But, the write IOPS seems to be limited around 19k ... 
>>>> RBD 4M 64k (= optimal_io_size) 
>>>> fio_randread_512_128 53286 55925 
>>>> fio_randread_4k_128 51110 44382 
>>>> fio_randread_8k_128 30854 29938 
>>>> fio_randwrite_512_128 18888 2386 
>>>> fio_randwrite_512_64 18844 2582 
>>>> fio_randwrite_8k_64 17350 2445 
>>>> (...) 
>>>> fio_read_4k_128 10073 53151 
>>>> fio_read_4k_64 9500 39757 
>>>> fio_read_4k_32 9220 23650 
>>>> (...) 
>>>> fio_read_4k_16 9122 14322 
>>>> fio_write_4k_128 2190 14306 
>>>> fio_read_8k_32 706 13894 
>>>> fio_write_4k_64 2197 12297 
>>>> fio_write_8k_64 3563 11705 
>>>> fio_write_8k_128 3444 11219 
>>>> 
>>>> 
>>>> Any hints for tuning the IOPS (read and/or write) would be appreciated. 
>>>> 
>>>> How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full) 
>>>> 
>>>> 
>>>> Kind Regards, 
>>>> -Dieter 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> 
>>>> -- 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Alexandre D e rumier 
>>>> 
>>>> Ingénieur Systèmes et Réseaux 
>>>> 
>>>> 
>>>> Fixe : 03 20 68 88 85 
>>>> 
>>>> Fax : 03 20 68 90 88 
>>>> 
>>>> 
>>>> 45 Bvd du Général Leclerc 59100 Roubaix 
>>>> 12 rue Marivaux 75002 Paris 
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
>>>> the body of a message to majordomo@vger.kernel.org 
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> -- 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Alexandre D e rumier 
>>> 
>>> Ingénieur Systèmes et Réseaux 
>>> 
>>> 
>>> Fixe : 03 20 68 88 85 
>>> 
>>> Fax : 03 20 68 90 88 
>>> 
>>> 
>>> 45 Bvd du Général Leclerc 59100 Roubaix 
>>> 12 rue Marivaux 75002 Paris 
>>> 
>> 
>> 
>> 
>> -- 
>> 
>> -- 
>> 
>> 
>> 
>> 
>> 
>> Alexandre D e rumier 
>> 
>> Ingénieur Systèmes et Réseaux 
>> 
>> 
>> Fixe : 03 20 68 88 85 
>> 
>> Fax : 03 20 68 90 88 
>> 
>> 
>> 45 Bvd du Général Leclerc 59100 Roubaix 
>> 12 rue Marivaux 75002 Paris 
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
>> the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 
> 
> 
> 




-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RBD performance - tuning hints

[Dietmar Maurer]

>>RBD waits for the data to be on disk on all replicas. It's pretty easy
>>to relax this to in memory on all replicas, but there's no option for
>>that right now.

I thought that is dangerous, because you can loose data?

Re: RBD performance - tuning hints

[Mark Kirkwood]

On 31/08/12 20:11, Dietmar Maurer wrote:
>>> RBD waits for the data to be on disk on all replicas. It's pretty easy
>>> to relax this to in memory on all replicas, but there's no option for
>>> that right now.
> I thought that is dangerous, because you can loose data?
> N�����r��y���b�X��ǧv�^�)޺{.n�+���z�]z�{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+��ݢj"��!tml=

And it is not immediately obvious that this is the bottleneck - from 
what I can see the 'sync' call being used (sync_file_range) is extremely 
fast and is *not* the major slowdown effect...

Regards

Mark

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RBD performance - tuning hints / major slowdown effect(s)

[Dieter Kasper]

Mark, Inktank,

OK, it is very likely that 'sync_file_range' is not the major slowdown 'culprit'.

But, which areas (design, current implementation, protocol, interconnect, tuning parameter, ...)
would you rate as 'major slowdown effect(s)' ?

Best Regards,
-Dieter


On Fri, Aug 31, 2012 at 08:48:34PM +1200, Mark Kirkwood wrote:
> On 31/08/12 20:11, Dietmar Maurer wrote:
> >>>RBD waits for the data to be on disk on all replicas. It's pretty easy
> >>>to relax this to in memory on all replicas, but there's no option for
> >>>that right now.
> >I thought that is dangerous, because you can loose data?
> >N???????????????r??????y?????????b???X????????v???^???)??{.n???+?????????z???]z???{ay???\x1d???????,j\a??????f?????????h?????????z???\x1e???w?????????\f?????????j:+v?????????w???j???m????????????\a????????????zZ+????????j"??????!tml=
> 
> And it is not immediately obvious that this is the bottleneck - from
> what I can see the 'sync' call being used (sync_file_range) is
> extremely fast and is *not* the major slowdown effect...
> 
> Regards
> 
> Mark
> 

Re: RBD performance - tuning hints / major slowdown effect(s)

[Mark Kirkwood]

Sorry Dieter,

Not trying to say "you are wrong" or anything like that - just trying to 
add to the problem solving body of knowledge that from what *I* have 
tried out the 'sync' issue does not look to be the bad guy here - altho 
more analysis is always welcome (usual story - my findings should be 
confirm-able by others doing similar tests)!

regards

Mark

On 31/08/12 21:49, Dieter Kasper wrote:
> Mark, Inktank,
>
> OK, it is very likely that 'sync_file_range' is not the major slowdown 'culprit'.
>
> But, which areas (design, current implementation, protocol, interconnect, tuning parameter, ...)
> would you rate as 'major slowdown effect(s)' ?
>

RE: RBD performance - tuning hints

[Jerker Nyberg]

On Fri, 31 Aug 2012, Dietmar Maurer wrote:

>>> RBD waits for the data to be on disk on all replicas. It's pretty easy
>>> to relax this to in memory on all replicas, but there's no option for
>>> that right now.
>
> I thought that is dangerous, because you can loose data?

By putting the journal in a tmpfs then data written to the journal does 
not hit disk. If all replicas fail data will be lost.

For some use cases that might be ok. For example incremental backups or 
fast scratch space or volatile virtual machines etc.

Also see this previous discussion:

http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg06070.html

--jerker