expected I/O / rand 4k iops

expected I/O / rand 4k iops

[Stefan Priebe - Profihost AG]

Hello list,

is there any calculation of expected I/O available?

I've a test system running 6 hosts 4 OSDs each using SSD - i get 20.000
to 40.000 IOP/s not as much as i expected but OK right now.

If i replace the SSDs on one host with spinning disks but still using
dedicated journal on ssd (20GB / Disk/OSDK), i'm not able to get more
than 300 to 400 iop/s this seems to be pretty low.

Everything tested using 0.56.4 and Qemu RBD.

Greets,
Stefan

Re: expected I/O / rand 4k iops

[Mark Nelson]

On 04/11/2013 02:27 AM, Stefan Priebe - Profihost AG wrote:
> Hello list,
>
> is there any calculation of expected I/O available?
>
> I've a test system running 6 hosts 4 OSDs each using SSD - i get 20.000
> to 40.000 IOP/s not as much as i expected but OK right now.

Hi!

How are you running your benchmarks Stefan?

Doing reads from pagecache with RADOS bench I can do up to about 22,000 
IOPs from a single host.

See the page cache section of our bobtail vs argonaut article by 
scrolling down a bit from here:

http://ceph.com/uncategorized/argonaut-vs-bobtail-performance-preview/#4kbradoswrite

I haven't tested putting OSDs directly on RAM disks recently, but it'd 
probably be a good idea to try again at some point.

>
> If i replace the SSDs on one host with spinning disks but still using
> dedicated journal on ssd (20GB / Disk/OSDK), i'm not able to get more
> than 300 to 400 iop/s this seems to be pretty low.

That's probably about right.  The journals really only absorb a small 
portion of the incoming writes for free, and then you end up having to 
wait on the disks behind the OSDs. If you have 4 spinning disks in the 
system, each one is only really capable of around 150-200 IOPs assuming 
typical 7200rpm units.  300-400 iops for 4 disks isn't great, but it's 
probably not totally unrealistic either.

So what would happen with 4 spinning disks in 1 node, but everything 
else on SSD?  All of your outstanding operations will end up backing up 
on the disks in the slow node while everything else sits mostly idle. 
That's because there's a (configurable) maximum number of outstanding 
operations that you can have in flight at once.  No matter how high you 
set that though, eventually *all* outstanding operations will back up on 
the slow node under a sustained workload.  You can mitigate this by 
weighting the slow OSDs to have less data vs the others, but that's not 
really an ideal solution.  Ceph really likes well balanced systems. 
It'll work on heterogeneous clusters but it's definitely not ideal.

As far as SSDs go, some folks seem to be having luck with Bcache and 
fastcache to improve performance of spinning disk backed OSDs.  I admit 
haven't had time to play with them yet but it's definitely on my list!

>
> Everything tested using 0.56.4 and Qemu RBD.

Out of curiosity, do you have RBD cache enabled?  I noticed on my test 
setup that with 64G VM images it provide quite a bit of benefit even for 
small random writes.

>
> Greets,
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: expected I/O / rand 4k iops

[Stefan Priebe - Profihost AG]

Am 11.04.2013 14:27, schrieb Mark Nelson:
> On 04/11/2013 02:27 AM, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> is there any calculation of expected I/O available?
>>
>> I've a test system running 6 hosts 4 OSDs each using SSD - i get 20.000
>> to 40.000 IOP/s not as much as i expected but OK right now.
> 
> How are you running your benchmarks Stefan?

I'm doing fio randwrite 4k from withing Qemu.

>> If i replace the SSDs on one host with spinning disks but still using
>> dedicated journal on ssd (20GB / Disk/OSDK), i'm not able to get more
>> than 300 to 400 iop/s this seems to be pretty low.
> 
> That's probably about right.  The journals really only absorb a small
> portion of the incoming writes for free, and then you end up having to
> wait on the disks behind the OSDs.

Why is it like that? I thought the OSD journal is like a small I/O
buffer merging them into bigger I/O chunks.

> If you have 4 spinning disks in the
> system, each one is only really capable of around 150-200 IOPs assuming
> typical 7200rpm units.  300-400 iops for 4 disks isn't great, but it's
> probably not totally unrealistic either.

Sure but i still have 20 SSDs so shouldn't there be only some I/O going
to the disks?

> As far as SSDs go, some folks seem to be having luck with Bcache and
> fastcache to improve performance of spinning disk backed OSDs.  I admit
> haven't had time to play with them yet but it's definitely on my list!

The bcache page is really small and i'm not sure how well it is
maintained. I may also try

https://github.com/stec-inc/EnhanceIO

which sounds really promising.

>> Everything tested using 0.56.4 and Qemu RBD.
> 
> Out of curiosity, do you have RBD cache enabled?  I noticed on my test
> setup that with 64G VM images it provide quite a bit of benefit even for
> small random writes.

Yes.

Stefan