Re: Guidelines for Calculating IOPS?

[Mike Dawson <mdawson@gammacode.com>]

Mark,

Thanks! I really appreciate your reply!

On 10/19/2012 1:45 PM, Mark Kampe wrote:
> Replication should have no effect on read throughput/IOPS.
>
> The client does a single write to the primary, and the
> primary then handles re-replication to the secondary
> copies.  As such the client does not pay (in terms of
> CPU or NIC bandwidth) for the replication.  Per-client
> throughput limitations should be largely independent of
> the replication.
>
> However, the replication does generate additional network
> and I/O activity between the OSDs.  This means that the
> available aggregate throughput (of the entire cluster)
> is effectively cut in half when you move from one-copy to two.
>
Thanks for this confirmation of read and write throughput penalty.

As this storage is for video surveillance, I'd estimate a ratio of 90% 
(or higher) write to 10% reads. So, I really only care about the write IOPS.

> I am confused by your math:
>
>     You say 385MB/s and 5250 IOPS (x8k)
>     5250 IOPS * 8192 = 43MB/s
>

I believe you mistook 385Mbps for 385MB/s. I'm estimating 385Mbps aka 
48MB/s. The remainder or the confusion is rounding (as it's not exactly 8k).

> Do you mean that some of your clients are generating
> a lot of small block writes (at up to 5250 IPS) and
> that others of your clients are doing larger writes
> (with an aggregate throughput of 385MB/s)?
>
The workload is consistent from all clients. 683 video cameras recording 
7 fps. We plan 13-15 VMs running Network Video Recorder (NVR) software 
with camera count spread as evenly as possible across NVRs.

> For RADOS throughput:
>     385MB/s is a fairly small number
>     5250 buffered sequential IOPS is a very small number
>     5250 random IOPS is not a particularly large
>          number, but will require several servers

I don't know if this use fits into buffered sequential IOPS or random 
IOPS territory. Need to learn more about the NVR software probably.

What role would putting the journal on an SSD play here?

>
> My guess is that the IOPS may drive the number of
> servers, and the drives per server will be the
> capacity divided by the number of required servers.
>
> So how many IOPS can you get per server?

I don't know the answer, but would love input here! Assuming I use 
7200rpm drives and conservatively estimate 75 IOPS/drive. I think I need:

5250 / 75 * 3 = 210 drives for 3x replication
5250 / 75 * 2 = 140 drives for 2x replication

Assuming that's a good starting point, then it's all about how to bring 
those drives online. Ignoring extra servers for spare capacity / 
failures, you could do:

18 12-drive servers
  9 24-drive servers
  6 36-drive servers

Is there any reason to assume any other bounding factors like CPU, RAM, 
etc that would make more servers a better architecture?

>
> You are using RBD, and depending on the particulars
> of your stack, there may be a great deal of buffering
> and caching on the client side that can make the
> RADOS traffic much more efficient than the tributary
> client requests.  Thus, I would suggest that you
> probably want to actually benchmark the application
> in question to measure the client-experienced throughput.

Working on a small scale benchmark next week. Thanks for your help!

Thanks,
Mike

>
>
> On 10/19/12 07:47, Mike Dawson wrote:
>> All,
>>
>> I am investigating the use of Ceph for a video surveillance project with
>> the following minimum block storage requirements:
>>
>> 385 Mbps of constant write bandwidth
>> 100TB storage requirement
>> 5250 IOPS (size of ~8 KB)
>>
>> I believe 2 replicas would be acceptable. We intend to use large
>> capacity (2 or 3TB) SATA 7200rpm 3.5" drives, if the IOPS work out
>> properly.
>>
>> Is there a method / formula to estimate IOPS for RDB? Specifically I
>> would like to understand:
>>
>> - How does replica count affect read/write IOPS?
>>
>> - I'm trying to understand best practice for when to optimize server
>> count, drives per server, and drive capacity as it relates to IOPS. Is
>> there a point of diminishing I/O performance using server chassis with
>> lots of drive slots, like the 36-drive Supermicro SC847a?
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html