Guidelines for Calculating IOPS?

Guidelines for Calculating IOPS?

[Mike Dawson]

All,

I am investigating the use of Ceph for a video surveillance project with 
the following minimum block storage requirements:

385 Mbps of constant write bandwidth
100TB storage requirement
5250 IOPS (size of ~8 KB)

I believe 2 replicas would be acceptable. We intend to use large 
capacity (2 or 3TB) SATA 7200rpm 3.5" drives, if the IOPS work out properly.

Is there a method / formula to estimate IOPS for RDB? Specifically I 
would like to understand:

- How does replica count affect read/write IOPS?

- I'm trying to understand best practice for when to optimize server 
count, drives per server, and drive capacity as it relates to IOPS. Is 
there a point of diminishing I/O performance using server chassis with 
lots of drive slots, like the 36-drive Supermicro SC847a?


Thanks,
Mike

Re: Guidelines for Calculating IOPS?

[Mark Kampe]

Replication should have no effect on read throughput/IOPS.

The client does a single write to the primary, and the
primary then handles re-replication to the secondary
copies.  As such the client does not pay (in terms of
CPU or NIC bandwidth) for the replication.  Per-client
throughput limitations should be largely independent of
the replication.

However, the replication does generate additional network
and I/O activity between the OSDs.  This means that the
available aggregate throughput (of the entire cluster)
is effectively cut in half when you move from one-copy to two.

I am confused by your math:

    You say 385MB/s and 5250 IOPS (x8k)
    5250 IOPS * 8192 = 43MB/s

Do you mean that some of your clients are generating
a lot of small block writes (at up to 5250 IPS) and
that others of your clients are doing larger writes
(with an aggregate throughput of 385MB/s)?

For RADOS throughput:
    385MB/s is a fairly small number
    5250 buffered sequential IOPS is a very small number
    5250 random IOPS is not a particularly large
         number, but will require several servers

My guess is that the IOPS may drive the number of
servers, and the drives per server will be the
capacity divided by the number of required servers.

So how many IOPS can you get per server?

You are using RBD, and depending on the particulars
of your stack, there may be a great deal of buffering
and caching on the client side that can make the
RADOS traffic much more efficient than the tributary
client requests.  Thus, I would suggest that you
probably want to actually benchmark the application
in question to measure the client-experienced throughput.


On 10/19/12 07:47, Mike Dawson wrote:
> All,
>
> I am investigating the use of Ceph for a video surveillance project with
> the following minimum block storage requirements:
>
> 385 Mbps of constant write bandwidth
> 100TB storage requirement
> 5250 IOPS (size of ~8 KB)
>
> I believe 2 replicas would be acceptable. We intend to use large
> capacity (2 or 3TB) SATA 7200rpm 3.5" drives, if the IOPS work out
> properly.
>
> Is there a method / formula to estimate IOPS for RDB? Specifically I
> would like to understand:
>
> - How does replica count affect read/write IOPS?
>
> - I'm trying to understand best practice for when to optimize server
> count, drives per server, and drive capacity as it relates to IOPS. Is
> there a point of diminishing I/O performance using server chassis with
> lots of drive slots, like the 36-drive Supermicro SC847a?

Re: Guidelines for Calculating IOPS?

[Wido den Hollander]

On 10/19/2012 07:45 PM, Mark Kampe wrote:
> Replication should have no effect on read throughput/IOPS.
>
> The client does a single write to the primary, and the
> primary then handles re-replication to the secondary
> copies.  As such the client does not pay (in terms of
> CPU or NIC bandwidth) for the replication.  Per-client
> throughput limitations should be largely independent of
> the replication.
>
> However, the replication does generate additional network
> and I/O activity between the OSDs.  This means that the
> available aggregate throughput (of the entire cluster)
> is effectively cut in half when you move from one-copy to two.
>

I think you can say this.

You have 100 disks each capable of doing ~100 IOps.

Read: 100 * 100 = 10.000 IOps
Write: (100 * 100 / 2) = 5.000 IOps

Since you are replicating everything twice you only have the speed of 
50% of the disks.

When reading the reads will be balanced over the available copies.

I've taken 100 IOps as a safe assumption for a regular SATA disk.

Wido

> I am confused by your math:
>
>     You say 385MB/s and 5250 IOPS (x8k)
>     5250 IOPS * 8192 = 43MB/s
>
> Do you mean that some of your clients are generating
> a lot of small block writes (at up to 5250 IPS) and
> that others of your clients are doing larger writes
> (with an aggregate throughput of 385MB/s)?
>
> For RADOS throughput:
>     385MB/s is a fairly small number
>     5250 buffered sequential IOPS is a very small number
>     5250 random IOPS is not a particularly large
>          number, but will require several servers
>
> My guess is that the IOPS may drive the number of
> servers, and the drives per server will be the
> capacity divided by the number of required servers.
>
> So how many IOPS can you get per server?
>
> You are using RBD, and depending on the particulars
> of your stack, there may be a great deal of buffering
> and caching on the client side that can make the
> RADOS traffic much more efficient than the tributary
> client requests.  Thus, I would suggest that you
> probably want to actually benchmark the application
> in question to measure the client-experienced throughput.
>
>
> On 10/19/12 07:47, Mike Dawson wrote:
>> All,
>>
>> I am investigating the use of Ceph for a video surveillance project with
>> the following minimum block storage requirements:
>>
>> 385 Mbps of constant write bandwidth
>> 100TB storage requirement
>> 5250 IOPS (size of ~8 KB)
>>
>> I believe 2 replicas would be acceptable. We intend to use large
>> capacity (2 or 3TB) SATA 7200rpm 3.5" drives, if the IOPS work out
>> properly.
>>
>> Is there a method / formula to estimate IOPS for RDB? Specifically I
>> would like to understand:
>>
>> - How does replica count affect read/write IOPS?
>>
>> - I'm trying to understand best practice for when to optimize server
>> count, drives per server, and drive capacity as it relates to IOPS. Is
>> there a point of diminishing I/O performance using server chassis with
>> lots of drive slots, like the 36-drive Supermicro SC847a?
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Guidelines for Calculating IOPS?

[Mike Dawson]

Mark,

Thanks! I really appreciate your reply!

On 10/19/2012 1:45 PM, Mark Kampe wrote:
> Replication should have no effect on read throughput/IOPS.
>
> The client does a single write to the primary, and the
> primary then handles re-replication to the secondary
> copies.  As such the client does not pay (in terms of
> CPU or NIC bandwidth) for the replication.  Per-client
> throughput limitations should be largely independent of
> the replication.
>
> However, the replication does generate additional network
> and I/O activity between the OSDs.  This means that the
> available aggregate throughput (of the entire cluster)
> is effectively cut in half when you move from one-copy to two.
>
Thanks for this confirmation of read and write throughput penalty.

As this storage is for video surveillance, I'd estimate a ratio of 90% 
(or higher) write to 10% reads. So, I really only care about the write IOPS.

> I am confused by your math:
>
>     You say 385MB/s and 5250 IOPS (x8k)
>     5250 IOPS * 8192 = 43MB/s
>

I believe you mistook 385Mbps for 385MB/s. I'm estimating 385Mbps aka 
48MB/s. The remainder or the confusion is rounding (as it's not exactly 8k).

> Do you mean that some of your clients are generating
> a lot of small block writes (at up to 5250 IPS) and
> that others of your clients are doing larger writes
> (with an aggregate throughput of 385MB/s)?
>
The workload is consistent from all clients. 683 video cameras recording 
7 fps. We plan 13-15 VMs running Network Video Recorder (NVR) software 
with camera count spread as evenly as possible across NVRs.

> For RADOS throughput:
>     385MB/s is a fairly small number
>     5250 buffered sequential IOPS is a very small number
>     5250 random IOPS is not a particularly large
>          number, but will require several servers

I don't know if this use fits into buffered sequential IOPS or random 
IOPS territory. Need to learn more about the NVR software probably.

What role would putting the journal on an SSD play here?

>
> My guess is that the IOPS may drive the number of
> servers, and the drives per server will be the
> capacity divided by the number of required servers.
>
> So how many IOPS can you get per server?

I don't know the answer, but would love input here! Assuming I use 
7200rpm drives and conservatively estimate 75 IOPS/drive. I think I need:

5250 / 75 * 3 = 210 drives for 3x replication
5250 / 75 * 2 = 140 drives for 2x replication

Assuming that's a good starting point, then it's all about how to bring 
those drives online. Ignoring extra servers for spare capacity / 
failures, you could do:

18 12-drive servers
  9 24-drive servers
  6 36-drive servers

Is there any reason to assume any other bounding factors like CPU, RAM, 
etc that would make more servers a better architecture?

>
> You are using RBD, and depending on the particulars
> of your stack, there may be a great deal of buffering
> and caching on the client side that can make the
> RADOS traffic much more efficient than the tributary
> client requests.  Thus, I would suggest that you
> probably want to actually benchmark the application
> in question to measure the client-experienced throughput.

Working on a small scale benchmark next week. Thanks for your help!

Thanks,
Mike

>
>
> On 10/19/12 07:47, Mike Dawson wrote:
>> All,
>>
>> I am investigating the use of Ceph for a video surveillance project with
>> the following minimum block storage requirements:
>>
>> 385 Mbps of constant write bandwidth
>> 100TB storage requirement
>> 5250 IOPS (size of ~8 KB)
>>
>> I believe 2 replicas would be acceptable. We intend to use large
>> capacity (2 or 3TB) SATA 7200rpm 3.5" drives, if the IOPS work out
>> properly.
>>
>> Is there a method / formula to estimate IOPS for RDB? Specifically I
>> would like to understand:
>>
>> - How does replica count affect read/write IOPS?
>>
>> - I'm trying to understand best practice for when to optimize server
>> count, drives per server, and drive capacity as it relates to IOPS. Is
>> there a point of diminishing I/O performance using server chassis with
>> lots of drive slots, like the 36-drive Supermicro SC847a?
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Guidelines for Calculating IOPS?

[Gregory Farnum]

On Fri, Oct 19, 2012 at 12:07 PM, Mike Dawson <mdawson@gammacode.com> wrote:
> I don't know if this use fits into buffered sequential IOPS or random IOPS
> territory. Need to learn more about the NVR software probably.

Video cameras are almost certainly not doing synchronous writes, which
means the data would be buffered and coalesced. You need to handle
~43MB/s of writes.
(At least, if that's not the case, then either something very strange
is happening or it's shockingly poorly written software.)

> What role would putting the journal on an SSD play here?

Not a ton — journals can absorb spikes in workload but everything
still needs to get written out to the main data store as individual
ops, so your long-term IOPs and throughput are capped at what your
main storage can provide (modulo replication, of course).


>> My guess is that the IOPS may drive the number of
>> servers, and the drives per server will be the
>> capacity divided by the number of required servers.
>>
>> So how many IOPS can you get per server?
>
>
> I don't know the answer, but would love input here! Assuming I use 7200rpm
> drives and conservatively estimate 75 IOPS/drive. I think I need:
>
> 5250 / 75 * 3 = 210 drives for 3x replication
> 5250 / 75 * 2 = 140 drives for 2x replication
>
> Assuming that's a good starting point, then it's all about how to bring
> those drives online. Ignoring extra servers for spare capacity / failures,
> you could do:
>
> 18 12-drive servers
>  9 24-drive servers
>  6 36-drive servers
>
> Is there any reason to assume any other bounding factors like CPU, RAM, etc
> that would make more servers a better architecture?

Well, each OSD takes up some memory and CPU for its own processes. We
generally recommend ~1GHz of modern CPU and 1GB of RAM per process;
these are very ballpark estimates but seem to work. The memory is
usually devoted mostly to page cache (the daemon itself is generally
at ~100MB last I checked) but it can balloon under certain failure
handling scenarios.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html