increase duplication level decrease performance

increase duplication level decrease performance

[Fyodor Ustinov]

Hi.

ceph osd pool set data size 1

dd if=/dev/zero of=aaa bs=1024000 count=4000
4096000000 bytes (4.1 GB) copied, 31.3153 s, 131 MB/s

ceph osd pool set data size 2
4096000000 bytes (4.1 GB) copied, 72.7146 s, 56.3 MB/s

ceph osd pool set data size 3
4096000000 bytes (4.1 GB) copied, 136.263 s, 30.1 MB/s

Why? I thought increase in the number of copies should increase the 
performance (in the worst case does not affect).

WBR,
    Fyodor.

Re: increase duplication level decrease performance

[Gregory Farnum]

On Monday, April 4, 2011 at 2:55 PM, Fyodor Ustinov wrote:
Hi.
> 
> ceph osd pool set data size 1
> 
> dd if=/dev/zero of=aaa bs=1024000 count=4000
> 4096000000 bytes (4.1 GB) copied, 31.3153 s, 131 MB/s
> 
> ceph osd pool set data size 2
> 4096000000 bytes (4.1 GB) copied, 72.7146 s, 56.3 MB/s
> 
> ceph osd pool set data size 3
> 4096000000 bytes (4.1 GB) copied, 136.263 s, 30.1 MB/s
> 
> Why? I thought increase in the number of copies should increase the 
> performance (in the worst case does not affect).
> 
> WBR,
>  Fyodor.

Data replicas don't service reads unless you specifically configure it for that (it doesn't provide POSIX semantics if you do that), and each replica needs to get the data written before it's considered safe. In your case you're doing a dd, which is a write to disk, so you're increasing how much data needs to get sent over the network and written to disk by increasing replication.

Without knowing more about your setup though we can't tell exactly why the performance is dropping as it is. Some possibilities:
1) You have a very small number of OSDs, so you're saturating the NIC on the primaries
2) You have a very small number of OSDs, so you're saturating the disk on all of them (this is particularly likely if you've got the journal on the same drive as the main store, since that's an automatic 50% cut in disk throughput).
3) Everything is connected via one cheap switch, which can't handle the traffic.
4) You've got some very slow OSDs, which will slow down each Placement Group they're a member of (and with higher replication levels you will hit them more often).
etc

-Greg 

Re: increase duplication level decrease performance

[Fyodor Ustinov]

On 04/05/2011 01:52 AM, Gregory Farnum wrote:
> Data replicas don't service reads unless you specifically configure it for that (it doesn't provide POSIX semantics if you do that), and each replica needs to get the data written before it's considered safe. In your case you're doing a dd, which is a write to disk, so you're increasing how much data needs to get sent over the network and written to disk by increasing replication.
>
> Without knowing more about your setup though we can't tell exactly why the performance is dropping as it is. Some possibilities:
> 1) You have a very small number of OSDs, so you're saturating the NIC on the primaries
> 2) You have a very small number of OSDs, so you're saturating the disk on all of them (this is particularly likely if you've got the journal on the same drive as the main store, since that's an automatic 50% cut in disk throughput).
> 3) Everything is connected via one cheap switch, which can't handle the traffic.
> 4) You've got some very slow OSDs, which will slow down each Placement Group they're a member of (and with higher replication levels you will hit them more often).
> etc
>
Many thanks for your reply.

If I correct understood - the maximum write speed to the file system in 
the ideal case is the network bandwidth from the client? Or bandwidth 
divided by the number of copies?

How much OSDs recommended for 3 replicas?

P.S. without documentation have to ask stupid questions.:)

Re: increase duplication level decrease performance

[Gregory Farnum]

On Monday, April 4, 2011 at 4:28 PM, Fyodor Ustinov wrote: 
> On 04/05/2011 01:52 AM, Gregory Farnum wrote:
> > Data replicas don't service reads unless you specifically configure it for that (it doesn't provide POSIX semantics if you do that), and each replica needs to get the data written before it's considered safe. In your case you're doing a dd, which is a write to disk, so you're increasing how much data needs to get sent over the network and written to disk by increasing replication.
> > 
> > Without knowing more about your setup though we can't tell exactly why the performance is dropping as it is. Some possibilities:
> > 1) You have a very small number of OSDs, so you're saturating the NIC on the primaries
> > 2) You have a very small number of OSDs, so you're saturating the disk on all of them (this is particularly likely if you've got the journal on the same drive as the main store, since that's an automatic 50% cut in disk throughput).
> > 3) Everything is connected via one cheap switch, which can't handle the traffic.
> > 4) You've got some very slow OSDs, which will slow down each Placement Group they're a member of (and with higher replication levels you will hit them more often).
> > etc
> Many thanks for your reply.
> 
> If I correct understood - the maximum write speed to the file system in 
> the ideal case is the network bandwidth from the client? Or bandwidth 
> divided by the number of copies?

In ideal case it would be the network bandwidth from the client, as it only sends data once, to the placement group primary. The primary OSDs send out the data to the replicas. The one thing here is that the primary OSD sends out the data to each replica, but with enough OSDs that shouldn't be the limiting factor since the primaries will be spread out all over the cluster.

> 
> How much OSDs recommended for 3 replicas?

Hmm, haven't looked at that too much. As many as you need for acceptable performance, I suppose. ;)

> P.S. without documentation have to ask stupid questions.:)
> 
Yeah -- there is some information in the wiki, which we encourage everybody to edit; there is lots more in the academic papers if you're up for those; but we hope to be generating more documentation in various forms over the coming months.
-Greg