bandwidth with Ceph - v0.59 (Bobtail)

bandwidth with Ceph - v0.59 (Bobtail)

[Xing]

Hi,

I also did a few other experiments, trying to get what the maximum bandwidth we can get from each data disk. The output is not encouraging: for disks that can provide 150 MB/s block-level sequential read bandwidths, we can only get about 90MB/s from each disk. Something that is particular interesting is that the replica size also affects the bandwidth we could get from the cluster. It seems that there is no such observation/conversations in the Ceph community and I think it may be helpful to share my findings.  

The experiment was run with two d820 machines in Emulab at University of Utah. One is used as the data node and the other is used as the client. They are connected by a 10 GB/s Ethernet. The data node has 7 disks, one for OS and the rest 6 for OSDs. For the rest 6 disks, we use one for journal and the other for data. Thus in total we have 3 OSDs. The network bandwidth is sufficient to support reading from 3 disks in full bandwidth. 

I varied the read-ahead size for the rbd block device (exp1), osd op threads for each osd (exp2), varied the replica size (exp3), and object size (exp4). The most interesting is varying the replica size. As I varied replica size from 1, to 2 and to 3, the aggregated bandwidth dropped from 267 MB/s to 211 and 180. The reason for the drop I believe is as we increase the number of replicas, we store more data into each OSD. then when we need to read it back, we have to read from a larger range (more seeks). The fundamental problem is likely because we are doing replication synchronously, and thus layout object files in a raid 10 - near format, rather than the far format. For the difference between the near format and far format for raid 10, you could have a look at the link provided below. 

http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt

For results about other experiments, you could download my slides at the link provided below. 
http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx


I do not know why Ceph only gets about 60% of the disk bandwidth. To do a comparison, I ran tar to read every rbd object files to create a tarball and see how much bandwidth I can get from this workload. Interestingly, the tar workload actually gets a higher bandwidth (80% of block level bandwidth), even though it is accessing the disk more randomly (tar reads each object file in a dir sequentially while the object files were created in a different order.). For more detail, please goto my blog to have a read. 
http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html

Here are a few questions. 
1. What are the maximum bandwidth people can get from each disk? I found Jiangang from Intel also reported 57% efficiency for disk bandwidth. He suggested one reason: interference among so many sequential read workloads. I agree but when I tried to run with one single workload, I still do not get a higher efficiency. 
2. If the efficiency is about 60%, what are the reasons that cause this? Could it be because of the locks (futex as I mentioned in my previous email) or anything else? 

Thanks very much for any feedback. 

Thanks,
Xing

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Gregory Farnum]

Bobtail is really too old to draw any meaningful conclusions from; why
did you choose it?

That's not to say that performance on current code will be better
(though it very much might be), but the internal architecture has
changed in some ways that will be particularly important for the futex
profiling you did, and are probably important for these throughput
results as well.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
> Hi,
>
> I also did a few other experiments, trying to get what the maximum bandwidth we can get from each data disk. The output is not encouraging: for disks that can provide 150 MB/s block-level sequential read bandwidths, we can only get about 90MB/s from each disk. Something that is particular interesting is that the replica size also affects the bandwidth we could get from the cluster. It seems that there is no such observation/conversations in the Ceph community and I think it may be helpful to share my findings.
>
> The experiment was run with two d820 machines in Emulab at University of Utah. One is used as the data node and the other is used as the client. They are connected by a 10 GB/s Ethernet. The data node has 7 disks, one for OS and the rest 6 for OSDs. For the rest 6 disks, we use one for journal and the other for data. Thus in total we have 3 OSDs. The network bandwidth is sufficient to support reading from 3 disks in full bandwidth.
>
> I varied the read-ahead size for the rbd block device (exp1), osd op threads for each osd (exp2), varied the replica size (exp3), and object size (exp4). The most interesting is varying the replica size. As I varied replica size from 1, to 2 and to 3, the aggregated bandwidth dropped from 267 MB/s to 211 and 180. The reason for the drop I believe is as we increase the number of replicas, we store more data into each OSD. then when we need to read it back, we have to read from a larger range (more seeks). The fundamental problem is likely because we are doing replication synchronously, and thus layout object files in a raid 10 - near format, rather than the far format. For the difference between the near format and far format for raid 10, you could have a look at the link provided below.
>
> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt
>
> For results about other experiments, you could download my slides at the link provided below.
> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>
>
> I do not know why Ceph only gets about 60% of the disk bandwidth. To do a comparison, I ran tar to read every rbd object files to create a tarball and see how much bandwidth I can get from this workload. Interestingly, the tar workload actually gets a higher bandwidth (80% of block level bandwidth), even though it is accessing the disk more randomly (tar reads each object file in a dir sequentially while the object files were created in a different order.). For more detail, please goto my blog to have a read.
> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html
>
> Here are a few questions.
> 1. What are the maximum bandwidth people can get from each disk? I found Jiangang from Intel also reported 57% efficiency for disk bandwidth. He suggested one reason: interference among so many sequential read workloads. I agree but when I tried to run with one single workload, I still do not get a higher efficiency.
> 2. If the efficiency is about 60%, what are the reasons that cause this? Could it be because of the locks (futex as I mentioned in my previous email) or anything else?
>
> Thanks very much for any feedback.
>
> Thanks,
> Xing
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Xing]

Hi Gregory,

Thanks very much for your quick reply. When I started to look into Ceph, Bobtail was the latest stable release and that was why I picked that version and started to make a few modifications. I have not ported my changes to 0.79 yet. The plan is if v-0.79 can provide a higher disk bandwidth efficiency, I will switch to 0.79. Unfortunately, that does not seem to be the case. 

The futex trace was done with version 0.79, not 0.59. I did a profile in 0.59 too. There are some improvements, such as the introduction of fd cache. But lots of futex calls are still there in v-0.79. I also measured the maximum bandwidth from each disk we can get in Version 0.79. It does not improve significantly: we can still only get 90~100 MB/s from each disk. 

Thanks,
Xing


On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:

> Bobtail is really too old to draw any meaningful conclusions from; why
> did you choose it?
> 
> That's not to say that performance on current code will be better
> (though it very much might be), but the internal architecture has
> changed in some ways that will be particularly important for the futex
> profiling you did, and are probably important for these throughput
> results as well.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> 
> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>> Hi,
>> 
>> I also did a few other experiments, trying to get what the maximum bandwidth we can get from each data disk. The output is not encouraging: for disks that can provide 150 MB/s block-level sequential read bandwidths, we can only get about 90MB/s from each disk. Something that is particular interesting is that the replica size also affects the bandwidth we could get from the cluster. It seems that there is no such observation/conversations in the Ceph community and I think it may be helpful to share my findings.
>> 
>> The experiment was run with two d820 machines in Emulab at University of Utah. One is used as the data node and the other is used as the client. They are connected by a 10 GB/s Ethernet. The data node has 7 disks, one for OS and the rest 6 for OSDs. For the rest 6 disks, we use one for journal and the other for data. Thus in total we have 3 OSDs. The network bandwidth is sufficient to support reading from 3 disks in full bandwidth.
>> 
>> I varied the read-ahead size for the rbd block device (exp1), osd op threads for each osd (exp2), varied the replica size (exp3), and object size (exp4). The most interesting is varying the replica size. As I varied replica size from 1, to 2 and to 3, the aggregated bandwidth dropped from 267 MB/s to 211 and 180. The reason for the drop I believe is as we increase the number of replicas, we store more data into each OSD. then when we need to read it back, we have to read from a larger range (more seeks). The fundamental problem is likely because we are doing replication synchronously, and thus layout object files in a raid 10 - near format, rather than the far format. For the difference between the near format and far format for raid 10, you could have a look at the link provided below.
 
>> 
>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt
>> 
>> For results about other experiments, you could download my slides at the link provided below.
>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>> 
>> 
>> I do not know why Ceph only gets about 60% of the disk bandwidth. To do a comparison, I ran tar to read every rbd object files to create a tarball and see how much bandwidth I can get from this workload. Interestingly, the tar workload actually gets a higher bandwidth (80% of block level bandwidth), even though it is accessing the disk more randomly (tar reads each object file in a dir sequentially while the object files were created in a different order.). For more detail, please goto my blog to have a read.
>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html
>> 
>> Here are a few questions.
>> 1. What are the maximum bandwidth people can get from each disk? I found Jiangang from Intel also reported 57% efficiency for disk bandwidth. He suggested one reason: interference among so many sequential read workloads. I agree but when I tried to run with one single workload, I still do not get a higher efficiency.
>> 2. If the efficiency is about 60%, what are the reasons that cause this? Could it be because of the locks (futex as I mentioned in my previous email) or anything else?
>> 
>> Thanks very much for any feedback.
>> 
>> Thanks,
>> Xing
>> 
>> 
>> 
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Mark Nelson]

For what it's worth, I've been able to achieve up to around 120MB/s with 
btrfs before things fragment.

Mark

On 04/25/2014 03:59 PM, Xing wrote:
> Hi Gregory,
>
> Thanks very much for your quick reply. When I started to look into Ceph, Bobtail was the latest stable release and that was why I picked that version and started to make a few modifications. I have not ported my changes to 0.79 yet. The plan is if v-0.79 can provide a higher disk bandwidth efficiency, I will switch to 0.79. Unfortunately, that does not seem to be the case.
>
> The futex trace was done with version 0.79, not 0.59. I did a profile in 0.59 too. There are some improvements, such as the introduction of fd cache. But lots of futex calls are still there in v-0.79. I also measured the maximum bandwidth from each disk we can get in Version 0.79. It does not improve significantly: we can still only get 90~100 MB/s from each disk.
>
> Thanks,
> Xing
>
>
> On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:
>
>> Bobtail is really too old to draw any meaningful conclusions from; why
>> did you choose it?
>>
>> That's not to say that performance on current code will be better
>> (though it very much might be), but the internal architecture has
>> changed in some ways that will be particularly important for the futex
>> profiling you did, and are probably important for these throughput
>> results as well.
>> -Greg
>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>
>>
>> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>>> Hi,
>>>
>>> I also did a few other experiments, trying to get what the maximum bandwidth we can get from each data disk. The output is not encouraging: for disks that can provide 150 MB/s block-level sequential read bandwidths, we can only get about 90MB/s from each disk. Something that is particular interesting is that the replica size also affects the bandwidth we could get from the cluster. It seems that there is no such observation/conversations in the Ceph community and I think it may be helpful to share my findings.
>>>
>>> The experiment was run with two d820 machines in Emulab at University of Utah. One is used as the data node and the other is used as the client. They are connected by a 10 GB/s Ethernet. The data node has 7 disks, one for OS and the rest 6 for OSDs. For the rest 6 disks, we use one for journal and the other for data. Thus in total we have 3 OSDs. The network bandwidth is sufficient to support reading from 3 disks in full bandwidth.
>>>
>>> I varied the read-ahead size for the rbd block device (exp1), osd op threads for each osd (exp2), varied the replica size (exp3), and object size (exp4). The most interesting is varying the replica size. As I varied replica size from 1, to 2 and to 3, the aggregated bandwidth dropped from 267 MB/s to 211 and 180. The reason for the drop I believe is as we increase the number of replicas, we store more data into each OSD. then when we need to read it back, we have to read from a larger range (more seeks). The fundamental problem is likely because we are doing replication synchronously, and thus layout object files in a raid 10 - near format, rather than the far format. For the difference between the near format and far format for raid 10, you could have a look at the link provided below
 .
>>>
>>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt
>>>
>>> For results about other experiments, you could download my slides at the link provided below.
>>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>>>
>>>
>>> I do not know why Ceph only gets about 60% of the disk bandwidth. To do a comparison, I ran tar to read every rbd object files to create a tarball and see how much bandwidth I can get from this workload. Interestingly, the tar workload actually gets a higher bandwidth (80% of block level bandwidth), even though it is accessing the disk more randomly (tar reads each object file in a dir sequentially while the object files were created in a different order.). For more detail, please goto my blog to have a read.
>>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html
>>>
>>> Here are a few questions.
>>> 1. What are the maximum bandwidth people can get from each disk? I found Jiangang from Intel also reported 57% efficiency for disk bandwidth. He suggested one reason: interference among so many sequential read workloads. I agree but when I tried to run with one single workload, I still do not get a higher efficiency.
>>> 2. If the efficiency is about 60%, what are the reasons that cause this? Could it be because of the locks (futex as I mentioned in my previous email) or anything else?
>>>
>>> Thanks very much for any feedback.
>>>
>>> Thanks,
>>> Xing
>>>
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Xing Lin]

Hi Mark,

That seems pretty good. What is the block level sequential read 
bandwidth of your disks? What configuration did you use? What was the 
replica size, read_ahead for your rbds and what were the number of 
workloads you used? I used btrfs in my experiments as well.

Thanks,
Xing

On 04/25/2014 03:36 PM, Mark Nelson wrote:
> For what it's worth, I've been able to achieve up to around 120MB/s 
> with btrfs before things fragment.
>
> Mark
>
> On 04/25/2014 03:59 PM, Xing wrote:
>> Hi Gregory,
>>
>> Thanks very much for your quick reply. When I started to look into 
>> Ceph, Bobtail was the latest stable release and that was why I picked 
>> that version and started to make a few modifications. I have not 
>> ported my changes to 0.79 yet. The plan is if v-0.79 can provide a 
>> higher disk bandwidth efficiency, I will switch to 0.79. 
>> Unfortunately, that does not seem to be the case.
>>
>> The futex trace was done with version 0.79, not 0.59. I did a profile 
>> in 0.59 too. There are some improvements, such as the introduction of 
>> fd cache. But lots of futex calls are still there in v-0.79. I also 
>> measured the maximum bandwidth from each disk we can get in Version 
>> 0.79. It does not improve significantly: we can still only get 90~100 
>> MB/s from each disk.
>>
>> Thanks,
>> Xing
>>
>>
>> On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:
>>
>>> Bobtail is really too old to draw any meaningful conclusions from; why
>>> did you choose it?
>>>
>>> That's not to say that performance on current code will be better
>>> (though it very much might be), but the internal architecture has
>>> changed in some ways that will be particularly important for the futex
>>> profiling you did, and are probably important for these throughput
>>> results as well.
>>> -Greg
>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>
>>>
>>> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>>>> Hi,
>>>>
>>>> I also did a few other experiments, trying to get what the maximum 
>>>> bandwidth we can get from each data disk. The output is not 
>>>> encouraging: for disks that can provide 150 MB/s block-level 
>>>> sequential read bandwidths, we can only get about 90MB/s from each 
>>>> disk. Something that is particular interesting is that the replica 
>>>> size also affects the bandwidth we could get from the cluster. It 
>>>> seems that there is no such observation/conversations in the Ceph 
>>>> community and I think it may be helpful to share my findings.
>>>>
>>>> The experiment was run with two d820 machines in Emulab at 
>>>> University of Utah. One is used as the data node and the other is 
>>>> used as the client. They are connected by a 10 GB/s Ethernet. The 
>>>> data node has 7 disks, one for OS and the rest 6 for OSDs. For the 
>>>> rest 6 disks, we use one for journal and the other for data. Thus 
>>>> in total we have 3 OSDs. The network bandwidth is sufficient to 
>>>> support reading from 3 disks in full bandwidth.
>>>>
>>>> I varied the read-ahead size for the rbd block device (exp1), osd 
>>>> op threads for each osd (exp2), varied the replica size (exp3), and 
>>>> object size (exp4). The most interesting is varying the replica 
>>>> size. As I varied replica size from 1, to 2 and to 3, the 
>>>> aggregated bandwidth dropped from 267 MB/s to 211 and 180. The 
>>>> reason for the drop I believe is as we increase the number of 
>>>> replicas, we store more data into each OSD. then when we need to 
>>>> read it back, we have to read from a larger range (more seeks). The 
>>>> fundamental problem is likely because we are doing replication 
>>>> synchronously, and thus layout object files in a raid 10 - near 
>>>> format, rather than the far format. For the difference between the 
>>>> near format and far format for raid 10, you could have a look at 
>>>> the link provided below.
>>>>
>>>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt 
>>>>
>>>>
>>>> For results about other experiments, you could download my slides 
>>>> at the link provided below.
>>>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>>>>
>>>>
>>>> I do not know why Ceph only gets about 60% of the disk bandwidth. 
>>>> To do a comparison, I ran tar to read every rbd object files to 
>>>> create a tarball and see how much bandwidth I can get from this 
>>>> workload. Interestingly, the tar workload actually gets a higher 
>>>> bandwidth (80% of block level bandwidth), even though it is 
>>>> accessing the disk more randomly (tar reads each object file in a 
>>>> dir sequentially while the object files were created in a different 
>>>> order.). For more detail, please goto my blog to have a read.
>>>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html 
>>>>
>>>>
>>>> Here are a few questions.
>>>> 1. What are the maximum bandwidth people can get from each disk? I 
>>>> found Jiangang from Intel also reported 57% efficiency for disk 
>>>> bandwidth. He suggested one reason: interference among so many 
>>>> sequential read workloads. I agree but when I tried to run with one 
>>>> single workload, I still do not get a higher efficiency.
>>>> 2. If the efficiency is about 60%, what are the reasons that cause 
>>>> this? Could it be because of the locks (futex as I mentioned in my 
>>>> previous email) or anything else?
>>>>
>>>> Thanks very much for any feedback.
>>>>
>>>> Thanks,
>>>> Xing
>>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Mark Nelson]

I don't have any recent results published, but you can see some of the 
older results from bobtail here:

http://ceph.com/performance-2/argonaut-vs-bobtail-performance-preview/

Specifically, look at the 256 concurrent 4MB rados bench tests.  In a 6 
disk, 2 SSD configuration we could push about 800MB/s for writes (no 
replication) and around 600-700MB/s for reads with BTRFS.  On this 
controller using a RAID0 configuration with WB cache helps quite a bit, 
but in other tests I've seen similar results with a 9207-8i that doesn't 
have WB cache when BTRFS filestores and SSD journals are used.

Regarding the drives, they can do somewhere around 140-150MB/s large 
block writes with fio.

Replication definitely adds additional latency so aggregate write 
throughput goes down, though it seems the penalty is worst after the 
first replica and doesn't hurt as much with subsequent ones.

Mark


On 04/25/2014 04:50 PM, Xing Lin wrote:
> Hi Mark,
>
> That seems pretty good. What is the block level sequential read
> bandwidth of your disks? What configuration did you use? What was the
> replica size, read_ahead for your rbds and what were the number of
> workloads you used? I used btrfs in my experiments as well.
>
> Thanks,
> Xing
>
> On 04/25/2014 03:36 PM, Mark Nelson wrote:
>> For what it's worth, I've been able to achieve up to around 120MB/s
>> with btrfs before things fragment.
>>
>> Mark
>>
>> On 04/25/2014 03:59 PM, Xing wrote:
>>> Hi Gregory,
>>>
>>> Thanks very much for your quick reply. When I started to look into
>>> Ceph, Bobtail was the latest stable release and that was why I picked
>>> that version and started to make a few modifications. I have not
>>> ported my changes to 0.79 yet. The plan is if v-0.79 can provide a
>>> higher disk bandwidth efficiency, I will switch to 0.79.
>>> Unfortunately, that does not seem to be the case.
>>>
>>> The futex trace was done with version 0.79, not 0.59. I did a profile
>>> in 0.59 too. There are some improvements, such as the introduction of
>>> fd cache. But lots of futex calls are still there in v-0.79. I also
>>> measured the maximum bandwidth from each disk we can get in Version
>>> 0.79. It does not improve significantly: we can still only get 90~100
>>> MB/s from each disk.
>>>
>>> Thanks,
>>> Xing
>>>
>>>
>>> On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:
>>>
>>>> Bobtail is really too old to draw any meaningful conclusions from; why
>>>> did you choose it?
>>>>
>>>> That's not to say that performance on current code will be better
>>>> (though it very much might be), but the internal architecture has
>>>> changed in some ways that will be particularly important for the futex
>>>> profiling you did, and are probably important for these throughput
>>>> results as well.
>>>> -Greg
>>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>>
>>>>
>>>> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>>>>> Hi,
>>>>>
>>>>> I also did a few other experiments, trying to get what the maximum
>>>>> bandwidth we can get from each data disk. The output is not
>>>>> encouraging: for disks that can provide 150 MB/s block-level
>>>>> sequential read bandwidths, we can only get about 90MB/s from each
>>>>> disk. Something that is particular interesting is that the replica
>>>>> size also affects the bandwidth we could get from the cluster. It
>>>>> seems that there is no such observation/conversations in the Ceph
>>>>> community and I think it may be helpful to share my findings.
>>>>>
>>>>> The experiment was run with two d820 machines in Emulab at
>>>>> University of Utah. One is used as the data node and the other is
>>>>> used as the client. They are connected by a 10 GB/s Ethernet. The
>>>>> data node has 7 disks, one for OS and the rest 6 for OSDs. For the
>>>>> rest 6 disks, we use one for journal and the other for data. Thus
>>>>> in total we have 3 OSDs. The network bandwidth is sufficient to
>>>>> support reading from 3 disks in full bandwidth.
>>>>>
>>>>> I varied the read-ahead size for the rbd block device (exp1), osd
>>>>> op threads for each osd (exp2), varied the replica size (exp3), and
>>>>> object size (exp4). The most interesting is varying the replica
>>>>> size. As I varied replica size from 1, to 2 and to 3, the
>>>>> aggregated bandwidth dropped from 267 MB/s to 211 and 180. The
>>>>> reason for the drop I believe is as we increase the number of
>>>>> replicas, we store more data into each OSD. then when we need to
>>>>> read it back, we have to read from a larger range (more seeks). The
>>>>> fundamental problem is likely because we are doing replication
>>>>> synchronously, and thus layout object files in a raid 10 - near
>>>>> format, rather than the far format. For the difference between the
>>>>> near format and far format for raid 10, you could have a look at
>>>>> the link provided below.
>>>>>
>>>>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt
>>>>>
>>>>>
>>>>> For results about other experiments, you could download my slides
>>>>> at the link provided below.
>>>>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>>>>>
>>>>>
>>>>> I do not know why Ceph only gets about 60% of the disk bandwidth.
>>>>> To do a comparison, I ran tar to read every rbd object files to
>>>>> create a tarball and see how much bandwidth I can get from this
>>>>> workload. Interestingly, the tar workload actually gets a higher
>>>>> bandwidth (80% of block level bandwidth), even though it is
>>>>> accessing the disk more randomly (tar reads each object file in a
>>>>> dir sequentially while the object files were created in a different
>>>>> order.). For more detail, please goto my blog to have a read.
>>>>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html
>>>>>
>>>>>
>>>>> Here are a few questions.
>>>>> 1. What are the maximum bandwidth people can get from each disk? I
>>>>> found Jiangang from Intel also reported 57% efficiency for disk
>>>>> bandwidth. He suggested one reason: interference among so many
>>>>> sequential read workloads. I agree but when I tried to run with one
>>>>> single workload, I still do not get a higher efficiency.
>>>>> 2. If the efficiency is about 60%, what are the reasons that cause
>>>>> this? Could it be because of the locks (futex as I mentioned in my
>>>>> previous email) or anything else?
>>>>>
>>>>> Thanks very much for any feedback.
>>>>>
>>>>> Thanks,
>>>>> Xing
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Xing Lin]

Hi Mark,

Thanks for sharing this. I did read these blogs early. If we look at the 
aggregated bandwidth, 600-700 MB/s for reads for 6 disks are quite good. 
But consider it is shared among 256 concurrent read streams, each one 
gets as little as 2-3 MB/s bandwidth. This does not sound that right.

I think the read bandwidth of a disk will be close to its write 
bandwidth. But just double-check: what the sequential read bandwidth 
your disks can provide?

I also read your follow-up blogs, comparing bobtail and cuttlefish. One 
thing I do not get from your experiments is that it hit the network 
bottleneck much earlier before being bottlenecked by disks. Could you 
setup a smaller cluster (e.g with 8 disks, rather than 24) such as a 10 
Gb/s link will not become the bottleneck and then test how much disk 
bandwidth can be achieved, preferably with new releases of Ceph. The 
other concern is I am not sure how close RADOS bench results are when 
compared with kernel RBD performance. I would appreciate it if you can 
do that. Thanks,

Xing

On 04/25/2014 04:16 PM, Mark Nelson wrote:
> I don't have any recent results published, but you can see some of the 
> older results from bobtail here:
>
> http://ceph.com/performance-2/argonaut-vs-bobtail-performance-preview/
>
> Specifically, look at the 256 concurrent 4MB rados bench tests. In a 6 
> disk, 2 SSD configuration we could push about 800MB/s for writes (no 
> replication) and around 600-700MB/s for reads with BTRFS.  On this 
> controller using a RAID0 configuration with WB cache helps quite a 
> bit, but in other tests I've seen similar results with a 9207-8i that 
> doesn't have WB cache when BTRFS filestores and SSD journals are used.
>
> Regarding the drives, they can do somewhere around 140-150MB/s large 
> block writes with fio.
>
> Replication definitely adds additional latency so aggregate write 
> throughput goes down, though it seems the penalty is worst after the 
> first replica and doesn't hurt as much with subsequent ones.
>
> Mark
>
>
> On 04/25/2014 04:50 PM, Xing Lin wrote:
>> Hi Mark,
>>
>> That seems pretty good. What is the block level sequential read
>> bandwidth of your disks? What configuration did you use? What was the
>> replica size, read_ahead for your rbds and what were the number of
>> workloads you used? I used btrfs in my experiments as well.
>>
>> Thanks,
>> Xing
>>
>> On 04/25/2014 03:36 PM, Mark Nelson wrote:
>>> For what it's worth, I've been able to achieve up to around 120MB/s
>>> with btrfs before things fragment.
>>>
>>> Mark
>>>
>>> On 04/25/2014 03:59 PM, Xing wrote:
>>>> Hi Gregory,
>>>>
>>>> Thanks very much for your quick reply. When I started to look into
>>>> Ceph, Bobtail was the latest stable release and that was why I picked
>>>> that version and started to make a few modifications. I have not
>>>> ported my changes to 0.79 yet. The plan is if v-0.79 can provide a
>>>> higher disk bandwidth efficiency, I will switch to 0.79.
>>>> Unfortunately, that does not seem to be the case.
>>>>
>>>> The futex trace was done with version 0.79, not 0.59. I did a profile
>>>> in 0.59 too. There are some improvements, such as the introduction of
>>>> fd cache. But lots of futex calls are still there in v-0.79. I also
>>>> measured the maximum bandwidth from each disk we can get in Version
>>>> 0.79. It does not improve significantly: we can still only get 90~100
>>>> MB/s from each disk.
>>>>
>>>> Thanks,
>>>> Xing
>>>>
>>>>
>>>> On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:
>>>>
>>>>> Bobtail is really too old to draw any meaningful conclusions from; 
>>>>> why
>>>>> did you choose it?
>>>>>
>>>>> That's not to say that performance on current code will be better
>>>>> (though it very much might be), but the internal architecture has
>>>>> changed in some ways that will be particularly important for the 
>>>>> futex
>>>>> profiling you did, and are probably important for these throughput
>>>>> results as well.
>>>>> -Greg
>>>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>>>
>>>>>
>>>>> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I also did a few other experiments, trying to get what the maximum
>>>>>> bandwidth we can get from each data disk. The output is not
>>>>>> encouraging: for disks that can provide 150 MB/s block-level
>>>>>> sequential read bandwidths, we can only get about 90MB/s from each
>>>>>> disk. Something that is particular interesting is that the replica
>>>>>> size also affects the bandwidth we could get from the cluster. It
>>>>>> seems that there is no such observation/conversations in the Ceph
>>>>>> community and I think it may be helpful to share my findings.
>>>>>>
>>>>>> The experiment was run with two d820 machines in Emulab at
>>>>>> University of Utah. One is used as the data node and the other is
>>>>>> used as the client. They are connected by a 10 GB/s Ethernet. The
>>>>>> data node has 7 disks, one for OS and the rest 6 for OSDs. For the
>>>>>> rest 6 disks, we use one for journal and the other for data. Thus
>>>>>> in total we have 3 OSDs. The network bandwidth is sufficient to
>>>>>> support reading from 3 disks in full bandwidth.
>>>>>>
>>>>>> I varied the read-ahead size for the rbd block device (exp1), osd
>>>>>> op threads for each osd (exp2), varied the replica size (exp3), and
>>>>>> object size (exp4). The most interesting is varying the replica
>>>>>> size. As I varied replica size from 1, to 2 and to 3, the
>>>>>> aggregated bandwidth dropped from 267 MB/s to 211 and 180. The
>>>>>> reason for the drop I believe is as we increase the number of
>>>>>> replicas, we store more data into each OSD. then when we need to
>>>>>> read it back, we have to read from a larger range (more seeks). The
>>>>>> fundamental problem is likely because we are doing replication
>>>>>> synchronously, and thus layout object files in a raid 10 - near
>>>>>> format, rather than the far format. For the difference between the
>>>>>> near format and far format for raid 10, you could have a look at
>>>>>> the link provided below.
>>>>>>
>>>>>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt 
>>>>>>
>>>>>>
>>>>>>
>>>>>> For results about other experiments, you could download my slides
>>>>>> at the link provided below.
>>>>>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>>>>>>
>>>>>>
>>>>>> I do not know why Ceph only gets about 60% of the disk bandwidth.
>>>>>> To do a comparison, I ran tar to read every rbd object files to
>>>>>> create a tarball and see how much bandwidth I can get from this
>>>>>> workload. Interestingly, the tar workload actually gets a higher
>>>>>> bandwidth (80% of block level bandwidth), even though it is
>>>>>> accessing the disk more randomly (tar reads each object file in a
>>>>>> dir sequentially while the object files were created in a different
>>>>>> order.). For more detail, please goto my blog to have a read.
>>>>>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html 
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here are a few questions.
>>>>>> 1. What are the maximum bandwidth people can get from each disk? I
>>>>>> found Jiangang from Intel also reported 57% efficiency for disk
>>>>>> bandwidth. He suggested one reason: interference among so many
>>>>>> sequential read workloads. I agree but when I tried to run with one
>>>>>> single workload, I still do not get a higher efficiency.
>>>>>> 2. If the efficiency is about 60%, what are the reasons that cause
>>>>>> this? Could it be because of the locks (futex as I mentioned in my
>>>>>> previous email) or anything else?
>>>>>>
>>>>>> Thanks very much for any feedback.
>>>>>>
>>>>>> Thanks,
>>>>>> Xing
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> ceph-devel" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Xing]

One more thing that needs to be considered is as we add more sequential workloads into the disk, the aggregated bandwidth will start to drop. For example, for the TOSHIBA MBF2600RC SCSI disk, we could get 155 MB/s sequential read bandwidth for a single workload. As we add more, the aggregated bandwidth will drop. 

number of SRs, aggregated bandwidth (KB/s)
1 154145
2 144296
3 147994
4 141063
5 134698
6 133874
7 130915
8 132366
9 97068
10 111897
11 108508.5
12 106450.9
13 105521.9
14 102411.7
15 102618.2
16 102779.1
17 102745

As we can see, we can only get about 100MB/s once we are running ~10 concurrent workloads. For your case, you were running 256 concurrent read streams for 6/8 disks, I would expect the aggregated disk bandwidth too be lower than 100 MB/s per disk. Any thoughts? 

Thanks,
Xing

On Apr 25, 2014, at 5:47 PM, Xing Lin <xinglin@cs.utah.edu> wrote:

> Hi Mark,
> 
> Thanks for sharing this. I did read these blogs early. If we look at the aggregated bandwidth, 600-700 MB/s for reads for 6 disks are quite good. But consider it is shared among 256 concurrent read streams, each one gets as little as 2-3 MB/s bandwidth. This does not sound that right.
> 
> I think the read bandwidth of a disk will be close to its write bandwidth. But just double-check: what the sequential read bandwidth your disks can provide?
> 
> I also read your follow-up blogs, comparing bobtail and cuttlefish. One thing I do not get from your experiments is that it hit the network bottleneck much earlier before being bottlenecked by disks. Could you setup a smaller cluster (e.g with 8 disks, rather than 24) such as a 10 Gb/s link will not become the bottleneck and then test how much disk bandwidth can be achieved, preferably with new releases of Ceph. The other concern is I am not sure how close RADOS bench results are when compared with kernel RBD performance. I would appreciate it if you can do that. Thanks,
> 
> Xing
> 
> On 04/25/2014 04:16 PM, Mark Nelson wrote:
>> I don't have any recent results published, but you can see some of the older results from bobtail here:
>> 
>> http://ceph.com/performance-2/argonaut-vs-bobtail-performance-preview/
>> 
>> Specifically, look at the 256 concurrent 4MB rados bench tests. In a 6 disk, 2 SSD configuration we could push about 800MB/s for writes (no replication) and around 600-700MB/s for reads with BTRFS.  On this controller using a RAID0 configuration with WB cache helps quite a bit, but in other tests I've seen similar results with a 9207-8i that doesn't have WB cache when BTRFS filestores and SSD journals are used.
>> 
>> Regarding the drives, they can do somewhere around 140-150MB/s large block writes with fio.
>> 
>> Replication definitely adds additional latency so aggregate write throughput goes down, though it seems the penalty is worst after the first replica and doesn't hurt as much with subsequent ones.
>> 
>> Mark
>> 
>> 
>> On 04/25/2014 04:50 PM, Xing Lin wrote:
>>> Hi Mark,
>>> 
>>> That seems pretty good. What is the block level sequential read
>>> bandwidth of your disks? What configuration did you use? What was the
>>> replica size, read_ahead for your rbds and what were the number of
>>> workloads you used? I used btrfs in my experiments as well.
>>> 
>>> Thanks,
>>> Xing
>>> 
>>> On 04/25/2014 03:36 PM, Mark Nelson wrote:
>>>> For what it's worth, I've been able to achieve up to around 120MB/s
>>>> with btrfs before things fragment.
>>>> 
>>>> Mark
>>>> 
>>>> On 04/25/2014 03:59 PM, Xing wrote:
>>>>> Hi Gregory,
>>>>> 
>>>>> Thanks very much for your quick reply. When I started to look into
>>>>> Ceph, Bobtail was the latest stable release and that was why I picked
>>>>> that version and started to make a few modifications. I have not
>>>>> ported my changes to 0.79 yet. The plan is if v-0.79 can provide a
>>>>> higher disk bandwidth efficiency, I will switch to 0.79.
>>>>> Unfortunately, that does not seem to be the case.
>>>>> 
>>>>> The futex trace was done with version 0.79, not 0.59. I did a profile
>>>>> in 0.59 too. There are some improvements, such as the introduction of
>>>>> fd cache. But lots of futex calls are still there in v-0.79. I also
>>>>> measured the maximum bandwidth from each disk we can get in Version
>>>>> 0.79. It does not improve significantly: we can still only get 90~100
>>>>> MB/s from each disk.
>>>>> 
>>>>> Thanks,
>>>>> Xing
>>>>> 
>>>>> 
>>>>> On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:
>>>>> 
>>>>>> Bobtail is really too old to draw any meaningful conclusions from; why
>>>>>> did you choose it?
>>>>>> 
>>>>>> That's not to say that performance on current code will be better
>>>>>> (though it very much might be), but the internal architecture has
>>>>>> changed in some ways that will be particularly important for the futex
>>>>>> profiling you did, and are probably important for these throughput
>>>>>> results as well.
>>>>>> -Greg
>>>>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>>>> 
>>>>>> 
>>>>>> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I also did a few other experiments, trying to get what the maximum
>>>>>>> bandwidth we can get from each data disk. The output is not
>>>>>>> encouraging: for disks that can provide 150 MB/s block-level
>>>>>>> sequential read bandwidths, we can only get about 90MB/s from each
>>>>>>> disk. Something that is particular interesting is that the replica
>>>>>>> size also affects the bandwidth we could get from the cluster. It
>>>>>>> seems that there is no such observation/conversations in the Ceph
>>>>>>> community and I think it may be helpful to share my findings.
>>>>>>> 
>>>>>>> The experiment was run with two d820 machines in Emulab at
>>>>>>> University of Utah. One is used as the data node and the other is
>>>>>>> used as the client. They are connected by a 10 GB/s Ethernet. The
>>>>>>> data node has 7 disks, one for OS and the rest 6 for OSDs. For the
>>>>>>> rest 6 disks, we use one for journal and the other for data. Thus
>>>>>>> in total we have 3 OSDs. The network bandwidth is sufficient to
>>>>>>> support reading from 3 disks in full bandwidth.
>>>>>>> 
>>>>>>> I varied the read-ahead size for the rbd block device (exp1), osd
>>>>>>> op threads for each osd (exp2), varied the replica size (exp3), and
>>>>>>> object size (exp4). The most interesting is varying the replica
>>>>>>> size. As I varied replica size from 1, to 2 and to 3, the
>>>>>>> aggregated bandwidth dropped from 267 MB/s to 211 and 180. The
>>>>>>> reason for the drop I believe is as we increase the number of
>>>>>>> replicas, we store more data into each OSD. then when we need to
>>>>>>> read it back, we have to read from a larger range (more seeks). The
>>>>>>> fundamental problem is likely because we are doing replication
>>>>>>> synchronously, and thus layout object files in a raid 10 - near
>>>>>>> format, rather than the far format. For the difference between the
>>>>>>> near format and far format for raid 10, you could have a look at
>>>>>>> the link provided below.
>>>>>>> 
>>>>>>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt 
>>>>>>> 
>>>>>>> 
>>>>>>> For results about other experiments, you could download my slides
>>>>>>> at the link provided below.
>>>>>>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>>>>>>> 
>>>>>>> 
>>>>>>> I do not know why Ceph only gets about 60% of the disk bandwidth.
>>>>>>> To do a comparison, I ran tar to read every rbd object files to
>>>>>>> create a tarball and see how much bandwidth I can get from this
>>>>>>> workload. Interestingly, the tar workload actually gets a higher
>>>>>>> bandwidth (80% of block level bandwidth), even though it is
>>>>>>> accessing the disk more randomly (tar reads each object file in a
>>>>>>> dir sequentially while the object files were created in a different
>>>>>>> order.). For more detail, please goto my blog to have a read.
>>>>>>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html 
>>>>>>> 
>>>>>>> 
>>>>>>> Here are a few questions.
>>>>>>> 1. What are the maximum bandwidth people can get from each disk? I
>>>>>>> found Jiangang from Intel also reported 57% efficiency for disk
>>>>>>> bandwidth. He suggested one reason: interference among so many
>>>>>>> sequential read workloads. I agree but when I tried to run with one
>>>>>>> single workload, I still do not get a higher efficiency.
>>>>>>> 2. If the efficiency is about 60%, what are the reasons that cause
>>>>>>> this? Could it be because of the locks (futex as I mentioned in my
>>>>>>> previous email) or anything else?
>>>>>>> 
>>>>>>> Thanks very much for any feedback.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Xing
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> ceph-devel" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>> 
>>>>> -- 
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>> 
>>> 
> 

Re: bandwidth with Ceph - v0.59 (Bobtail)

[Milosz Tanski]

Xing,

I would be interested to see some more re-search into the lock
contention in Ceph using the futex benchmark, mutrace or similar
tools. It's entirely possible that Ceph could benefit from some more
complex locking strategies or data structure (just like the kernel has
in the last 5 years).

The good news is that there's actually portable C++ libraries for
these container nowadays (like http://libcds.sourceforge.net/). I know
there's a few places in OSD that could *possibly* benefit from this
(like some of the LRU caches that could do away with locks for
lookups).

Best,
- Milosz

On Fri, Apr 25, 2014 at 8:06 PM, Xing <xinglin@cs.utah.edu> wrote:
> One more thing that needs to be considered is as we add more sequential workloads into the disk, the aggregated bandwidth will start to drop. For example, for the TOSHIBA MBF2600RC SCSI disk, we could get 155 MB/s sequential read bandwidth for a single workload. As we add more, the aggregated bandwidth will drop.
>
> number of SRs, aggregated bandwidth (KB/s)
> 1 154145
> 2 144296
> 3 147994
> 4 141063
> 5 134698
> 6 133874
> 7 130915
> 8 132366
> 9 97068
> 10 111897
> 11 108508.5
> 12 106450.9
> 13 105521.9
> 14 102411.7
> 15 102618.2
> 16 102779.1
> 17 102745
>
> As we can see, we can only get about 100MB/s once we are running ~10 concurrent workloads. For your case, you were running 256 concurrent read streams for 6/8 disks, I would expect the aggregated disk bandwidth too be lower than 100 MB/s per disk. Any thoughts?
>
> Thanks,
> Xing
>
> On Apr 25, 2014, at 5:47 PM, Xing Lin <xinglin@cs.utah.edu> wrote:
>
>> Hi Mark,
>>
>> Thanks for sharing this. I did read these blogs early. If we look at the aggregated bandwidth, 600-700 MB/s for reads for 6 disks are quite good. But consider it is shared among 256 concurrent read streams, each one gets as little as 2-3 MB/s bandwidth. This does not sound that right.
>>
>> I think the read bandwidth of a disk will be close to its write bandwidth. But just double-check: what the sequential read bandwidth your disks can provide?
>>
>> I also read your follow-up blogs, comparing bobtail and cuttlefish. One thing I do not get from your experiments is that it hit the network bottleneck much earlier before being bottlenecked by disks. Could you setup a smaller cluster (e.g with 8 disks, rather than 24) such as a 10 Gb/s link will not become the bottleneck and then test how much disk bandwidth can be achieved, preferably with new releases of Ceph. The other concern is I am not sure how close RADOS bench results are when compared with kernel RBD performance. I would appreciate it if you can do that. Thanks,
>>
>> Xing
>>
>> On 04/25/2014 04:16 PM, Mark Nelson wrote:
>>> I don't have any recent results published, but you can see some of the older results from bobtail here:
>>>
>>> http://ceph.com/performance-2/argonaut-vs-bobtail-performance-preview/
>>>
>>> Specifically, look at the 256 concurrent 4MB rados bench tests. In a 6 disk, 2 SSD configuration we could push about 800MB/s for writes (no replication) and around 600-700MB/s for reads with BTRFS.  On this controller using a RAID0 configuration with WB cache helps quite a bit, but in other tests I've seen similar results with a 9207-8i that doesn't have WB cache when BTRFS filestores and SSD journals are used.
>>>
>>> Regarding the drives, they can do somewhere around 140-150MB/s large block writes with fio.
>>>
>>> Replication definitely adds additional latency so aggregate write throughput goes down, though it seems the penalty is worst after the first replica and doesn't hurt as much with subsequent ones.
>>>
>>> Mark
>>>
>>>
>>> On 04/25/2014 04:50 PM, Xing Lin wrote:
>>>> Hi Mark,
>>>>
>>>> That seems pretty good. What is the block level sequential read
>>>> bandwidth of your disks? What configuration did you use? What was the
>>>> replica size, read_ahead for your rbds and what were the number of
>>>> workloads you used? I used btrfs in my experiments as well.
>>>>
>>>> Thanks,
>>>> Xing
>>>>
>>>> On 04/25/2014 03:36 PM, Mark Nelson wrote:
>>>>> For what it's worth, I've been able to achieve up to around 120MB/s
>>>>> with btrfs before things fragment.
>>>>>
>>>>> Mark
>>>>>
>>>>> On 04/25/2014 03:59 PM, Xing wrote:
>>>>>> Hi Gregory,
>>>>>>
>>>>>> Thanks very much for your quick reply. When I started to look into
>>>>>> Ceph, Bobtail was the latest stable release and that was why I picked
>>>>>> that version and started to make a few modifications. I have not
>>>>>> ported my changes to 0.79 yet. The plan is if v-0.79 can provide a
>>>>>> higher disk bandwidth efficiency, I will switch to 0.79.
>>>>>> Unfortunately, that does not seem to be the case.
>>>>>>
>>>>>> The futex trace was done with version 0.79, not 0.59. I did a profile
>>>>>> in 0.59 too. There are some improvements, such as the introduction of
>>>>>> fd cache. But lots of futex calls are still there in v-0.79. I also
>>>>>> measured the maximum bandwidth from each disk we can get in Version
>>>>>> 0.79. It does not improve significantly: we can still only get 90~100
>>>>>> MB/s from each disk.
>>>>>>
>>>>>> Thanks,
>>>>>> Xing
>>>>>>
>>>>>>
>>>>>> On Apr 25, 2014, at 2:42 PM, Gregory Farnum <greg@inktank.com> wrote:
>>>>>>
>>>>>>> Bobtail is really too old to draw any meaningful conclusions from; why
>>>>>>> did you choose it?
>>>>>>>
>>>>>>> That's not to say that performance on current code will be better
>>>>>>> (though it very much might be), but the internal architecture has
>>>>>>> changed in some ways that will be particularly important for the futex
>>>>>>> profiling you did, and are probably important for these throughput
>>>>>>> results as well.
>>>>>>> -Greg
>>>>>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 25, 2014 at 1:38 PM, Xing <xinglin@cs.utah.edu> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I also did a few other experiments, trying to get what the maximum
>>>>>>>> bandwidth we can get from each data disk. The output is not
>>>>>>>> encouraging: for disks that can provide 150 MB/s block-level
>>>>>>>> sequential read bandwidths, we can only get about 90MB/s from each
>>>>>>>> disk. Something that is particular interesting is that the replica
>>>>>>>> size also affects the bandwidth we could get from the cluster. It
>>>>>>>> seems that there is no such observation/conversations in the Ceph
>>>>>>>> community and I think it may be helpful to share my findings.
>>>>>>>>
>>>>>>>> The experiment was run with two d820 machines in Emulab at
>>>>>>>> University of Utah. One is used as the data node and the other is
>>>>>>>> used as the client. They are connected by a 10 GB/s Ethernet. The
>>>>>>>> data node has 7 disks, one for OS and the rest 6 for OSDs. For the
>>>>>>>> rest 6 disks, we use one for journal and the other for data. Thus
>>>>>>>> in total we have 3 OSDs. The network bandwidth is sufficient to
>>>>>>>> support reading from 3 disks in full bandwidth.
>>>>>>>>
>>>>>>>> I varied the read-ahead size for the rbd block device (exp1), osd
>>>>>>>> op threads for each osd (exp2), varied the replica size (exp3), and
>>>>>>>> object size (exp4). The most interesting is varying the replica
>>>>>>>> size. As I varied replica size from 1, to 2 and to 3, the
>>>>>>>> aggregated bandwidth dropped from 267 MB/s to 211 and 180. The
>>>>>>>> reason for the drop I believe is as we increase the number of
>>>>>>>> replicas, we store more data into each OSD. then when we need to
>>>>>>>> read it back, we have to read from a larger range (more seeks). The
>>>>>>>> fundamental problem is likely because we are doing replication
>>>>>>>> synchronously, and thus layout object files in a raid 10 - near
>>>>>>>> format, rather than the far format. For the difference between the
>>>>>>>> near format and far format for raid 10, you could have a look at
>>>>>>>> the link provided below.
>>>>>>>>
>>>>>>>> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt
>>>>>>>>
>>>>>>>>
>>>>>>>> For results about other experiments, you could download my slides
>>>>>>>> at the link provided below.
>>>>>>>> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>>>>>>>>
>>>>>>>>
>>>>>>>> I do not know why Ceph only gets about 60% of the disk bandwidth.
>>>>>>>> To do a comparison, I ran tar to read every rbd object files to
>>>>>>>> create a tarball and see how much bandwidth I can get from this
>>>>>>>> workload. Interestingly, the tar workload actually gets a higher
>>>>>>>> bandwidth (80% of block level bandwidth), even though it is
>>>>>>>> accessing the disk more randomly (tar reads each object file in a
>>>>>>>> dir sequentially while the object files were created in a different
>>>>>>>> order.). For more detail, please goto my blog to have a read.
>>>>>>>> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html
>>>>>>>>
>>>>>>>>
>>>>>>>> Here are a few questions.
>>>>>>>> 1. What are the maximum bandwidth people can get from each disk? I
>>>>>>>> found Jiangang from Intel also reported 57% efficiency for disk
>>>>>>>> bandwidth. He suggested one reason: interference among so many
>>>>>>>> sequential read workloads. I agree but when I tried to run with one
>>>>>>>> single workload, I still do not get a higher efficiency.
>>>>>>>> 2. If the efficiency is about 60%, what are the reasons that cause
>>>>>>>> this? Could it be because of the locks (futex as I mentioned in my
>>>>>>>> previous email) or anything else?
>>>>>>>>
>>>>>>>> Thanks very much for any feedback.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Xing
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> ceph-devel" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: milosz@adfin.com