Re: Problems when benchmarking Ceph

[Gregory Farnum <greg@inktank.com>]

The library you're looking for is libceph. It does exist and it's fairly full-featured, but it's not nearly as well documented as the librados C api is. 
However, you'll probably get more use out of one of the Hadoop bindings. If you check out the git repository you'll find one set in src/client/hadoop, with limited instructions, that I believe currently apply to the .20/1.0.x branch. Or you might look at Noah's ongoing work on a much cleaner set of bindings at https://github.com/noahdesu/ceph/tree/wip-java-cephfs

I dunno about the MDS performance, though; there are too many possibilities with that. Maybe try out these options and see how they do first?
-Greg


On Wednesday, May 30, 2012 at 4:15 AM, Nam Dang wrote:

> Hi,
> 
> i just figured out one problem in my benchmark: all the concurrent
> threads use the same file layer provided by the OS - so probably this
> can be a bottleneck when the number of threads increases.
> I wonder if I can connect directly to the MDS and access the the
> underlying file system through some library? Sorry for my inexperience
> but I haven't found any mentioning of IO operations for files in the
> API. Did I miss something?
> 
> Best regards,
> 
> Nam Dang
> Email: namd@de.cs.titech.ac.jp (mailto:namd@de.cs.titech.ac.jp)
> Tokyo Institute of Technology
> Tokyo, Japan
> 
> 
> On Wed, May 30, 2012 at 7:28 PM, Nam Dang <namd@de.cs.titech.ac.jp (mailto:namd@de.cs.titech.ac.jp)> wrote:
> > Dear all,
> > 
> > I am using Ceph as a baseline for Hadoop. In Hadoop there is a
> > NNThroughputBenchmark, which tries to test the upper limit of the
> > namenode (a.k.a MDS in Ceph).
> > This NNThroughputBenchmark basically creates a master node, and
> > creates many threads that sends requests to the master node as
> > possible. This approach minimizes communication overhead when
> > employing actual clients. The code can be found here:
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.19/src/test/org/apache/hadoop/hdfs/NNThroughputBenchmark.java
> > 
> > So I'm testing Ceph in similar manner:
> > - Mount Ceph to a folder
> > - Create many threads and send requests to the MDS on the MDS (so no
> > network communication)
> > - I do not write any data to the files: just mere file creation.
> > 
> > However, I notice very poor performance on Ceph (only about 485
> > ops/sec, as oppose to 8000ops/sec in Hadoop, and I'm not sure why. I
> > also notice that when I tried to remove the folder created by the
> > interrupted test of the benchmark mentioned above, I took too long I
> > had to Ctrl+Break out of the rm program. I'm thinking that the reason
> > could be that I'm using Java IO instead of Ceph direct data
> > manipulation code. Also, I didn't write any data so there shouldn't be
> > any overhead of communicating with the OSDs (or is my assumption
> > wrong?)
> > 
> > So do you have any idea on this?
> > 
> > My configuration at the moment:
> > - Ceph 0.47.1
> > - Intel Xeon 5 2.4Ghz, 4x2 cores
> > - 24GB of RAM
> > - One node for Monitor, One for MDS, 5 for OSD (of the same configuration)
> > - I mount Ceph to a folder on the MDS and run the simulation on that
> > folder (creating, opening, deleting files) - Right now I'm just
> > working on creating files so I haven't tested with others.
> > 
> > And I'm wondering if there is anyway I can use the API to manipulate
> > the file system directly instead of mounting through the OS and use
> > the OS's basic file manipulation layer.
> > I checked the API doc at http://ceph.com/docs/master/api/librados/ and
> > it appears that there is no clear way of accessing the Ceph's file
> > system directly, only object-based storage system.
> > 
> > Thank you very much for your help!
> > 
> > Below is the configuration of my Ceph installation:
> > 
> > ; disable authentication
> > [mon]
> > mon data = /home/namd/ceph/mon
> > 
> > [osd]
> > osd data = /home/namd/ceph/osd
> > osd journal = /home/namd/ceph/osd.journal
> > osd journal size = 1000
> > osd min rep = 3
> > osd max rep = 3
> > ; the following line is for ext4 partition
> > filestore xattr use omap = true
> > 
> > [mds.1]
> > host=sakura09
> > 
> > [mds.2]
> > host=sakura10
> > 
> > [mds.3]
> > host=sakura11
> > 
> > [mds.4]
> > host=sakura12
> > 
> > [mon.0]
> > host=sakura08
> > mon addr=192.168.172.178:6789
> > 
> > [osd.0]
> > host=sakura13
> > 
> > [osd.1]
> > host=sakura14
> > 
> > [osd.2]
> > host=sakura15
> > 
> > [osd.3]
> > host=sakura16
> > 
> > [osd.4]
> > host=sakura17
> > 
> > [osd.5]
> > host=sakura18
> > 
> > 
> > 
> > Best regards,
> > 
> > Nam Dang
> > Email: namd@de.cs.titech.ac.jp (mailto:namd@de.cs.titech.ac.jp)
> > Tokyo Institute of Technology
> > Tokyo, Japan
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo-info.html