Re: radosrgw performance problems

[Mark Nelson <mark.nelson@inktank.com>]

On 06/11/2013 08:27 AM, Jäger, Philipp wrote:
> Hello,
>
> we have a performance problem with radosrgw.
> Only 8mb/s-9 per upload, also tested with s3cmd on the rgw itself.
> (2 uploads at the same time: combined 15mb/s, 3 uploads at the same time: comb. 21mb/s)
> But when putting a file via rados rbd , we get 40mb/s upload, so no network or other problem in general.

One thing to check is to make sure that the rgw pool you are writing to 
has enough placement groups for your cluster.  The default may be 
extremely low.

>
> Same speed with the inktank apache/fastcgi and the original one. Hardware also fast enough. We use Ubuntu 12.04 lts, ceph 0.61.2
>
> So have you any idea why the rgw is so slow? How can we identify where the problem is?

RBD is pretty streamlined so you can get good performance with it.  On 
my test setup I'm seeing 80-90% of the performance of raw rados object 
writes/reads (and in some cases much faster with RBD cache enabled!). 
RGW, Apache, fastcgi, and simply the requirements of supporting the S3 
protocol itself add a lot of overhead.  MD5 calculations by themselves 
start chewing up a ton of CPU once you try to support high throughput 
scenarios and there is a non-trivial amount of extra latency added as 
well.  You may be able to improve things with some tweaks, but I 
wouldn't be surprised if RBD is always going to be faster to an extent.

For folks who want really fast object storage I think directly utilizing 
rados is probably the way to go, but that requires modifying the app and 
it's not for everyone.

>
> (I've heard something about the rgw admin socket to check perfcounters, but it seems that this is deprecated? Because when i type ceph --admin-daemon ... it says unknown command and I cannot find it in the ceph docu. Then i wanted to bench via rest-bench, but it says "ERROR: failed to create bucket: XmlParseFailure -failed initializing benchmark", so I could not bench the speed.)

connecting with the admin daemon should still be supported. 
Documentation is here:

http://ceph.com/docs/next/radosgw/troubleshooting/

If this doesn't work please let me know!

Also, I've created a bug for the rest-bench issue:

http://tracker.ceph.com/issues/5302

Personally I've been using swift-bench for most of my recent rgw testing.

Mark

>
> Ceph.conf- rgw part:
>
> [client.radosgw.connect2]
> host = hcrgwko2
> rgw socket path = /tmp/connect2.sock
> log file = /var/log/ceph/connect2.log
> rgw dns name =  FQDN
>
> Thank you very much.
>
>
> Regards
>
> Philipp
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html