From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark Nelson Subject: Re: rados bench single instance vs. multiple instances Date: Wed, 13 May 2015 11:16:35 -0500 Message-ID: <555378E3.4000309@redhat.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Received: from mx1.redhat.com ([209.132.183.28]:39004 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965125AbbEMQuc (ORCPT ); Wed, 13 May 2015 12:50:32 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: "Deneau, Tom" , Sage Weil Cc: ceph-devel On 05/13/2015 10:05 AM, Deneau, Tom wrote: > > >> -----Original Message----- >> From: Sage Weil [mailto:sage@newdream.net] >> Sent: Monday, May 11, 2015 12:04 PM >> To: Deneau, Tom >> Cc: ceph-devel >> Subject: Re: rados bench single instance vs. multiple instances >> >> On Mon, 11 May 2015, Deneau, Tom wrote: >>> I have noticed the following while running rados bench seq read tes= ts >>> with a 40M object size >>> >>> single rados bench, 4 concurrent ops, band= width =3D >> 190 MB/s >>> 4 copies of rados bench, 1 concurrent op each, aggregate >>> bandwidth =3D 310 MB/s >>> >>> and in fact the single rados bench seems limited to 190, no matter = how many >> concurrent ops. >>> >>> I don't see this kind of behavior with a 4M object size. >>> >>> (The above are with caches dropped on the osd targets) >>> >>> It doesn't seem to be related to the total number of bytes being >>> processed by the single because if I don't drop the caches, both th= e >>> single rados bench and the 4-copy rados bench get much higher numbe= rs >>> (600 vs. 900) but still the single rados bench appears limited, no = matter >> how many concurrent ops are used. >>> >>> Is there kind of throttling going on by design here? >> >> It might be the librados throttles: >> >> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in= -flight >> data (both directions) >> OPTION(objecter_inflight_ops, OPT_U64, 1024) // max in= -flight >> ios >> >> IIRC these only affect librados.. which would include 'rados bench'. >> >> sage >> > > Just a follow-up that changing the limits of the two options mentione= d above > did indeed solve my problem. Yay! I suspect max bytes more so than max ops? Mark > > Also, my na=EFve understanding of the architecture was that things li= ke RBD and RGW were > layered on librados as shown in http://ceph.com/docs/master/architect= ure/. So wouldn't these > throttles apply to those stacks as well? > > -- Tom Deneau > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel"= in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html