rados bench single instance vs. multiple instances

rados bench single instance vs. multiple instances

[Deneau, Tom]

I have noticed the following while running rados bench seq read tests with a 40M object size

    single rados bench, 4 concurrent ops,                     bandwidth = 190 MB/s
    4 copies of rados bench, 1 concurrent op each,  aggregate bandwidth = 310 MB/s

and in fact the single rados bench seems limited to 190, no matter how many concurrent ops.

I don't see this kind of behavior with a 4M object size.

(The above are with caches dropped on the osd targets)

It doesn't seem to be related to the total number of bytes being processed by the single 
because if I don't drop the caches, both the single rados bench and the 4-copy rados bench
get much higher numbers (600 vs. 900) but still the single rados bench appears limited, no matter
how many concurrent ops are used.

Is there kind of throttling going on by design here?

-- Tom Deneau, AMD

Re: rados bench single instance vs. multiple instances

[Mark Nelson]

On 05/11/2015 09:18 AM, Deneau, Tom wrote:
> I have noticed the following while running rados bench seq read tests with a 40M object size
>
>      single rados bench, 4 concurrent ops,                     bandwidth = 190 MB/s
>      4 copies of rados bench, 1 concurrent op each,  aggregate bandwidth = 310 MB/s
>
> and in fact the single rados bench seems limited to 190, no matter how many concurrent ops.

What's your CPU utilization like?

>
> I don't see this kind of behavior with a 4M object size.
>
> (The above are with caches dropped on the osd targets)
>
> It doesn't seem to be related to the total number of bytes being processed by the single
> because if I don't drop the caches, both the single rados bench and the 4-copy rados bench
> get much higher numbers (600 vs. 900) but still the single rados bench appears limited, no matter
> how many concurrent ops are used.
>
> Is there kind of throttling going on by design here?

Nope, probably just client side inefficiency, especially if you are 
running on slower cores.  It's one of the reasons we tooled support for 
multiple concurrent copies of rados bench into cbt.

Mark

>
> -- Tom Deneau, AMD
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: rados bench single instance vs. multiple instances

[Deneau, Tom]

It's a beefy client with 32 cores.  Less than 1% of the total CPU used on the
client side. in the single rados case.  And on the OSD node side, only 25% of
the cpu being used, and disks and network are not at their limits

-- Tom


> -----Original Message-----
> From: Mark Nelson [mailto:mnelson@redhat.com]
> Sent: Monday, May 11, 2015 9:23 AM
> To: Deneau, Tom; ceph-devel
> Subject: Re: rados bench single instance vs. multiple instances
> 
> 
> 
> On 05/11/2015 09:18 AM, Deneau, Tom wrote:
> > I have noticed the following while running rados bench seq read tests
> > with a 40M object size
> >
> >      single rados bench, 4 concurrent ops,                     bandwidth =
> 190 MB/s
> >      4 copies of rados bench, 1 concurrent op each,  aggregate
> > bandwidth = 310 MB/s
> >
> > and in fact the single rados bench seems limited to 190, no matter how many
> concurrent ops.
> 
> What's your CPU utilization like?
> 
> >
> > I don't see this kind of behavior with a 4M object size.
> >
> > (The above are with caches dropped on the osd targets)
> >
> > It doesn't seem to be related to the total number of bytes being
> > processed by the single because if I don't drop the caches, both the
> > single rados bench and the 4-copy rados bench get much higher numbers
> > (600 vs. 900) but still the single rados bench appears limited, no matter
> how many concurrent ops are used.
> >
> > Is there kind of throttling going on by design here?
> 
> Nope, probably just client side inefficiency, especially if you are running
> on slower cores.  It's one of the reasons we tooled support for multiple
> concurrent copies of rados bench into cbt.
> 
> Mark
> 
> >
> > -- Tom Deneau, AMD
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> >

Re: rados bench single instance vs. multiple instances

[Sage Weil]

On Mon, 11 May 2015, Deneau, Tom wrote:
> I have noticed the following while running rados bench seq read tests with a 40M object size
> 
>     single rados bench, 4 concurrent ops,                     bandwidth = 190 MB/s
>     4 copies of rados bench, 1 concurrent op each,  aggregate bandwidth = 310 MB/s
> 
> and in fact the single rados bench seems limited to 190, no matter how many concurrent ops.
> 
> I don't see this kind of behavior with a 4M object size.
> 
> (The above are with caches dropped on the osd targets)
> 
> It doesn't seem to be related to the total number of bytes being processed by the single 
> because if I don't drop the caches, both the single rados bench and the 4-copy rados bench
> get much higher numbers (600 vs. 900) but still the single rados bench appears limited, no matter
> how many concurrent ops are used.
> 
> Is there kind of throttling going on by design here?

It might be the librados throttles:

OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight data (both directions)
OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight ios

IIRC these only affect librados.. which would include 'rados bench'.

sage

Re: rados bench single instance vs. multiple instances

[Mark Nelson]

On 05/11/2015 12:03 PM, Sage Weil wrote:
> On Mon, 11 May 2015, Deneau, Tom wrote:
>> I have noticed the following while running rados bench seq read tests with a 40M object size
>>
>>      single rados bench, 4 concurrent ops,                     bandwidth = 190 MB/s
>>      4 copies of rados bench, 1 concurrent op each,  aggregate bandwidth = 310 MB/s
>>
>> and in fact the single rados bench seems limited to 190, no matter how many concurrent ops.
>>
>> I don't see this kind of behavior with a 4M object size.
>>
>> (The above are with caches dropped on the osd targets)
>>
>> It doesn't seem to be related to the total number of bytes being processed by the single
>> because if I don't drop the caches, both the single rados bench and the 4-copy rados bench
>> get much higher numbers (600 vs. 900) but still the single rados bench appears limited, no matter
>> how many concurrent ops are used.
>>
>> Is there kind of throttling going on by design here?
>
> It might be the librados throttles:
>
> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight data (both directions)
> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight ios
>
> IIRC these only affect librados.. which would include 'rados bench'.

That's a very good point.  I missed in my first scan that this was 40MB 
objects.  Increasing the objecter_inflight_op_bytes would be an 
interesting experiment.

>
> sage
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: rados bench single instance vs. multiple instances

[Deneau, Tom]

> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Monday, May 11, 2015 12:04 PM
> To: Deneau, Tom
> Cc: ceph-devel
> Subject: Re: rados bench single instance vs. multiple instances
> 
> On Mon, 11 May 2015, Deneau, Tom wrote:
> > I have noticed the following while running rados bench seq read tests
> > with a 40M object size
> >
> >     single rados bench, 4 concurrent ops,                     bandwidth =
> 190 MB/s
> >     4 copies of rados bench, 1 concurrent op each,  aggregate
> > bandwidth = 310 MB/s
> >
> > and in fact the single rados bench seems limited to 190, no matter how many
> concurrent ops.
> >
> > I don't see this kind of behavior with a 4M object size.
> >
> > (The above are with caches dropped on the osd targets)
> >
> > It doesn't seem to be related to the total number of bytes being
> > processed by the single because if I don't drop the caches, both the
> > single rados bench and the 4-copy rados bench get much higher numbers
> > (600 vs. 900) but still the single rados bench appears limited, no matter
> how many concurrent ops are used.
> >
> > Is there kind of throttling going on by design here?
> 
> It might be the librados throttles:
> 
> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight
> data (both directions)
> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight
> ios
> 
> IIRC these only affect librados.. which would include 'rados bench'.
> 
> sage
> 

I noticed those, and tried setting them higher but it didn't seem to have an effect.
But maybe I didn't change it correctly.

What I did:
   * stop all osds
   * add the following lines to ceph.conf
     [osd]
        objecter_inflight_op_bytes = 1048576000
   * restart all osds
   * admin-daemon osd config show shows the new value
   * made sure /etc/ceph/ceph.conf on the client also had those lines added
      (although I was not sure if that was needed?)

-- Tom

Re: rados bench single instance vs. multiple instances

[Gregory Farnum]

On Mon, May 11, 2015 at 10:15 AM, Deneau, Tom <tom.deneau@amd.com> wrote:
>
>
>> -----Original Message-----
>> From: Sage Weil [mailto:sage@newdream.net]
>> Sent: Monday, May 11, 2015 12:04 PM
>> To: Deneau, Tom
>> Cc: ceph-devel
>> Subject: Re: rados bench single instance vs. multiple instances
>>
>> On Mon, 11 May 2015, Deneau, Tom wrote:
>> > I have noticed the following while running rados bench seq read tests
>> > with a 40M object size
>> >
>> >     single rados bench, 4 concurrent ops,                     bandwidth =
>> 190 MB/s
>> >     4 copies of rados bench, 1 concurrent op each,  aggregate
>> > bandwidth = 310 MB/s
>> >
>> > and in fact the single rados bench seems limited to 190, no matter how many
>> concurrent ops.
>> >
>> > I don't see this kind of behavior with a 4M object size.
>> >
>> > (The above are with caches dropped on the osd targets)
>> >
>> > It doesn't seem to be related to the total number of bytes being
>> > processed by the single because if I don't drop the caches, both the
>> > single rados bench and the 4-copy rados bench get much higher numbers
>> > (600 vs. 900) but still the single rados bench appears limited, no matter
>> how many concurrent ops are used.
>> >
>> > Is there kind of throttling going on by design here?
>>
>> It might be the librados throttles:
>>
>> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight
>> data (both directions)
>> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight
>> ios
>>
>> IIRC these only affect librados.. which would include 'rados bench'.
>>
>> sage
>>
>
> I noticed those, and tried setting them higher but it didn't seem to have an effect.
> But maybe I didn't change it correctly.
>
> What I did:
>    * stop all osds
>    * add the following lines to ceph.conf
>      [osd]
>         objecter_inflight_op_bytes = 1048576000

This is a client-side configuration, so putting it in the OSD section
won't have any effect.

Do note that even without this, having 1x4 requests outstanding is not
quite the same as having 4x1 requests outstanding — a single client
only gets one thread to communicate with each OSD, so the cluster has
a bit less parallelism when outstanding requests map to the same OSD.
But I don't think that should be too obvious at this point.
-Greg

>    * restart all osds
>    * admin-daemon osd config show shows the new value
>    * made sure /etc/ceph/ceph.conf on the client also had those lines added
>       (although I was not sure if that was needed?)
>
> -- Tom
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rados bench single instance vs. multiple instances

[Deneau, Tom]

> -----Original Message-----
> From: Gregory Farnum [mailto:greg@gregs42.com]
> Sent: Monday, May 11, 2015 12:18 PM
> To: Deneau, Tom
> Cc: Sage Weil; ceph-devel
> Subject: Re: rados bench single instance vs. multiple instances
> 
> On Mon, May 11, 2015 at 10:15 AM, Deneau, Tom <tom.deneau@amd.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Sage Weil [mailto:sage@newdream.net]
> >> Sent: Monday, May 11, 2015 12:04 PM
> >> To: Deneau, Tom
> >> Cc: ceph-devel
> >> Subject: Re: rados bench single instance vs. multiple instances
> >>
> >> On Mon, 11 May 2015, Deneau, Tom wrote:
> >> > I have noticed the following while running rados bench seq read
> >> > tests with a 40M object size
> >> >
> >> >     single rados bench, 4 concurrent ops,                     bandwidth
> =
> >> 190 MB/s
> >> >     4 copies of rados bench, 1 concurrent op each,  aggregate
> >> > bandwidth = 310 MB/s
> >> >
> >> > and in fact the single rados bench seems limited to 190, no matter
> >> > how many
> >> concurrent ops.
> >> >
> >> > I don't see this kind of behavior with a 4M object size.
> >> >
> >> > (The above are with caches dropped on the osd targets)
> >> >
> >> > It doesn't seem to be related to the total number of bytes being
> >> > processed by the single because if I don't drop the caches, both
> >> > the single rados bench and the 4-copy rados bench get much higher
> >> > numbers
> >> > (600 vs. 900) but still the single rados bench appears limited, no
> >> > matter
> >> how many concurrent ops are used.
> >> >
> >> > Is there kind of throttling going on by design here?
> >>
> >> It might be the librados throttles:
> >>
> >> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max
> >> in-flight data (both directions)
> >> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-
> flight
> >> ios
> >>
> >> IIRC these only affect librados.. which would include 'rados bench'.
> >>
> >> sage
> >>
> >
> > I noticed those, and tried setting them higher but it didn't seem to have
> an effect.
> > But maybe I didn't change it correctly.
> >
> > What I did:
> >    * stop all osds
> >    * add the following lines to ceph.conf
> >      [osd]
> >         objecter_inflight_op_bytes = 1048576000
> 
> This is a client-side configuration, so putting it in the OSD section won't
> have any effect.
> 
> Do note that even without this, having 1x4 requests outstanding is not quite
> the same as having 4x1 requests outstanding — a single client only gets one
> thread to communicate with each OSD, so the cluster has a bit less
> parallelism when outstanding requests map to the same OSD.
> But I don't think that should be too obvious at this point.
> -Greg
> 

OK, on the client side, I just put it in the [global] section.
That seemed to do the trick.

(I assume nothing else needs to be done for the client side).

What was the reason for this limit?
Should I expect any undesirable side effects from changing this value?

-- Tom


> >    * restart all osds
> >    * admin-daemon osd config show shows the new value
> >    * made sure /etc/ceph/ceph.conf on the client also had those lines added
> >       (although I was not sure if that was needed?)
> >
> > -- Tom
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html

Re: rados bench single instance vs. multiple instances

[Gregory Farnum]

On Mon, May 11, 2015 at 10:25 AM, Deneau, Tom <tom.deneau@amd.com> wrote:
>
>
>> -----Original Message-----
>> From: Gregory Farnum [mailto:greg@gregs42.com]
>> Sent: Monday, May 11, 2015 12:18 PM
>> To: Deneau, Tom
>> Cc: Sage Weil; ceph-devel
>> Subject: Re: rados bench single instance vs. multiple instances
>>
>> On Mon, May 11, 2015 at 10:15 AM, Deneau, Tom <tom.deneau@amd.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Sage Weil [mailto:sage@newdream.net]
>> >> Sent: Monday, May 11, 2015 12:04 PM
>> >> To: Deneau, Tom
>> >> Cc: ceph-devel
>> >> Subject: Re: rados bench single instance vs. multiple instances
>> >>
>> >> On Mon, 11 May 2015, Deneau, Tom wrote:
>> >> > I have noticed the following while running rados bench seq read
>> >> > tests with a 40M object size
>> >> >
>> >> >     single rados bench, 4 concurrent ops,                     bandwidth
>> =
>> >> 190 MB/s
>> >> >     4 copies of rados bench, 1 concurrent op each,  aggregate
>> >> > bandwidth = 310 MB/s
>> >> >
>> >> > and in fact the single rados bench seems limited to 190, no matter
>> >> > how many
>> >> concurrent ops.
>> >> >
>> >> > I don't see this kind of behavior with a 4M object size.
>> >> >
>> >> > (The above are with caches dropped on the osd targets)
>> >> >
>> >> > It doesn't seem to be related to the total number of bytes being
>> >> > processed by the single because if I don't drop the caches, both
>> >> > the single rados bench and the 4-copy rados bench get much higher
>> >> > numbers
>> >> > (600 vs. 900) but still the single rados bench appears limited, no
>> >> > matter
>> >> how many concurrent ops are used.
>> >> >
>> >> > Is there kind of throttling going on by design here?
>> >>
>> >> It might be the librados throttles:
>> >>
>> >> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max
>> >> in-flight data (both directions)
>> >> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-
>> flight
>> >> ios
>> >>
>> >> IIRC these only affect librados.. which would include 'rados bench'.
>> >>
>> >> sage
>> >>
>> >
>> > I noticed those, and tried setting them higher but it didn't seem to have
>> an effect.
>> > But maybe I didn't change it correctly.
>> >
>> > What I did:
>> >    * stop all osds
>> >    * add the following lines to ceph.conf
>> >      [osd]
>> >         objecter_inflight_op_bytes = 1048576000
>>
>> This is a client-side configuration, so putting it in the OSD section won't
>> have any effect.
>>
>> Do note that even without this, having 1x4 requests outstanding is not quite
>> the same as having 4x1 requests outstanding — a single client only gets one
>> thread to communicate with each OSD, so the cluster has a bit less
>> parallelism when outstanding requests map to the same OSD.
>> But I don't think that should be too obvious at this point.
>> -Greg
>>
>
> OK, on the client side, I just put it in the [global] section.
> That seemed to do the trick.
>
> (I assume nothing else needs to be done for the client side).
>
> What was the reason for this limit?
> Should I expect any undesirable side effects from changing this value?

The limit is there to prevent undue memory usage by the Ceph clients;
it's just a feedback throttle for anybody putting in data. If you've
got the memory to allow increased usage, there's no downside.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rados bench single instance vs. multiple instances

[Deneau, Tom]

> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Monday, May 11, 2015 12:04 PM
> To: Deneau, Tom
> Cc: ceph-devel
> Subject: Re: rados bench single instance vs. multiple instances
> 
> On Mon, 11 May 2015, Deneau, Tom wrote:
> > I have noticed the following while running rados bench seq read tests
> > with a 40M object size
> >
> >     single rados bench, 4 concurrent ops,                     bandwidth =
> 190 MB/s
> >     4 copies of rados bench, 1 concurrent op each,  aggregate
> > bandwidth = 310 MB/s
> >
> > and in fact the single rados bench seems limited to 190, no matter how many
> concurrent ops.
> >
> > I don't see this kind of behavior with a 4M object size.
> >
> > (The above are with caches dropped on the osd targets)
> >
> > It doesn't seem to be related to the total number of bytes being
> > processed by the single because if I don't drop the caches, both the
> > single rados bench and the 4-copy rados bench get much higher numbers
> > (600 vs. 900) but still the single rados bench appears limited, no matter
> how many concurrent ops are used.
> >
> > Is there kind of throttling going on by design here?
> 
> It might be the librados throttles:
> 
> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight
> data (both directions)
> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight
> ios
> 
> IIRC these only affect librados.. which would include 'rados bench'.
> 
> sage
> 

Just a follow-up that changing the limits of the two options mentioned above
did indeed solve my problem.

Also, my naïve understanding of the architecture was that things like RBD and RGW were
layered on librados as shown in http://ceph.com/docs/master/architecture/.  So wouldn't these
throttles apply to those stacks as well?

-- Tom Deneau


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rados bench single instance vs. multiple instances

[Mark Nelson]

On 05/13/2015 10:05 AM, Deneau, Tom wrote:
>
>
>> -----Original Message-----
>> From: Sage Weil [mailto:sage@newdream.net]
>> Sent: Monday, May 11, 2015 12:04 PM
>> To: Deneau, Tom
>> Cc: ceph-devel
>> Subject: Re: rados bench single instance vs. multiple instances
>>
>> On Mon, 11 May 2015, Deneau, Tom wrote:
>>> I have noticed the following while running rados bench seq read tests
>>> with a 40M object size
>>>
>>>      single rados bench, 4 concurrent ops,                     bandwidth =
>> 190 MB/s
>>>      4 copies of rados bench, 1 concurrent op each,  aggregate
>>> bandwidth = 310 MB/s
>>>
>>> and in fact the single rados bench seems limited to 190, no matter how many
>> concurrent ops.
>>>
>>> I don't see this kind of behavior with a 4M object size.
>>>
>>> (The above are with caches dropped on the osd targets)
>>>
>>> It doesn't seem to be related to the total number of bytes being
>>> processed by the single because if I don't drop the caches, both the
>>> single rados bench and the 4-copy rados bench get much higher numbers
>>> (600 vs. 900) but still the single rados bench appears limited, no matter
>> how many concurrent ops are used.
>>>
>>> Is there kind of throttling going on by design here?
>>
>> It might be the librados throttles:
>>
>> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight
>> data (both directions)
>> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight
>> ios
>>
>> IIRC these only affect librados.. which would include 'rados bench'.
>>
>> sage
>>
>
> Just a follow-up that changing the limits of the two options mentioned above
> did indeed solve my problem.

Yay!  I suspect max bytes more so than max ops?


Mark

>
> Also, my naïve understanding of the architecture was that things like RBD and RGW were
> layered on librados as shown in http://ceph.com/docs/master/architecture/.  So wouldn't these
> throttles apply to those stacks as well?
>
> -- Tom Deneau
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rados bench single instance vs. multiple instances

[Deneau, Tom]

> -----Original Message-----
> From: Mark Nelson [mailto:mnelson@redhat.com]
> Sent: Wednesday, May 13, 2015 11:17 AM
> To: Deneau, Tom; Sage Weil
> Cc: ceph-devel
> Subject: Re: rados bench single instance vs. multiple instances
> 
> On 05/13/2015 10:05 AM, Deneau, Tom wrote:
> >
> >
> >> -----Original Message-----
> >> From: Sage Weil [mailto:sage@newdream.net]
> >> Sent: Monday, May 11, 2015 12:04 PM
> >> To: Deneau, Tom
> >> Cc: ceph-devel
> >> Subject: Re: rados bench single instance vs. multiple instances
> >>
> >> On Mon, 11 May 2015, Deneau, Tom wrote:
> >>> I have noticed the following while running rados bench seq read
> >>> tests with a 40M object size
> >>>
> >>>      single rados bench, 4 concurrent ops,                     bandwidth
> =
> >> 190 MB/s
> >>>      4 copies of rados bench, 1 concurrent op each,  aggregate
> >>> bandwidth = 310 MB/s
> >>>
> >>> and in fact the single rados bench seems limited to 190, no matter
> >>> how many
> >> concurrent ops.
> >>>
> >>> I don't see this kind of behavior with a 4M object size.
> >>>
> >>> (The above are with caches dropped on the osd targets)
> >>>
> >>> It doesn't seem to be related to the total number of bytes being
> >>> processed by the single because if I don't drop the caches, both the
> >>> single rados bench and the 4-copy rados bench get much higher
> >>> numbers
> >>> (600 vs. 900) but still the single rados bench appears limited, no
> >>> matter
> >> how many concurrent ops are used.
> >>>
> >>> Is there kind of throttling going on by design here?
> >>
> >> It might be the librados throttles:
> >>
> >> OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max
> >> in-flight data (both directions)
> >> OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-
> flight
> >> ios
> >>
> >> IIRC these only affect librados.. which would include 'rados bench'.
> >>
> >> sage
> >>
> >
> > Just a follow-up that changing the limits of the two options mentioned
> > above did indeed solve my problem.
> 
> Yay!  I suspect max bytes more so than max ops?
> 
> 
> Mark
> 

Yes, we were really only hitting the limit of inflight bytes.
(rather easy to do since the object size I was trying was 40M).

Another question: 

I had noticed previously that rados bench (since it does its writes as a single request
for the entire length of the object) is limited by osd_max_write_size to objects of 90MB or less.

If I wanted to experiment with larger object sizes, should I expect any negative
side effects from increasing osd_max_write_size?

-- Tom

> >
> > Also, my naïve understanding of the architecture was that things like
> > RBD and RGW were layered on librados as shown in
> > http://ceph.com/docs/master/architecture/.  So wouldn't these throttles
> apply to those stacks as well?
> >
> > -- Tom Deneau
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> >
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rados bench single instance vs. multiple instances

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2030 bytes --]

On Wed, 13 May 2015, Deneau, Tom wrote:
> 
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Monday, May 11, 2015 12:04 PM
> > To: Deneau, Tom
> > Cc: ceph-devel
> > Subject: Re: rados bench single instance vs. multiple instances
> > 
> > On Mon, 11 May 2015, Deneau, Tom wrote:
> > > I have noticed the following while running rados bench seq read tests
> > > with a 40M object size
> > >
> > >     single rados bench, 4 concurrent ops,                     bandwidth =
> > 190 MB/s
> > >     4 copies of rados bench, 1 concurrent op each,  aggregate
> > > bandwidth = 310 MB/s
> > >
> > > and in fact the single rados bench seems limited to 190, no matter how many
> > concurrent ops.
> > >
> > > I don't see this kind of behavior with a 4M object size.
> > >
> > > (The above are with caches dropped on the osd targets)
> > >
> > > It doesn't seem to be related to the total number of bytes being
> > > processed by the single because if I don't drop the caches, both the
> > > single rados bench and the 4-copy rados bench get much higher numbers
> > > (600 vs. 900) but still the single rados bench appears limited, no matter
> > how many concurrent ops are used.
> > >
> > > Is there kind of throttling going on by design here?
> > 
> > It might be the librados throttles:
> > 
> > OPTION(objecter_inflight_op_bytes, OPT_U64, 1024*1024*100) // max in-flight
> > data (both directions)
> > OPTION(objecter_inflight_ops, OPT_U64, 1024)               // max in-flight
> > ios
> > 
> > IIRC these only affect librados.. which would include 'rados bench'.
> > 
> > sage
> > 
> 
> Just a follow-up that changing the limits of the two options mentioned above
> did indeed solve my problem.

Great!

> Also, my naïve understanding of the architecture was that things like RBD and RGW were
> layered on librados as shown in http://ceph.com/docs/master/architecture/.  So wouldn't these
> throttles apply to those stacks as well?

Yep!

sage