v0.80.8 and librbd performance

v0.80.8 and librbd performance

[Sage Weil]

Hi,

This is just a heads up that we've identified a performance regression in 
v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way 
through QA and should be out in a few days.  If you haven't upgraded yet 
you may want to wait.

Thanks!
sage

Re: v0.80.8 and librbd performance

[Ken Dreyer]

On 03/03/2015 04:19 PM, Sage Weil wrote:
> Hi,
> 
> This is just a heads up that we've identified a performance regression in 
> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way 
> through QA and should be out in a few days.  If you haven't upgraded yet 
> you may want to wait.
> 
> Thanks!
> sage

Hi Sage,

I've seen a couple Redmine tickets on this (eg
http://tracker.ceph.com/issues/9854 ,
http://tracker.ceph.com/issues/10956). It's not totally clear to me
which of the 70+ unreleased commits on the firefly branch fix this
librbd issue.  Is it only the three commits in
https://github.com/ceph/ceph/pull/3410 , or are there more?

- Ken

Re: v0.80.8 and librbd performance

[Yuri Weinstein]

Ken

PLs se http://tracker.ceph.com/issues/10641 for more details

Thx
YuriW

----- Original Message -----
From: "Ken Dreyer" <kdreyer@redhat.com>
To: "Sage Weil" <sweil@redhat.com>, ceph-devel@vger.kernel.org, ceph-users@ceph.com
Sent: Tuesday, March 3, 2015 3:28:02 PM
Subject: Re: v0.80.8 and librbd performance

On 03/03/2015 04:19 PM, Sage Weil wrote:
> Hi,
> 
> This is just a heads up that we've identified a performance regression in 
> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way 
> through QA and should be out in a few days.  If you haven't upgraded yet 
> you may want to wait.
> 
> Thanks!
> sage

Hi Sage,

I've seen a couple Redmine tickets on this (eg
http://tracker.ceph.com/issues/9854 ,
http://tracker.ceph.com/issues/10956). It's not totally clear to me
which of the 70+ unreleased commits on the firefly branch fix this
librbd issue.  Is it only the three commits in
https://github.com/ceph/ceph/pull/3410 , or are there more?

- Ken
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: v0.80.8 and librbd performance

[Josh Durgin]

On 03/03/2015 03:28 PM, Ken Dreyer wrote:
> On 03/03/2015 04:19 PM, Sage Weil wrote:
>> Hi,
>>
>> This is just a heads up that we've identified a performance regression in
>> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way
>> through QA and should be out in a few days.  If you haven't upgraded yet
>> you may want to wait.
>>
>> Thanks!
>> sage
>
> Hi Sage,
>
> I've seen a couple Redmine tickets on this (eg
> http://tracker.ceph.com/issues/9854 ,
> http://tracker.ceph.com/issues/10956). It's not totally clear to me
> which of the 70+ unreleased commits on the firefly branch fix this
> librbd issue.  Is it only the three commits in
> https://github.com/ceph/ceph/pull/3410 , or are there more?

Those are the only ones needed to fix the librbd performance
regression, yes.

Josh

Re: v0.80.8 and librbd performance

[shiva rkreddy]

[-- Attachment #1.1: Type: text/plain, Size: 1509 bytes --]

Hi Josh,

We are using firefly 0.80.9 and see both cinder create/delete numbers slow
down compared 0.80.7.
I don't see any specific tuning requirements and our cluster is run pretty
much on default configuration.
Do you recommend any tuning or can you please suggest some log signatures
we need to be looking at?

Thanks
shiva

On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On 03/03/2015 03:28 PM, Ken Dreyer wrote:
>
>> On 03/03/2015 04:19 PM, Sage Weil wrote:
>>
>>> Hi,
>>>
>>> This is just a heads up that we've identified a performance regression in
>>> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way
>>> through QA and should be out in a few days.  If you haven't upgraded yet
>>> you may want to wait.
>>>
>>> Thanks!
>>> sage
>>>
>>
>> Hi Sage,
>>
>> I've seen a couple Redmine tickets on this (eg
>> http://tracker.ceph.com/issues/9854 ,
>> http://tracker.ceph.com/issues/10956). It's not totally clear to me
>> which of the 70+ unreleased commits on the firefly branch fix this
>> librbd issue.  Is it only the three commits in
>> https://github.com/ceph/ceph/pull/3410 , or are there more?
>>
>
> Those are the only ones needed to fix the librbd performance
> regression, yes.
>
> Josh
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

[-- Attachment #1.2: Type: text/html, Size: 2751 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.80.8 and librbd performance

[Olivier Bonvalet]

Does kernel client affected by the problem ?

Le mardi 03 mars 2015 à 15:19 -0800, Sage Weil a écrit :
> Hi,
> 
> This is just a heads up that we've identified a performance regression in 
> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way 
> through QA and should be out in a few days.  If you haven't upgraded yet 
> you may want to wait.
> 
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.80.8 and librbd performance

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 750 bytes --]

On Wed, 4 Mar 2015, Olivier Bonvalet wrote:
> Does kernel client affected by the problem ?

Nope.  The kernel client is unaffected.. the issue is in librbd.

sage


> 
> Le mardi 03 mars 2015 à 15:19 -0800, Sage Weil a écrit :
> > Hi,
> > 
> > This is just a heads up that we've identified a performance regression in 
> > v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way 
> > through QA and should be out in a few days.  If you haven't upgraded yet 
> > you may want to wait.
> > 
> > Thanks!
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
> 

Re: v0.80.8 and librbd performance

[Olivier Bonvalet]

Le mardi 03 mars 2015 à 16:32 -0800, Sage Weil a écrit :
> On Wed, 4 Mar 2015, Olivier Bonvalet wrote:
> > Does kernel client affected by the problem ?
> 
> Nope.  The kernel client is unaffected.. the issue is in librbd.
> 
> sage
> 


Ok, thanks for the clarification.
So I have to dig !


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: v0.80.8 and librbd performance

[Josh Durgin]

[-- Attachment #1.1: Type: text/plain, Size: 2119 bytes --]

I don't see any commits that would be likely to affect that between 0.80.7 and 0.80.9.

Is this after upgrading an existing cluster?
Could this be due to fs aging beneath your osds?

How are you measuring create/delete performance?

You can try increasing rbd concurrent management ops in ceph.conf on the cinder node. This affects delete speed, since rbd tries to delete each object in a volume.

Josh


From: shiva rkreddy <shiva.rkreddy@gmail.com>
Sent: Apr 14, 2015 5:53 AM
To: Josh Durgin
Cc: Ken Dreyer; Sage Weil; Ceph Development; ceph-users@ceph.com
Subject: Re: v0.80.8 and librbd performance

> Hi Josh,
>
> We are using firefly 0.80.9 and see both cinder create/delete numbers slow down compared 0.80.7.
> I don't see any specific tuning requirements and our cluster is run pretty much on default configuration.
> Do you recommend any tuning or can you please suggest some log signatures we need to be looking at?
>
> Thanks
> shiva
>
> On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin <jdurgin@redhat.com> wrote:
>>
>> On 03/03/2015 03:28 PM, Ken Dreyer wrote:
>>>
>>> On 03/03/2015 04:19 PM, Sage Weil wrote:
>>>>
>>>> Hi,
>>>>
>>>> This is just a heads up that we've identified a performance regression in
>>>> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way
>>>> through QA and should be out in a few days.  If you haven't upgraded yet
>>>> you may want to wait.
>>>>
>>>> Thanks!
>>>> sage
>>>
>>>
>>> Hi Sage,
>>>
>>> I've seen a couple Redmine tickets on this (eg
>>> http://tracker.ceph.com/issues/9854 ,
>>> http://tracker.ceph.com/issues/10956). It's not totally clear to me
>>> which of the 70+ unreleased commits on the firefly branch fix this
>>> librbd issue.  Is it only the three commits in
>>> https://github.com/ceph/ceph/pull/3410 , or are there more?
>>
>>
>> Those are the only ones needed to fix the librbd performance
>> regression, yes.
>>
>> Josh
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

[-- Attachment #1.2: Type: text/html, Size: 3826 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.80.8 and librbd performance

[shiva rkreddy]

[-- Attachment #1.1: Type: text/plain, Size: 2964 bytes --]

The clusters are in test environment, so its a new deployment of 0.80.9. OS
on the cluster nodes is reinstalled as well, so there shouldn't be any fs
aging unless the disks are slowing down.

The perf measurement is done initiating multiple cinder create/delete
commands and tracking the volume to be in available or completely gone from
"cinder list" output.

Even running  "rbd rm " command from cinder node results in similar
behaviour.

I'll try with  increasing  rbd_concurrent_management in ceph.conf.
 Is the param name rbd_concurrent_management or rbd-concurrent-management ?


On Tue, Apr 14, 2015 at 12:36 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> I don't see any commits that would be likely to affect that between 0.80.7
> and 0.80.9.
>
> Is this after upgrading an existing cluster?
> Could this be due to fs aging beneath your osds?
>
> How are you measuring create/delete performance?
>
> You can try increasing rbd concurrent management ops in ceph.conf on the
> cinder node. This affects delete speed, since rbd tries to delete each
> object in a volume.
>
> Josh
>
> *From:* shiva rkreddy <shiva.rkreddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> *Sent:* Apr 14, 2015 5:53 AM
> *To:* Josh Durgin
> *Cc:* Ken Dreyer; Sage Weil; Ceph Development; ceph-users-Qp0mS5GaXlQ@public.gmane.org
> *Subject:* Re: v0.80.8 and librbd performance
>
> Hi Josh,
>
> We are using firefly 0.80.9 and see both cinder create/delete numbers slow
> down compared 0.80.7.
> I don't see any specific tuning requirements and our cluster is run pretty
> much on default configuration.
> Do you recommend any tuning or can you please suggest some log signatures
> we need to be looking at?
>
> Thanks
> shiva
>
> On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
>> On 03/03/2015 03:28 PM, Ken Dreyer wrote:
>>
>>> On 03/03/2015 04:19 PM, Sage Weil wrote:
>>>
>>>> Hi,
>>>>
>>>> This is just a heads up that we've identified a performance regression
>>>> in
>>>> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way
>>>> through QA and should be out in a few days.  If you haven't upgraded yet
>>>> you may want to wait.
>>>>
>>>> Thanks!
>>>> sage
>>>>
>>>
>>> Hi Sage,
>>>
>>> I've seen a couple Redmine tickets on this (eg
>>> http://tracker.ceph.com/issues/9854 ,
>>> http://tracker.ceph.com/issues/10956). It's not totally clear to me
>>> which of the 70+ unreleased commits on the firefly branch fix this
>>> librbd issue.  Is it only the three commits in
>>> https://github.com/ceph/ceph/pull/3410 , or are there more?
>>>
>>
>> Those are the only ones needed to fix the librbd performance
>> regression, yes.
>>
>> Josh
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 5529 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.80.8 and librbd performance

[shiva rkreddy]

[-- Attachment #1.1: Type: text/plain, Size: 3660 bytes --]

Retried the test with by setting: rbd_concurrent_management_ops and
rbd-concurrent-management-ops to 20 (default 10?) and didn't see any
difference in the delete time.

Steps:
1. Create 20, 500GB volumes
2. run : rbd -n clientkey -p cindervols rbd rm $volumeId &
3. run rbd ls command in with 1 second sleep and capture output  : rbd -n
clientkey -p cindervols rbd ls

It took the same amount of time to remove all entries in the pool when the
ops setting was default.

Thanks

On Tue, Apr 14, 2015 at 10:01 PM, shiva rkreddy <shiva.rkreddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:

> The clusters are in test environment, so its a new deployment of 0.80.9.
> OS on the cluster nodes is reinstalled as well, so there shouldn't be any
> fs aging unless the disks are slowing down.
>
> The perf measurement is done initiating multiple cinder create/delete
> commands and tracking the volume to be in available or completely gone from
> "cinder list" output.
> Even running  "rbd rm " command from cinder node results in similar
> behaviour.
>
> I'll try with  increasing  rbd_concurrent_management in ceph.conf.
>  Is the param name rbd_concurrent_management or rbd-concurrent-management ?
>
>
> On Tue, Apr 14, 2015 at 12:36 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
>> I don't see any commits that would be likely to affect that between
>> 0.80.7 and 0.80.9.
>>
>> Is this after upgrading an existing cluster?
>> Could this be due to fs aging beneath your osds?
>>
>> How are you measuring create/delete performance?
>>
>> You can try increasing rbd concurrent management ops in ceph.conf on the
>> cinder node. This affects delete speed, since rbd tries to delete each
>> object in a volume.
>>
>> Josh
>>
>> *From:* shiva rkreddy <shiva.rkreddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>> *Sent:* Apr 14, 2015 5:53 AM
>> *To:* Josh Durgin
>> *Cc:* Ken Dreyer; Sage Weil; Ceph Development; ceph-users-Qp0mS5GaXlQ@public.gmane.org
>> *Subject:* Re: v0.80.8 and librbd performance
>>
>> Hi Josh,
>>
>> We are using firefly 0.80.9 and see both cinder create/delete numbers
>> slow down compared 0.80.7.
>> I don't see any specific tuning requirements and our cluster is run
>> pretty much on default configuration.
>> Do you recommend any tuning or can you please suggest some log signatures
>> we need to be looking at?
>>
>> Thanks
>> shiva
>>
>> On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>
>>> On 03/03/2015 03:28 PM, Ken Dreyer wrote:
>>>
>>>> On 03/03/2015 04:19 PM, Sage Weil wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> This is just a heads up that we've identified a performance regression
>>>>> in
>>>>> v0.80.8 from previous firefly releases.  A v0.80.9 is working it's way
>>>>> through QA and should be out in a few days.  If you haven't upgraded
>>>>> yet
>>>>> you may want to wait.
>>>>>
>>>>> Thanks!
>>>>> sage
>>>>>
>>>>
>>>> Hi Sage,
>>>>
>>>> I've seen a couple Redmine tickets on this (eg
>>>> http://tracker.ceph.com/issues/9854 ,
>>>> http://tracker.ceph.com/issues/10956). It's not totally clear to me
>>>> which of the 70+ unreleased commits on the firefly branch fix this
>>>> librbd issue.  Is it only the three commits in
>>>> https://github.com/ceph/ceph/pull/3410 , or are there more?
>>>>
>>>
>>> Those are the only ones needed to fix the librbd performance
>>> regression, yes.
>>>
>>> Josh
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 6606 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.80.8 and librbd performance

[Josh Durgin]

On 04/14/2015 08:01 PM, shiva rkreddy wrote:
> The clusters are in test environment, so its a new deployment of 0.80.9.
> OS on the cluster nodes is reinstalled as well, so there shouldn't be
> any fs aging unless the disks are slowing down.
>
> The perf measurement is done initiating multiple cinder create/delete
> commands and tracking the volume to be in available or completely gone
> from "cinder list" output.
>
> Even running  "rbd rm " command from cinder node results in similar
> behaviour.
>
> I'll try with  increasing  rbd_concurrent_management in ceph.conf.
>   Is the param name rbd_concurrent_management or rbd-concurrent-management ?

'rbd concurrent management ops' - spaces, hyphens, and underscores are
equivalent in ceph configuration.

A log with 'debug ms = 1' and 'debug rbd = 20' from 'rbd rm' on both 
versions might give clues about what's going slower.

Josh

> On Tue, Apr 14, 2015 at 12:36 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> <mailto:jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>> wrote:
>
>     I don't see any commits that would be likely to affect that between
>     0.80.7 and 0.80.9.
>
>     Is this after upgrading an existing cluster?
>     Could this be due to fs aging beneath your osds?
>
>     How are you measuring create/delete performance?
>
>     You can try increasing rbd concurrent management ops in ceph.conf on
>     the cinder node. This affects delete speed, since rbd tries to
>     delete each object in a volume.
>
>     Josh
>
>
>     *From:* shiva rkreddy <shiva.rkreddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
>     <mailto:shiva.rkreddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>>
>     *Sent:* Apr 14, 2015 5:53 AM
>     *To:* Josh Durgin
>     *Cc:* Ken Dreyer; Sage Weil; Ceph Development; ceph-users-Qp0mS5GaXlQ@public.gmane.org
>     <mailto:ceph-users-Qp0mS5GaXlQ@public.gmane.org>
>     *Subject:* Re: v0.80.8 and librbd performance
>
>         Hi Josh,
>
>         We are using firefly 0.80.9 and see both cinder create/delete
>         numbers slow down compared 0.80.7.
>         I don't see any specific tuning requirements and our cluster is
>         run pretty much on default configuration.
>         Do you recommend any tuning or can you please suggest some log
>         signatures we need to be looking at?
>
>         Thanks
>         shiva
>
>         On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin <jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
>         <mailto:jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>> wrote:
>
>             On 03/03/2015 03:28 PM, Ken Dreyer wrote:
>
>                 On 03/03/2015 04:19 PM, Sage Weil wrote:
>
>                     Hi,
>
>                     This is just a heads up that we've identified a
>                     performance regression in
>                     v0.80.8 from previous firefly releases.  A v0.80.9
>                     is working it's way
>                     through QA and should be out in a few days.  If you
>                     haven't upgraded yet
>                     you may want to wait.
>
>                     Thanks!
>                     sage
>
>
>                 Hi Sage,
>
>                 I've seen a couple Redmine tickets on this (eg
>                 http://tracker.ceph.com/__issues/9854
>                 <http://tracker.ceph.com/issues/9854> ,
>                 http://tracker.ceph.com/__issues/10956
>                 <http://tracker.ceph.com/issues/10956>). It's not
>                 totally clear to me
>                 which of the 70+ unreleased commits on the firefly
>                 branch fix this
>                 librbd issue.  Is it only the three commits in
>                 https://github.com/ceph/ceph/__pull/3410
>                 <https://github.com/ceph/ceph/pull/3410> , or are there
>                 more?
>
>
>             Those are the only ones needed to fix the librbd performance
>             regression, yes.
>
>             Josh
>
>             --
>             To unsubscribe from this list: send the line "unsubscribe
>             ceph-devel" in
>             the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>             <mailto:majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
>             More majordomo info at
>             http://vger.kernel.org/__majordomo-info.html
>             <http://vger.kernel.org/majordomo-info.html>
>
>
>