Re: mon memory leak

[Joao Eduardo Luis <joao.luis@inktank.com>]

On 03/13/2013 01:42 PM, Travis Rhoden wrote:
> Hi Stefan,
>
> I have seen a mon grow quite high on Bobtail.  Joao had some theories
> as for why, and I was able to provide him with a memory dump of the
> running process.  No word yet on whether it revealed anything, but I
> know it is on his stack.  Background here:
> http://thread.gmane.org/gmane.comp.file-systems.ceph.user/14
>
>   - Travis

Hi all,

As Travis points out, this has been seen before.  I've spent a fair 
amount of time chasing this down now, but haven't got to any useful 
conclusions yet.  Although this is concerning, chasing down this kind of 
behaviour tends to be a time sink and there are other more pressing 
issues that have been requiring most of my attention.  This hasn't been 
forgotten and I've been allocating my time to it as possible.

On Sébastien's issue, that may very well be caused by some of the 
memleaks that have been fixed some time since argonaut, prior to bobtail.

I'm expecting to be able to find the time Real Soon Now to put a doc 
together with ways, for anyone willing, to provide us further insight on 
what's happening.  The monitor is supposed to be able to dump a heap 
profile (using gperftools) on-the-fly, but I recall having some issues 
with that not so long ago, so that's probably one thing to look into asap.

In any case, Stefan, do you by chance still have that monitor going?  If 
so, are you able to tell us what's that monitor rank? 'ceph mon_status' 
should help you assessing that.  Furthermore, assuming you are using a 
version prior to v0.58, any chance you can run a 'du -chs 
/var/lib/ceph/mon/ceph-foo', with foo being the mem-hogging monitor's 
id, and then, if you notice an abnormal disk consumption on one of the 
directories, dive in to check where said consumption is happening?

   -Joao

>
> On Wed, Mar 13, 2013 at 7:02 AM, Sébastien Han <han.sebastien@gmail.com> wrote:
>>
>> Hi,
>>
>> I do have some ceph-mon memory leak as well but it's on Argonaut.
>>
>> See the memory consumption here:
>> [snipped]
>>
>> Sorry for the long post.
>> --
>> Regards,
>> Sébastien Han.
>>
>>
>> On Wed, Mar 13, 2013 at 11:44 AM, Stefan Priebe - Profihost AG
>> <s.priebe@profihost.ag> wrote:
>>> Hi,
>>>
>>> are there any known ceph-mon memory leaks in bobtail? Today i've seen a
>>> ceph-mon process consuming 50GB Memory.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html