ceph-mds jewel backports leak

ceph-mds jewel backports leak

[Loic Dachary]

Hi John,

While testing the CephFS jewel backports found in the jewel-backports branch, we ran into a memory leak that consistently happen:

http://pulpito.ceph.com/loic-2016-06-16_02:28:14-fs-jewel-backports---basic-smithi/263842

and that is not present in the jewel branch

http://pulpito.ceph.com/loic-2016-06-28_23:43:39-fs-jewel---basic-smithi/282493/

Meaning one of the backports is doing something wrong. Or there is a backport fixing the leak that is missing.

This one looks interesting to me

https://github.com/ceph/ceph/commit/37b7b09ee19ad64362c2faf63461353c03a36ada

There also are related to leaks but I'm not so sure.

https://github.com/ceph/ceph/commit/89f74c98ea43df46291c8bec890b3a75c7ce38b4
https://github.com/ceph/ceph/commit/4c52bf9ed47d52510536ebc34bd6c56adba1df53

If you have ideas on how to approach this and save random exploration time, it would be most welcome. If not, I'll get it eventually, no worries :-)

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-mds jewel backports leak

[Loic Dachary]

To be precise, the backports that are related to CephFS and in the jewel-backports branch are:

https://github.com/ceph/ceph/pull/9405/files
https://github.com/ceph/ceph/pull/9547/files
https://github.com/ceph/ceph/pull/9558/files
https://github.com/ceph/ceph/pull/9559/files
https://github.com/ceph/ceph/pull/9560/files
https://github.com/ceph/ceph/pull/9562/files

On 06/07/2016 12:51, Loic Dachary wrote:
> Hi John,
> 
> While testing the CephFS jewel backports found in the jewel-backports branch, we ran into a memory leak that consistently happen:
> 
> http://pulpito.ceph.com/loic-2016-06-16_02:28:14-fs-jewel-backports---basic-smithi/263842
> 
> and that is not present in the jewel branch
> 
> http://pulpito.ceph.com/loic-2016-06-28_23:43:39-fs-jewel---basic-smithi/282493/
> 
> Meaning one of the backports is doing something wrong. Or there is a backport fixing the leak that is missing.
> 
> This one looks interesting to me
> 
> https://github.com/ceph/ceph/commit/37b7b09ee19ad64362c2faf63461353c03a36ada
> 
> There also are related to leaks but I'm not so sure.
> 
> https://github.com/ceph/ceph/commit/89f74c98ea43df46291c8bec890b3a75c7ce38b4
> https://github.com/ceph/ceph/commit/4c52bf9ed47d52510536ebc34bd6c56adba1df53
> 
> If you have ideas on how to approach this and save random exploration time, it would be most welcome. If not, I'll get it eventually, no worries :-)
> 
> Cheers
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-mds jewel backports leak

[John Spray]

On Wed, Jul 6, 2016 at 11:51 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi John,
>
> While testing the CephFS jewel backports found in the jewel-backports branch, we ran into a memory leak that consistently happen:
>
> http://pulpito.ceph.com/loic-2016-06-16_02:28:14-fs-jewel-backports---basic-smithi/263842

The client leak is a false positive, I'm not sure what is triggering
it (it seems to only happen on centos nodes, and not be silenced by
the suppression that we have in the valgrind config).
http://tracker.ceph.com/issues/14794

Is it definitely not happening on the jewel branch, or is it possible
that the jewel run just wasn't executing that test on centos nodes?

The MDS leaks are currently ignored in the ceph.py task (yuck!) so
they're probably happening everywhere else too and you can ignore
them.

John

> and that is not present in the jewel branch
>
> http://pulpito.ceph.com/loic-2016-06-28_23:43:39-fs-jewel---basic-smithi/282493/
>
> Meaning one of the backports is doing something wrong. Or there is a backport fixing the leak that is missing.
>
> This one looks interesting to me
>
> https://github.com/ceph/ceph/commit/37b7b09ee19ad64362c2faf63461353c03a36ada
>
> There also are related to leaks but I'm not so sure.
>
> https://github.com/ceph/ceph/commit/89f74c98ea43df46291c8bec890b3a75c7ce38b4
> https://github.com/ceph/ceph/commit/4c52bf9ed47d52510536ebc34bd6c56adba1df53
>
> If you have ideas on how to approach this and save random exploration time, it would be most welcome. If not, I'll get it eventually, no worries :-)
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-mds jewel backports leak

[Loic Dachary]

Hi John,

On 07/07/2016 12:54, John Spray wrote:
> On Wed, Jul 6, 2016 at 11:51 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi John,
>>
>> While testing the CephFS jewel backports found in the jewel-backports branch, we ran into a memory leak that consistently happen:
>>
>> http://pulpito.ceph.com/loic-2016-06-16_02:28:14-fs-jewel-backports---basic-smithi/263842
> 
> The client leak is a false positive, I'm not sure what is triggering
> it (it seems to only happen on centos nodes, and not be silenced by
> the suppression that we have in the valgrind config).
> http://tracker.ceph.com/issues/14794

Thanks for that, I'm relieved :-)

> Is it definitely not happening on the jewel branch, or is it possible
> that the jewel run just wasn't executing that test on centos nodes?

http://pulpito.ceph.com/loic-2016-06-28_23:43:39-fs-jewel---basic-smithi/282493/ is indeed running on Ubuntu

> 
> The MDS leaks are currently ignored in the ceph.py task (yuck!) so
> they're probably happening everywhere else too and you can ignore
> them.

And the reason why it show on jewel is probably because the ceph.py task in ceph-qa-suite jewel branch does not ignore leaks. I took notes so we remember it's a false positive http://tracker.ceph.com/issues/16344#note-9

Thanks !

> John
> 
>> and that is not present in the jewel branch
>>
>> http://pulpito.ceph.com/loic-2016-06-28_23:43:39-fs-jewel---basic-smithi/282493/
>>
>> Meaning one of the backports is doing something wrong. Or there is a backport fixing the leak that is missing.
>>
>> This one looks interesting to me
>>
>> https://github.com/ceph/ceph/commit/37b7b09ee19ad64362c2faf63461353c03a36ada
>>
>> There also are related to leaks but I'm not so sure.
>>
>> https://github.com/ceph/ceph/commit/89f74c98ea43df46291c8bec890b3a75c7ce38b4
>> https://github.com/ceph/ceph/commit/4c52bf9ed47d52510536ebc34bd6c56adba1df53
>>
>> If you have ideas on how to approach this and save random exploration time, it would be most welcome. If not, I'll get it eventually, no worries :-)
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html