final mon issues

final mon issues

[Sage Weil]

I've merged the double-forward fix.  I still have one remaining problem
that I can reproduce with my tests which looks like an overlap of mon 
overload forwarded message priorities; I'm testing two possible fixes to 
see which are necessary to resolve it.

The memory issues with people using the gitbuilder appear to be because 
our gitbuilders weren't linking in tcmalloc.  The latest 'next' build 
*does* link it now, so if you are one of those people, please re-test!

I don't think either of those problems is related to the sync issues Mike 
is seeing, though.  I believe that is the final outstanding issue...?

sage

Re: final mon issues

[Wido den Hollander]

On 04/26/2013 09:43 PM, Sage Weil wrote:
> I've merged the double-forward fix.  I still have one remaining problem
> that I can reproduce with my tests which looks like an overlap of mon
> overload forwarded message priorities; I'm testing two possible fixes to
> see which are necessary to resolve it.
>
> The memory issues with people using the gitbuilder appear to be because
> our gitbuilders weren't linking in tcmalloc.  The latest 'next' build
> *does* link it now, so if you are one of those people, please re-test!
>
> I don't think either of those problems is related to the sync issues Mike
> is seeing, though.  I believe that is the final outstanding issue...?
>

I think I also just noticed a sync issue, reported it as: 
http://tracker.ceph.com/issues/4837

This e-mail and my report just crossed each other I think.

> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: final mon issues

[Mike Dawson]

Wido,

I think you saw the same issue I did 
(http://tracker.ceph.com/issues/4793). If so, I believe there is a fix 
in "next".

- Mike

On 4/26/2013 3:53 PM, Wido den Hollander wrote:
> On 04/26/2013 09:43 PM, Sage Weil wrote:
>> I've merged the double-forward fix.  I still have one remaining problem
>> that I can reproduce with my tests which looks like an overlap of mon
>> overload forwarded message priorities; I'm testing two possible fixes to
>> see which are necessary to resolve it.
>>
>> The memory issues with people using the gitbuilder appear to be because
>> our gitbuilders weren't linking in tcmalloc.  The latest 'next' build
>> *does* link it now, so if you are one of those people, please re-test!
>>
>> I don't think either of those problems is related to the sync issues Mike
>> is seeing, though.  I believe that is the final outstanding issue...?
>>
>
> I think I also just noticed a sync issue, reported it as:
> http://tracker.ceph.com/issues/4837
>
> This e-mail and my report just crossed each other I think.
>
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

Re: final mon issues

[Mike Dawson]

Oops. Looks like you are already on "next".

Greg,
Perhaps the issue still exists?

- Mike

On 4/26/2013 3:55 PM, Mike Dawson wrote:
> Wido,
>
> I think you saw the same issue I did
> (http://tracker.ceph.com/issues/4793). If so, I believe there is a fix
> in "next".
>
> - Mike
>
> On 4/26/2013 3:53 PM, Wido den Hollander wrote:
>> On 04/26/2013 09:43 PM, Sage Weil wrote:
>>> I've merged the double-forward fix.  I still have one remaining problem
>>> that I can reproduce with my tests which looks like an overlap of mon
>>> overload forwarded message priorities; I'm testing two possible fixes to
>>> see which are necessary to resolve it.
>>>
>>> The memory issues with people using the gitbuilder appear to be because
>>> our gitbuilders weren't linking in tcmalloc.  The latest 'next' build
>>> *does* link it now, so if you are one of those people, please re-test!
>>>
>>> I don't think either of those problems is related to the sync issues
>>> Mike
>>> is seeing, though.  I believe that is the final outstanding issue...?
>>>
>>
>> I think I also just noticed a sync issue, reported it as:
>> http://tracker.ceph.com/issues/4837
>>
>> This e-mail and my report just crossed each other I think.
>>
>>> sage
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>

Re: final mon issues

[Gregory Farnum]

Yeah, looking again... (and at your logs, too).
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 26, 2013 at 12:57 PM, Mike Dawson
<mike.dawson@scholarstack.com> wrote:
> Oops. Looks like you are already on "next".
>
> Greg,
> Perhaps the issue still exists?
>
> - Mike
>
>
> On 4/26/2013 3:55 PM, Mike Dawson wrote:
>>
>> Wido,
>>
>> I think you saw the same issue I did
>> (http://tracker.ceph.com/issues/4793). If so, I believe there is a fix
>> in "next".
>>
>> - Mike
>>
>> On 4/26/2013 3:53 PM, Wido den Hollander wrote:
>>>
>>> On 04/26/2013 09:43 PM, Sage Weil wrote:
>>>>
>>>> I've merged the double-forward fix.  I still have one remaining problem
>>>> that I can reproduce with my tests which looks like an overlap of mon
>>>> overload forwarded message priorities; I'm testing two possible fixes to
>>>> see which are necessary to resolve it.
>>>>
>>>> The memory issues with people using the gitbuilder appear to be because
>>>> our gitbuilders weren't linking in tcmalloc.  The latest 'next' build
>>>> *does* link it now, so if you are one of those people, please re-test!
>>>>
>>>> I don't think either of those problems is related to the sync issues
>>>> Mike
>>>> is seeing, though.  I believe that is the final outstanding issue...?
>>>>
>>>
>>> I think I also just noticed a sync issue, reported it as:
>>> http://tracker.ceph.com/issues/4837
>>>
>>> This e-mail and my report just crossed each other I think.
>>>
>>>> sage
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>>
>

Re: final mon issues

[Mark Nelson]

On 04/26/2013 02:43 PM, Sage Weil wrote:
> I've merged the double-forward fix.  I still have one remaining problem
> that I can reproduce with my tests which looks like an overlap of mon
> overload forwarded message priorities; I'm testing two possible fixes to
> see which are necessary to resolve it.
>
> The memory issues with people using the gitbuilder appear to be because
> our gitbuilders weren't linking in tcmalloc.  The latest 'next' build
> *does* link it now, so if you are one of those people, please re-test!
>
> I don't think either of those problems is related to the sync issues Mike
> is seeing, though.  I believe that is the final outstanding issue...?
>
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

fwiw, I'm seeing consistently lower mon usage now.  1-2GB instead of 
5-14GB during performance tests.

Mark

Re: final mon issues

[Mike Dawson]

Agreed. I can confirm ceph version 0.60-669-ga2a23cc 
(a2a23ccd959f6e7ebe1533b27e7320902624523b) fixes the tcmalloc issue.

root@xxx:~# ceph -m 10.1.0.67 heap stats
mon.btcmalloc heap stats:------------------------------------------------
MALLOC:       25694552 (   24.5 MiB) Bytes in use by application
MALLOC: +     53272576 (   50.8 MiB) Bytes in page heap freelist
MALLOC: +      1756920 (    1.7 MiB) Bytes in central cache freelist
MALLOC: +      5027328 (    4.8 MiB) Bytes in transfer cache freelist
MALLOC: +      8341936 (    8.0 MiB) Bytes in thread cache freelists
MALLOC: +      1372312 (    1.3 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =     95465624 (   91.0 MiB) Actual memory used (physical + swap)
MALLOC: +      7954432 (    7.6 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =    103420056 (   98.6 MiB) Virtual address space used
MALLOC:
MALLOC:           3760              Spans in use
MALLOC:             34              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via 
madvise()).
Bytes released to the


- Mike


On 4/26/2013 4:13 PM, Mark Nelson wrote:
> On 04/26/2013 02:43 PM, Sage Weil wrote:
>> I've merged the double-forward fix.  I still have one remaining problem
>> that I can reproduce with my tests which looks like an overlap of mon
>> overload forwarded message priorities; I'm testing two possible fixes to
>> see which are necessary to resolve it.
>>
>> The memory issues with people using the gitbuilder appear to be because
>> our gitbuilders weren't linking in tcmalloc.  The latest 'next' build
>> *does* link it now, so if you are one of those people, please re-test!
>>
>> I don't think either of those problems is related to the sync issues Mike
>> is seeing, though.  I believe that is the final outstanding issue...?
>>
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> fwiw, I'm seeing consistently lower mon usage now.  1-2GB instead of
> 5-14GB during performance tests.
>
> Mark