OSD and MON memory usage

OSD and MON memory usage

[Cláudio Martins]

 Hi,

 We're testing ceph using a recent build from the 'next' branch (commit
b40387d) and we've run into some interesting problems related to memory
usage.

 The setup consists of 64 OSDs (4 boxes, each with 16 disks, most of
them 2TB, some 1.5TB, XFS filesystems, Debian Wheezy). After the
initial mkcephfs, a 'ceph -s' reports 12480 pgs total.

 For generating some load we used

rados -p rbd bench 28000 write -t 25

and left it running overnight.

 After several hours most of the OSDs had eaten up around 1GB or more
of memory each, which caused trashing on the servers (12GB of RAM
per box), and eventually the OOM killer was invoked, killing many OSDs
and even the SSH daemons. This seems to have caused a domino effect,
and in the morning only around 18 of the OSD were still up.

 After a hard reboot of the boxes that were unresponsive, we are now in
a situation in which there is simply not enough memory for the cluster
to recover. That is, after restarting the OSDs, in 2 to 3 minutes we
have many of them using 1~1.5GB of RAM and the trashing starts all over
again, the OOM killer comes in and things go downhill again. Efectively
the cluster is not able to recover no matter how many times we restart
the daemons.

 We're not using any non-default options in the OSD section of the
config. file. We checked that there is free space for logging on the
system partitions.

 While I know that 12GB per machine can be hardly called to much RAM,
the question I put forward is: is it reasonable for a OSD to consume so
much memory in normal usage, or even recovery situations, when there is
just around ~200 PGs per OSD and only around ~3TB of objects created by
rados bench?

 Is there a rule of thumb to estimate the amount of memory consumed as
a function of PG count, object count and perhaps the number of PGs
trying to recover in a given instant? One of my concerns here is also
to understand if memory consumption during recovery is bounded and
deterministic at all, or if we're simply hitting a severe memory leak
in the OSDs.

 As for the monitor daemon on this cluster (running on a dedicated
machine), it is currently using 3.2GB of memory, and it got to that
point again in a matter of minutes after being restarted. Would it be
good if we tested with the changes from the wip-mon-leaks-fix branch?

 We would appreciate any advice on the best way to understand if the
OSDs are leaking memory or not.

 We will gladly provide any config or debug info that you might be
interested in, or run any tests.

 Thanks in advance

Best regards

Cláudio

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Joao Eduardo Luis]

On 11/16/2012 05:24 PM, Cláudio Martins wrote:
>  As for the monitor daemon on this cluster (running on a dedicated
> machine), it is currently using 3.2GB of memory, and it got to that
> point again in a matter of minutes after being restarted. Would it be
> good if we tested with the changes from the wip-mon-leaks-fix branch?

The wip-mon-leaks-fix branch had a regression introduced that would
cause your monitors to crash under certain conditions. The patch that
caused it has been identified and will be removed, but the branch still
needs some more testing before we consider it ready to be merged. We are
aiming however towards early next week.

For the time being, I wouldn't advise you to use that branch.

  -Joao
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Gregory Farnum]

On Fri, Nov 16, 2012 at 9:24 AM, Cláudio Martins <ctpm@ist.utl.pt> wrote:
>
>  Hi,
>
>  We're testing ceph using a recent build from the 'next' branch (commit
> b40387d) and we've run into some interesting problems related to memory
> usage.
>
>  The setup consists of 64 OSDs (4 boxes, each with 16 disks, most of
> them 2TB, some 1.5TB, XFS filesystems, Debian Wheezy). After the
> initial mkcephfs, a 'ceph -s' reports 12480 pgs total.
>
>  For generating some load we used
>
> rados -p rbd bench 28000 write -t 25
>
> and left it running overnight.
>
>  After several hours most of the OSDs had eaten up around 1GB or more
> of memory each, which caused trashing on the servers (12GB of RAM
> per box), and eventually the OOM killer was invoked, killing many OSDs
> and even the SSH daemons. This seems to have caused a domino effect,
> and in the morning only around 18 of the OSD were still up.

That's not good! Sam thinks you may have hit a memory leak, and I
believe I heard him making noises about discovering it elsewhere
earlier today. I will let him talk about that.


>  After a hard reboot of the boxes that were unresponsive, we are now in
> a situation in which there is simply not enough memory for the cluster
> to recover. That is, after restarting the OSDs, in 2 to 3 minutes we
> have many of them using 1~1.5GB of RAM and the trashing starts all over
> again, the OOM killer comes in and things go downhill again. Efectively
> the cluster is not able to recover no matter how many times we restart
> the daemons.
>
>  We're not using any non-default options in the OSD section of the
> config. file. We checked that there is free space for logging on the
> system partitions.
>
>  While I know that 12GB per machine can be hardly called to much RAM,
> the question I put forward is: is it reasonable for a OSD to consume so
> much memory in normal usage, or even recovery situations, when there is
> just around ~200 PGs per OSD and only around ~3TB of objects created by
> rados bench?

In normal usage, absolutely not. In recovery, for now, yes, we
consider that reasonable usage. We may be able to continue bringing
that down, but since it's a good number to have available for page
cache anyway we haven't been focusing on it much lately.

>  Is there a rule of thumb to estimate the amount of memory consumed as
> a function of PG count, object count and perhaps the number of PGs
> trying to recover in a given instant? One of my concerns here is also
> to understand if memory consumption during recovery is bounded and
> deterministic at all, or if we're simply hitting a severe memory leak
> in the OSDs.

It's going to vary based mostly on the number of PGs it's going to
recover, and especially based on if it's recovering or not. We
generally recommend 1GB per daemon for a well-configured (PGs in the
50-200 per OSD range) cluster.


>  As for the monitor daemon on this cluster (running on a dedicated
> machine), it is currently using 3.2GB of memory, and it got to that
> point again in a matter of minutes after being restarted. Would it be
> good if we tested with the changes from the wip-mon-leaks-fix branch?

Can you restart your monitor, then run "ceph heap start_profiler", and
then once the memory has gone way up run "ceph heap dump"? You should
find some files in the same directory as your monitor log files that
you can analyze using tcmalloc's tools (see
http://ceph.com/wiki/Memory_Profiling for some info), or send them our
way for analysis.
This will go easier if you also have Ceph's debug symbol packages installed. :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Joao Eduardo Luis]

On 11/19/2012 11:15 PM, Gregory Farnum wrote:
> Can you restart your monitor, then run "ceph heap start_profiler", and
> then once the memory has gone way up run "ceph heap dump"? You should
> find some files in the same directory as your monitor log files that
> you can analyze using tcmalloc's tools (see
> http://ceph.com/wiki/Memory_Profiling for some info), or send them our
> way for analysis.
> This will go easier if you also have Ceph's debug symbol packages installed. :)

The wiki has been somewhat officially deprecated and is now redirecting
to the docs. That page can still be accessed through

http://ceph.com/deprecated/Memory_Profiling

> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: OSD and MON memory usage

[Joao Eduardo Luis]

On 11/16/2012 05:24 PM, Cláudio Martins wrote:
>  As for the monitor daemon on this cluster (running on a dedicated
> machine), it is currently using 3.2GB of memory, and it got to that
> point again in a matter of minutes after being restarted. Would it be
> good if we tested with the changes from the wip-mon-leaks-fix branch?

Following up on this, wip-mon-leaks-fix was merged into master a couple
of days ago. If you have the chance to check if that fixes your memory
consumption issues on the monitor, it would be much appreciated!

  -Joao

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Cláudio Martins]

On Fri, 23 Nov 2012 16:46:00 +0000 Joao Eduardo Luis <joao.luis@inktank.com> wrote:
> On 11/16/2012 05:24 PM, Cláudio Martins wrote:
> >  As for the monitor daemon on this cluster (running on a dedicated
> > machine), it is currently using 3.2GB of memory, and it got to that
> > point again in a matter of minutes after being restarted. Would it be
> > good if we tested with the changes from the wip-mon-leaks-fix branch?
> 
> Following up on this, wip-mon-leaks-fix was merged into master a couple
> of days ago. If you have the chance to check if that fixes your memory
> consumption issues on the monitor, it would be much appreciated!
> 

 Hi João,

 I've had a chance to test it and it does indeed seem to make a big
difference on mon memory usage.

 As for the OSD memory usage issue, it's still looking really bad. I'm
preparing to do more testing and send more info about this, but a lot
of unrelated stuff crept up this week and things are going slowly on
this front. I hope to talk more about this before the weekend.

Thanks!

Cláudio

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Samuel Just]

What replication level are you using?
-Sam

On Tue, Nov 27, 2012 at 9:23 AM, Cláudio Martins <ctpm@ist.utl.pt> wrote:
>
> On Fri, 23 Nov 2012 16:46:00 +0000 Joao Eduardo Luis <joao.luis@inktank.com> wrote:
>> On 11/16/2012 05:24 PM, Cláudio Martins wrote:
>> >  As for the monitor daemon on this cluster (running on a dedicated
>> > machine), it is currently using 3.2GB of memory, and it got to that
>> > point again in a matter of minutes after being restarted. Would it be
>> > good if we tested with the changes from the wip-mon-leaks-fix branch?
>>
>> Following up on this, wip-mon-leaks-fix was merged into master a couple
>> of days ago. If you have the chance to check if that fixes your memory
>> consumption issues on the monitor, it would be much appreciated!
>>
>
>  Hi João,
>
>  I've had a chance to test it and it does indeed seem to make a big
> difference on mon memory usage.
>
>  As for the OSD memory usage issue, it's still looking really bad. I'm
> preparing to do more testing and send more info about this, but a lot
> of unrelated stuff crept up this week and things are going slowly on
> this front. I hope to talk more about this before the weekend.
>
> Thanks!
>
> Cláudio
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Cláudio Martins]

On Wed, 28 Nov 2012 13:00:17 -0800 Samuel Just <sam.just@inktank.com> wrote:
> What replication level are you using?

 Hi,

 The replication level is 3.

Thanks

Cláudio

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Samuel Just]

Can you post the output of ceph -s?
-Sam

On Wed, Nov 28, 2012 at 1:05 PM, Cláudio Martins <ctpm@ist.utl.pt> wrote:
>
> On Wed, 28 Nov 2012 13:00:17 -0800 Samuel Just <sam.just@inktank.com> wrote:
>> What replication level are you using?
>
>  Hi,
>
>  The replication level is 3.
>
> Thanks
>
> Cláudio
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Cláudio Martins]

On Wed, 28 Nov 2012 13:08:08 -0800 Samuel Just <sam.just@inktank.com> wrote:
> Can you post the output of ceph -s?

 'ceph -s' right now gives

   health HEALTH_WARN 923 pgs degraded; 8666 pgs down; 9606 pgs peering; 7 pgs recovering; 406 pgs recovery_wait; 3769 pgs stale; 9606 pgs stuck inactive; 3769 pgs stuck stale; 11052 pgs stuck unclean; recovery 121068/902868 degraded (13.409%); 4824/300956 unfound (1.603%); 2/18 in osds are down
   monmap e1: 1 mons at {0=193.136.128.202:6789/0}, election epoch 1, quorum 0 0
   osdmap e7669: 62 osds: 16 up, 18 in
    pgmap v47643: 12480 pgs: 35 active, 1223 active+clean, 129 stale+active, 321 active+recovery_wait, 198 stale+active+clean, 236 peering, 2 active+remapped, 2 stale+active+recovery_wait, 6126 down+peering, 249 active+degraded, 2 stale+active+recovering+degraded, 598 stale+peering, 7 active+clean+scrubbing, 29 active+recovery_wait+remapped, 2067 stale+down+peering, 618 stale+active+degraded, 52 active+recovery_wait+degraded, 61 remapped+peering, 365 down+remapped+peering, 2 stale+active+recovery_wait+degraded, 45 stale+remapped+peering, 108 stale+down+remapped+peering, 5 active+recovering; 1175 GB data, 1794 GB used, 25969 GB / 27764 GB avail; 121068/902868 degraded (13.409%); 4824/300956 unfound (1.603%)
   mdsmap e1: 0/0/1 up



 The cluster has been in this state since the last attempt to get it
going. I added about 100GB of swap on each machine to avoid the OOM
killer. Running like this resulted in the machines trashing wildly and
getting to ~2000 load avg, and after a while the osds started
dying/commited suicide, but *not* from OOM. Some of the few that remain
have bloated to around 1.9GB of mem usage.

 If you want, I can try to restart the whole thing tomorrow and collect
fresh log output from the dying OSDs, or any other action or debug info
that you might find useful.


Thanks!

Cláudio

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Sylvain Munaut]

Hi,

>  If you want, I can try to restart the whole thing tomorrow and collect
> fresh log output from the dying OSDs, or any other action or debug info
> that you might find useful.

Is the clock synchronized on all machines ?

What you describe (growing mem, recovery that doesn't seem to end)
seems pretty similar to what I experienced when clocks of OSD were off
...

Cheers,

    Sylvain

Re: OSD and MON memory usage

[Cláudio Martins]

On Thu, 29 Nov 2012 00:13:25 +0100 Sylvain Munaut <s.munaut@whatever-company.com> wrote:
> Hi,
> 
> >  If you want, I can try to restart the whole thing tomorrow and collect
> > fresh log output from the dying OSDs, or any other action or debug info
> > that you might find useful.
> 
> Is the clock synchronized on all machines ?
> 


 Yup. All machines synched by ntp.

Cláudio


> What you describe (growing mem, recovery that doesn't seem to end)
> seems pretty similar to what I experienced when clocks of OSD were off
> ...
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD and MON memory usage

[Gregory Farnum]

On Wed, Nov 28, 2012 at 1:30 PM, Cláudio Martins <ctpm@ist.utl.pt> wrote:
>
> On Wed, 28 Nov 2012 13:08:08 -0800 Samuel Just <sam.just@inktank.com> wrote:
>> Can you post the output of ceph -s?
>
>  'ceph -s' right now gives
>
>    health HEALTH_WARN 923 pgs degraded; 8666 pgs down; 9606 pgs peering; 7 pgs recovering; 406 pgs recovery_wait; 3769 pgs stale; 9606 pgs stuck inactive; 3769 pgs stuck stale; 11052 pgs stuck unclean; recovery 121068/902868 degraded (13.409%); 4824/300956 unfound (1.603%); 2/18 in osds are down
>    monmap e1: 1 mons at {0=193.136.128.202:6789/0}, election epoch 1, quorum 0 0
>    osdmap e7669: 62 osds: 16 up, 18 in
>     pgmap v47643: 12480 pgs: 35 active, 1223 active+clean, 129 stale+active, 321 active+recovery_wait, 198 stale+active+clean, 236 peering, 2 active+remapped, 2 stale+active+recovery_wait, 6126 down+peering, 249 active+degraded, 2 stale+active+recovering+degraded, 598 stale+peering, 7 active+clean+scrubbing, 29 active+recovery_wait+remapped, 2067 stale+down+peering, 618 stale+active+degraded, 52 active+recovery_wait+degraded, 61 remapped+peering, 365 down+remapped+peering, 2 stale+active+recovery_wait+degraded, 45 stale+remapped+peering, 108 stale+down+remapped+peering, 5 active+recovering; 1175 GB data, 1794 GB used, 25969 GB / 27764 GB avail; 121068/902868 degraded (13.409%); 4824/300956 unfound (1.603%)
>    mdsmap e1: 0/0/1 up
>
>
>
>  The cluster has been in this state since the last attempt to get it
> going. I added about 100GB of swap on each machine to avoid the OOM
> killer. Running like this resulted in the machines trashing wildly and
> getting to ~2000 load avg, and after a while the osds started
> dying/commited suicide, but *not* from OOM. Some of the few that remain
> have bloated to around 1.9GB of mem usage.
>
>  If you want, I can try to restart the whole thing tomorrow and collect
> fresh log output from the dying OSDs, or any other action or debug info
> that you might find useful.

Made any progress here? Given your RAM limitations I'd suggest doing
an incremental startup of the OSDs, maybe 4 per node, let me stabilize
as much as they can, then add another couple per node, let them
stabilize, etc.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html