Re: Possible memory leak in mon?

[Greg Farnum <gregory.farnum@dreamhost.com>]

On Wednesday, May 2, 2012 at 3:28 PM, Vladimir Bashkirtsev wrote:
> Dear devs,
> 
> I have three mons and two of them suddenly consumed around 4G of RAM 
> while third one happily lived with 150M. This immediately prompts few 
> questions:
> 
> 1. What is expected memory use of mon? I believed that mon merely 
> directs clients to relevant OSDs and should not consume a lot of 
> resources - please correct me if I am wrong.
> 2. In both cases where mon consumed a lot of memory it was preceded by 
> disk-full condition and both machines where incidents happened are 64 
> bit, rest of cluster 32 bit. mon fs and log files happened to be in the 
> same partition - ceph osd produced a lot of messages, filled up disk, 
> mon crashed (no core as disk was full), manually deleted logs, restarted 
> mon without any issue, some time later found mon using 4G of RAM. 
> Running 0.45. Should I deliberately recreate conditions and crash mon to 
> get more debug info (if you need it of course, and if yes then what)?
> 3. Does figure 4G per process coming from 32 bit pointers in mon? Or mon 
> potentially can consume more than 4G?
> 
> Regards,
> Vladimir
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo-info.html

First: one email is enough. 

Second: in normal use your monitors should not consume very much memory. It sounds like something's wrong. Can you please provide the output of "ceph -s"?
Also, do you have any monitor logging on? My best guess is that for some reason the monitors aren't all communicating with each other and so they are buffering messages.
-Greg