Re: [ceph-users] mon IO usage

[Mike Dawson <mike.dawson@scholarstack.com>]

Sylvain,

I can confirm I see a similar traffic pattern.

Any time I have lots of writes going to my cluster (like heavy writes 
from RBD or remapping/backfilling after losing an OSD), I see all sorts 
of monitor issues.

If my monitor leveldb store.db directories grow past some unknown point 
(maybe ~1GB or so), 'compact on trim' is insufficiently slow. The 
store.db grows faster than compact can trim the garbage. After that 
point, the only hope to rein in the store.db size is to stop the OSDs 
and get leveldb to compact without any ongoing writes.

I sent Sage and Joao a transaction dump of the growth yesterday. Sage 
looked, but the files are so large it is tough to get useful info.

http://tracker.ceph.com/issues/4895

I believe this issue has existed since 0.48.

- Mike

On 5/21/2013 8:16 AM, Sylvain Munaut wrote:
> Hi,
>
>
> I've just added some monitoring to the IO usage of mon (trying to
> track down that growing mon issue), and I'm kind of surprised by the
> amount of IO generated by the monitor process.
>
> I get continuous 4 Mo/s / 75 iops with added big spikes at each
> compaction every 3 min or so.
>
> Is there a description somewhere of what the monitor does exactly ?  I
> mean the monmap / pgmap / osdmap / mdsmap / election epoch don't
> change that often (pgmap is like 1 per second and that's the fastest
> change by several orders of magnitude). So what exactly does the
> monitor do with all that IO ???
>
>
> Cheers,
>
>      Sylvain
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>