Re: mon IO usage

[Mike Dawson <mike.dawson-9dgm/EUDD3RBYT3KYJiKsA@public.gmane.org>]

Thanks for the correction on IRC. I should have written that this issue 
started with 0.59 (when the monitor changes hit).

http://ceph.com/dev-notes/cephs-new-monitor-changes/

The writeup and release notes sometimes say they went in for 0.58, but I 
believe they were actually released in 0.59.

Thanks for the correction Sylvain.

- Mike

On 5/21/2013 8:57 AM, Mike Dawson wrote:
> Sylvain,
>
> I can confirm I see a similar traffic pattern.
>
> Any time I have lots of writes going to my cluster (like heavy writes
> from RBD or remapping/backfilling after losing an OSD), I see all sorts
> of monitor issues.
>
> If my monitor leveldb store.db directories grow past some unknown point
> (maybe ~1GB or so), 'compact on trim' is insufficiently slow. The
> store.db grows faster than compact can trim the garbage. After that
> point, the only hope to rein in the store.db size is to stop the OSDs
> and get leveldb to compact without any ongoing writes.
>
> I sent Sage and Joao a transaction dump of the growth yesterday. Sage
> looked, but the files are so large it is tough to get useful info.
>
> http://tracker.ceph.com/issues/4895
>
> I believe this issue has existed since 0.48.
>
> - Mike
>
> On 5/21/2013 8:16 AM, Sylvain Munaut wrote:
>> Hi,
>>
>>
>> I've just added some monitoring to the IO usage of mon (trying to
>> track down that growing mon issue), and I'm kind of surprised by the
>> amount of IO generated by the monitor process.
>>
>> I get continuous 4 Mo/s / 75 iops with added big spikes at each
>> compaction every 3 min or so.
>>
>> Is there a description somewhere of what the monitor does exactly ?  I
>> mean the monmap / pgmap / osdmap / mdsmap / election epoch don't
>> change that often (pgmap is like 1 per second and that's the fastest
>> change by several orders of magnitude). So what exactly does the
>> monitor do with all that IO ???
>>
>>
>> Cheers,
>>
>>      Sylvain
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>