From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Dawson Subject: Re: mon IO usage Date: Tue, 21 May 2013 09:25:06 -0400 Message-ID: <519B75B2.4070405@scholarstack.com> References: <519B6F28.9000400@scholarstack.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <519B6F28.9000400-9dgm/EUDD3RBYT3KYJiKsA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org To: Sylvain Munaut Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" List-Id: ceph-devel.vger.kernel.org Thanks for the correction on IRC. I should have written that this issue started with 0.59 (when the monitor changes hit). http://ceph.com/dev-notes/cephs-new-monitor-changes/ The writeup and release notes sometimes say they went in for 0.58, but I believe they were actually released in 0.59. Thanks for the correction Sylvain. - Mike On 5/21/2013 8:57 AM, Mike Dawson wrote: > Sylvain, > > I can confirm I see a similar traffic pattern. > > Any time I have lots of writes going to my cluster (like heavy writes > from RBD or remapping/backfilling after losing an OSD), I see all sorts > of monitor issues. > > If my monitor leveldb store.db directories grow past some unknown point > (maybe ~1GB or so), 'compact on trim' is insufficiently slow. The > store.db grows faster than compact can trim the garbage. After that > point, the only hope to rein in the store.db size is to stop the OSDs > and get leveldb to compact without any ongoing writes. > > I sent Sage and Joao a transaction dump of the growth yesterday. Sage > looked, but the files are so large it is tough to get useful info. > > http://tracker.ceph.com/issues/4895 > > I believe this issue has existed since 0.48. > > - Mike > > On 5/21/2013 8:16 AM, Sylvain Munaut wrote: >> Hi, >> >> >> I've just added some monitoring to the IO usage of mon (trying to >> track down that growing mon issue), and I'm kind of surprised by the >> amount of IO generated by the monitor process. >> >> I get continuous 4 Mo/s / 75 iops with added big spikes at each >> compaction every 3 min or so. >> >> Is there a description somewhere of what the monitor does exactly ? I >> mean the monmap / pgmap / osdmap / mdsmap / election epoch don't >> change that often (pgmap is like 1 per second and that's the fastest >> change by several orders of magnitude). So what exactly does the >> monitor do with all that IO ??? >> >> >> Cheers, >> >> Sylvain >> _______________________________________________ >> ceph-users mailing list >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>