From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Dawson Subject: Re: [ceph-users] mon IO usage Date: Tue, 21 May 2013 08:57:12 -0400 Message-ID: <519B6F28.9000400@scholarstack.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: Received: from mail-ie0-f179.google.com ([209.85.223.179]:52336 "EHLO mail-ie0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753794Ab3EUM5Q (ORCPT ); Tue, 21 May 2013 08:57:16 -0400 Received: by mail-ie0-f179.google.com with SMTP id c13so1361399ieb.24 for ; Tue, 21 May 2013 05:57:14 -0700 (PDT) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Sylvain Munaut Cc: ceph-users@lists.ceph.com, "ceph-devel@vger.kernel.org" Sylvain, I can confirm I see a similar traffic pattern. Any time I have lots of writes going to my cluster (like heavy writes from RBD or remapping/backfilling after losing an OSD), I see all sorts of monitor issues. If my monitor leveldb store.db directories grow past some unknown point (maybe ~1GB or so), 'compact on trim' is insufficiently slow. The store.db grows faster than compact can trim the garbage. After that point, the only hope to rein in the store.db size is to stop the OSDs and get leveldb to compact without any ongoing writes. I sent Sage and Joao a transaction dump of the growth yesterday. Sage looked, but the files are so large it is tough to get useful info. http://tracker.ceph.com/issues/4895 I believe this issue has existed since 0.48. - Mike On 5/21/2013 8:16 AM, Sylvain Munaut wrote: > Hi, > > > I've just added some monitoring to the IO usage of mon (trying to > track down that growing mon issue), and I'm kind of surprised by the > amount of IO generated by the monitor process. > > I get continuous 4 Mo/s / 75 iops with added big spikes at each > compaction every 3 min or so. > > Is there a description somewhere of what the monitor does exactly ? I > mean the monmap / pgmap / osdmap / mdsmap / election epoch don't > change that often (pgmap is like 1 per second and that's the fastest > change by several orders of magnitude). So what exactly does the > monitor do with all that IO ??? > > > Cheers, > > Sylvain > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >