* Log files with 0.45 @ 2012-04-18 4:53 Danny Kukawka 2012-04-18 5:09 ` Greg Farnum 0 siblings, 1 reply; 6+ messages in thread From: Danny Kukawka @ 2012-04-18 4:53 UTC (permalink / raw) To: ceph-devel; +Cc: Danny Kukawka [-- Attachment #1: Type: text/plain, Size: 250 bytes --] Hi, did something change with the default log levels for OSDs on v0.45? With 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I get by default ~3Gbyte per OSD over 12 hours without any change in the config file. Danny [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 316 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45 2012-04-18 4:53 Log files with 0.45 Danny Kukawka @ 2012-04-18 5:09 ` Greg Farnum 2012-04-20 20:05 ` Nick Bartos 0 siblings, 1 reply; 6+ messages in thread From: Greg Farnum @ 2012-04-18 5:09 UTC (permalink / raw) To: Danny Kukawka, Samuel Just; +Cc: ceph-devel, Danny Kukawka On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: > Hi, > > did something change with the default log levels for OSDs on v0.45? With > 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I > get by default ~3Gbyte per OSD over 12 hours without any change in the > config file. > > Danny I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though! Can you comment on this, Sam? -Greg ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45 2012-04-18 5:09 ` Greg Farnum @ 2012-04-20 20:05 ` Nick Bartos 2012-04-20 20:09 ` Calvin Morrow 2012-04-20 20:40 ` Greg Farnum 0 siblings, 2 replies; 6+ messages in thread From: Nick Bartos @ 2012-04-20 20:05 UTC (permalink / raw) To: Greg Farnum; +Cc: Danny Kukawka, Samuel Just, ceph-devel, Danny Kukawka Is there a recommended log config for production systems? I'm also trying to decrease the verbosity in 0.45, using the options specified here: http://ceph.newdream.net/wiki/Debugging. Setting them down to '1' doesn't end the insane log sprawl I'm seeing. On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum <gregory.farnum@dreamhost.com> wrote: > On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: >> Hi, >> >> did something change with the default log levels for OSDs on v0.45? With >> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I >> get by default ~3Gbyte per OSD over 12 hours without any change in the >> config file. >> >> Danny > I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though! > > Can you comment on this, Sam? > -Greg > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45 2012-04-20 20:05 ` Nick Bartos @ 2012-04-20 20:09 ` Calvin Morrow 2012-04-20 20:21 ` Sage Weil 2012-04-20 20:40 ` Greg Farnum 1 sibling, 1 reply; 6+ messages in thread From: Calvin Morrow @ 2012-04-20 20:09 UTC (permalink / raw) To: ceph-devel I'm seeing the same. < 12 hours with 6 OSDs resulted in ~18 GB of logs. I had to change my log rotate config to compress based on size instead of once a day or I ended up with a full root partition. I would love to know if there's a better way to handle it. Calvin On Fri, Apr 20, 2012 at 5:05 PM, Nick Bartos <nick@pistoncloud.com> wrote: > Is there a recommended log config for production systems? I'm also > trying to decrease the verbosity in 0.45, using the options specified > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to > '1' doesn't end the insane log sprawl I'm seeing. > > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum > <gregory.farnum@dreamhost.com> wrote: >> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: >>> Hi, >>> >>> did something change with the default log levels for OSDs on v0.45? With >>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I >>> get by default ~3Gbyte per OSD over 12 hours without any change in the >>> config file. >>> >>> Danny >> I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though! >> >> Can you comment on this, Sam? >> -Greg >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45 2012-04-20 20:09 ` Calvin Morrow @ 2012-04-20 20:21 ` Sage Weil 0 siblings, 0 replies; 6+ messages in thread From: Sage Weil @ 2012-04-20 20:21 UTC (permalink / raw) To: Calvin Morrow; +Cc: ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 2284 bytes --] On Fri, 20 Apr 2012, Calvin Morrow wrote: > I'm seeing the same. < 12 hours with 6 OSDs resulted in ~18 GB of > logs. I had to change my log rotate config to compress based on size > instead of once a day or I ended up with a full root partition. > > I would love to know if there's a better way to handle it. This will be fixed in 0.46. You can also cherry-pick ac874a62533c3845f9e68201bfd4f7020332da24 from the stable branch to fix this. You can also set 'debug optracker = 0' to turn this down. sage > > Calvin > > On Fri, Apr 20, 2012 at 5:05 PM, Nick Bartos <nick@pistoncloud.com> wrote: > > Is there a recommended log config for production systems? I'm also > > trying to decrease the verbosity in 0.45, using the options specified > > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to > > '1' doesn't end the insane log sprawl I'm seeing. > > > > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum > > <gregory.farnum@dreamhost.com> wrote: > >> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: > >>> Hi, > >>> > >>> did something change with the default log levels for OSDs on v0.45? With > >>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I > >>> get by default ~3Gbyte per OSD over 12 hours without any change in the > >>> config file. > >>> > >>> Danny > >> I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though! > >> > >> Can you comment on this, Sam? > >> -Greg > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45 2012-04-20 20:05 ` Nick Bartos 2012-04-20 20:09 ` Calvin Morrow @ 2012-04-20 20:40 ` Greg Farnum 1 sibling, 0 replies; 6+ messages in thread From: Greg Farnum @ 2012-04-20 20:40 UTC (permalink / raw) To: Nick Bartos, Danny Kukawka; +Cc: ceph-devel, Danny Kukawka I checked with Sam on this and it turns out he created a new subsystem whose output you can control with the "debug optracker" (or --debug-optracker) option (in the same way as the other debug log systems). In 0.45 the output for that system was at inappropriately high levels (1) and it's fixed in our current master (5 now), but you probably want to set "debug optracker = 0" in your config. That should restore things to the way they used to be! (And sorry for the long wait, Danny.) -Greg On Friday, April 20, 2012 at 1:05 PM, Nick Bartos wrote: > Is there a recommended log config for production systems? I'm also > trying to decrease the verbosity in 0.45, using the options specified > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to > '1' doesn't end the insane log sprawl I'm seeing. > > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum > <gregory.farnum@dreamhost.com (mailto:gregory.farnum@dreamhost.com)> wrote: > > On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: > > > Hi, > > > > > > did something change with the default log levels for OSDs on v0.45? With > > > 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I > > > get by default ~3Gbyte per OSD over 12 hours without any change in the > > > config file. > > > > > > Danny > > I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though! > > > > Can you comment on this, Sam? > > -Greg > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org) > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org) > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-04-20 20:40 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-04-18 4:53 Log files with 0.45 Danny Kukawka 2012-04-18 5:09 ` Greg Farnum 2012-04-20 20:05 ` Nick Bartos 2012-04-20 20:09 ` Calvin Morrow 2012-04-20 20:21 ` Sage Weil 2012-04-20 20:40 ` Greg Farnum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.