* Log files with 0.45
@ 2012-04-18 4:53 Danny Kukawka
2012-04-18 5:09 ` Greg Farnum
0 siblings, 1 reply; 6+ messages in thread
From: Danny Kukawka @ 2012-04-18 4:53 UTC (permalink / raw)
To: ceph-devel; +Cc: Danny Kukawka
[-- Attachment #1: Type: text/plain, Size: 250 bytes --]
Hi,
did something change with the default log levels for OSDs on v0.45? With
0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I
get by default ~3Gbyte per OSD over 12 hours without any change in the
config file.
Danny
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45
2012-04-18 4:53 Log files with 0.45 Danny Kukawka
@ 2012-04-18 5:09 ` Greg Farnum
2012-04-20 20:05 ` Nick Bartos
0 siblings, 1 reply; 6+ messages in thread
From: Greg Farnum @ 2012-04-18 5:09 UTC (permalink / raw)
To: Danny Kukawka, Samuel Just; +Cc: ceph-devel, Danny Kukawka
On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote:
> Hi,
>
> did something change with the default log levels for OSDs on v0.45? With
> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I
> get by default ~3Gbyte per OSD over 12 hours without any change in the
> config file.
>
> Danny
I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though!
Can you comment on this, Sam?
-Greg
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45
2012-04-18 5:09 ` Greg Farnum
@ 2012-04-20 20:05 ` Nick Bartos
2012-04-20 20:09 ` Calvin Morrow
2012-04-20 20:40 ` Greg Farnum
0 siblings, 2 replies; 6+ messages in thread
From: Nick Bartos @ 2012-04-20 20:05 UTC (permalink / raw)
To: Greg Farnum; +Cc: Danny Kukawka, Samuel Just, ceph-devel, Danny Kukawka
Is there a recommended log config for production systems? I'm also
trying to decrease the verbosity in 0.45, using the options specified
here: http://ceph.newdream.net/wiki/Debugging. Setting them down to
'1' doesn't end the insane log sprawl I'm seeing.
On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum
<gregory.farnum@dreamhost.com> wrote:
> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote:
>> Hi,
>>
>> did something change with the default log levels for OSDs on v0.45? With
>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I
>> get by default ~3Gbyte per OSD over 12 hours without any change in the
>> config file.
>>
>> Danny
> I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though!
>
> Can you comment on this, Sam?
> -Greg
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45
2012-04-20 20:05 ` Nick Bartos
@ 2012-04-20 20:09 ` Calvin Morrow
2012-04-20 20:21 ` Sage Weil
2012-04-20 20:40 ` Greg Farnum
1 sibling, 1 reply; 6+ messages in thread
From: Calvin Morrow @ 2012-04-20 20:09 UTC (permalink / raw)
To: ceph-devel
I'm seeing the same. < 12 hours with 6 OSDs resulted in ~18 GB of
logs. I had to change my log rotate config to compress based on size
instead of once a day or I ended up with a full root partition.
I would love to know if there's a better way to handle it.
Calvin
On Fri, Apr 20, 2012 at 5:05 PM, Nick Bartos <nick@pistoncloud.com> wrote:
> Is there a recommended log config for production systems? I'm also
> trying to decrease the verbosity in 0.45, using the options specified
> here: http://ceph.newdream.net/wiki/Debugging. Setting them down to
> '1' doesn't end the insane log sprawl I'm seeing.
>
> On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum
> <gregory.farnum@dreamhost.com> wrote:
>> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote:
>>> Hi,
>>>
>>> did something change with the default log levels for OSDs on v0.45? With
>>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I
>>> get by default ~3Gbyte per OSD over 12 hours without any change in the
>>> config file.
>>>
>>> Danny
>> I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though!
>>
>> Can you comment on this, Sam?
>> -Greg
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45
2012-04-20 20:09 ` Calvin Morrow
@ 2012-04-20 20:21 ` Sage Weil
0 siblings, 0 replies; 6+ messages in thread
From: Sage Weil @ 2012-04-20 20:21 UTC (permalink / raw)
To: Calvin Morrow; +Cc: ceph-devel
[-- Attachment #1: Type: TEXT/PLAIN, Size: 2284 bytes --]
On Fri, 20 Apr 2012, Calvin Morrow wrote:
> I'm seeing the same. < 12 hours with 6 OSDs resulted in ~18 GB of
> logs. I had to change my log rotate config to compress based on size
> instead of once a day or I ended up with a full root partition.
>
> I would love to know if there's a better way to handle it.
This will be fixed in 0.46. You can also cherry-pick
ac874a62533c3845f9e68201bfd4f7020332da24 from the stable branch to fix
this.
You can also set 'debug optracker = 0' to turn this down.
sage
>
> Calvin
>
> On Fri, Apr 20, 2012 at 5:05 PM, Nick Bartos <nick@pistoncloud.com> wrote:
> > Is there a recommended log config for production systems? I'm also
> > trying to decrease the verbosity in 0.45, using the options specified
> > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to
> > '1' doesn't end the insane log sprawl I'm seeing.
> >
> > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum
> > <gregory.farnum@dreamhost.com> wrote:
> >> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote:
> >>> Hi,
> >>>
> >>> did something change with the default log levels for OSDs on v0.45? With
> >>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I
> >>> get by default ~3Gbyte per OSD over 12 hours without any change in the
> >>> config file.
> >>>
> >>> Danny
> >> I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though!
> >>
> >> Can you comment on this, Sam?
> >> -Greg
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Log files with 0.45
2012-04-20 20:05 ` Nick Bartos
2012-04-20 20:09 ` Calvin Morrow
@ 2012-04-20 20:40 ` Greg Farnum
1 sibling, 0 replies; 6+ messages in thread
From: Greg Farnum @ 2012-04-20 20:40 UTC (permalink / raw)
To: Nick Bartos, Danny Kukawka; +Cc: ceph-devel, Danny Kukawka
I checked with Sam on this and it turns out he created a new subsystem whose output you can control with the "debug optracker" (or --debug-optracker) option (in the same way as the other debug log systems). In 0.45 the output for that system was at inappropriately high levels (1) and it's fixed in our current master (5 now), but you probably want to set "debug optracker = 0" in your config. That should restore things to the way they used to be!
(And sorry for the long wait, Danny.)
-Greg
On Friday, April 20, 2012 at 1:05 PM, Nick Bartos wrote:
> Is there a recommended log config for production systems? I'm also
> trying to decrease the verbosity in 0.45, using the options specified
> here: http://ceph.newdream.net/wiki/Debugging. Setting them down to
> '1' doesn't end the insane log sprawl I'm seeing.
>
> On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum
> <gregory.farnum@dreamhost.com (mailto:gregory.farnum@dreamhost.com)> wrote:
> > On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote:
> > > Hi,
> > >
> > > did something change with the default log levels for OSDs on v0.45? With
> > > 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I
> > > get by default ~3Gbyte per OSD over 12 hours without any change in the
> > > config file.
> > >
> > > Danny
> > I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though!
> >
> > Can you comment on this, Sam?
> > -Greg
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-04-20 20:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-04-18 4:53 Log files with 0.45 Danny Kukawka
2012-04-18 5:09 ` Greg Farnum
2012-04-20 20:05 ` Nick Bartos
2012-04-20 20:09 ` Calvin Morrow
2012-04-20 20:21 ` Sage Weil
2012-04-20 20:40 ` Greg Farnum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.