getting kernel debug output

getting kernel debug output

[Sage Weil]

I'm working on http://tracker.newdream.net/issues/3342 and was able to 
reproduce the msgr bug (some annoying msgr race I think) while generating 
full libceph debug output.  I used a teuthology yaml fragment like so:

tasks:
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds: null
- kclient: null
- exec:
    client.0:
      - echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control
- workunit:
    clients:
      all:
      - suites/ffsb.sh

I was pleasantly surprised that even though this is putting copious 
amounts of crap in dmesg it didn't slow things down enough to avoid 
tripping the bug.  And the 'dmesg' command in kdb appears to be working 
now (a couple months back it wasn't).  Yay!

Anyway, this might be useful in tracking down other bugs as well...

sage

Re: getting kernel debug output

[Alex Elder]

On 10/24/2012 12:11 PM, Sage Weil wrote:
> I'm working on http://tracker.newdream.net/issues/3342 and was able to 
> reproduce the msgr bug (some annoying msgr race I think) while generating 
> full libceph debug output.  I used a teuthology yaml fragment like so:

I have more trouble than that, but perhaps there's something
weird about having my serial console connected from 1500 miles
away.  I'm impressed full debugging didn't mess things up.

> tasks:
> - clock: null
> - ceph:
>     log-whitelist:
>     - wrongly marked me down
>     - objects unfound and apparently lost
> - thrashosds: null
> - kclient: null
> - exec:
>     client.0:
>       - echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control

This is cool, I didn't know you could do this.

> - workunit:
>     clients:
>       all:
>       - suites/ffsb.sh
> 
> I was pleasantly surprised that even though this is putting copious 
> amounts of crap in dmesg it didn't slow things down enough to avoid 
> tripping the bug.  And the 'dmesg' command in kdb appears to be working 
> now (a couple months back it wasn't).  Yay!

For me, dmesg has been working, but I'd like to know how to
truncate the output to just, say, the last 200 lines.  (Maybe
there is one.)

> Anyway, this might be useful in tracking down other bugs as well...

Yes, this is good news.

					-Alex

Re: getting kernel debug output

[David Zafman]

I also added a kcon_most teuthology task which does almost the same thing as ceph/src/script/kcon_most.sh to all or any set of clients.  The teuthology version does not raise the console log level.

For example:

tasks:
- ceph:
- kclient:
- kcon_most:
- interactive:


On Oct 24, 2012, at 11:14 AM, Alex Elder <elder@inktank.com> wrote:

> On 10/24/2012 12:11 PM, Sage Weil wrote:
>> I'm working on http://tracker.newdream.net/issues/3342 and was able to 
>> reproduce the msgr bug (some annoying msgr race I think) while generating 
>> full libceph debug output.  I used a teuthology yaml fragment like so:
> 
> I have more trouble than that, but perhaps there's something
> weird about having my serial console connected from 1500 miles
> away.  I'm impressed full debugging didn't mess things up.
> 
>> tasks:
>> - clock: null
>> - ceph:
>>    log-whitelist:
>>    - wrongly marked me down
>>    - objects unfound and apparently lost
>> - thrashosds: null
>> - kclient: null
>> - exec:
>>    client.0:
>>      - echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control
> 
> This is cool, I didn't know you could do this.
> 
>> - workunit:
>>    clients:
>>      all:
>>      - suites/ffsb.sh
>> 
>> I was pleasantly surprised that even though this is putting copious 
>> amounts of crap in dmesg it didn't slow things down enough to avoid 
>> tripping the bug.  And the 'dmesg' command in kdb appears to be working 
>> now (a couple months back it wasn't).  Yay!
> 
> For me, dmesg has been working, but I'd like to know how to
> truncate the output to just, say, the last 200 lines.  (Maybe
> there is one.)
> 
>> Anyway, this might be useful in tracking down other bugs as well...
> 
> Yes, this is good news.
> 
> 					-Alex
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: getting kernel debug output

[Sage Weil]

On Wed, 24 Oct 2012, Alex Elder wrote:
> On 10/24/2012 12:11 PM, Sage Weil wrote:
> > I'm working on http://tracker.newdream.net/issues/3342 and was able to 
> > reproduce the msgr bug (some annoying msgr race I think) while generating 
> > full libceph debug output.  I used a teuthology yaml fragment like so:
> 
> I have more trouble than that, but perhaps there's something
> weird about having my serial console connected from 1500 miles
> away.  I'm impressed full debugging didn't mess things up.

The serial doesn't have to be attached until it crashes.  And in any case, 
the output doesn't go to the console because it's pr_debug() or whatever.  

> > tasks:
> > - clock: null
> > - ceph:
> >     log-whitelist:
> >     - wrongly marked me down
> >     - objects unfound and apparently lost
> > - thrashosds: null
> > - kclient: null
> > - exec:
> >     client.0:
> >       - echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control
> 
> This is cool, I didn't know you could do this.

I just added it yesterday :)

> > - workunit:
> >     clients:
> >       all:
> >       - suites/ffsb.sh
> > 
> > I was pleasantly surprised that even though this is putting copious 
> > amounts of crap in dmesg it didn't slow things down enough to avoid 
> > tripping the bug.  And the 'dmesg' command in kdb appears to be working 
> > now (a couple months back it wasn't).  Yay!
> 
> For me, dmesg has been working, but I'd like to know how to
> truncate the output to just, say, the last 200 lines.  (Maybe
> there is one.)

Me too!  Not sure it does that..
 
> > Anyway, this might be useful in tracking down other bugs as well...
> 
> Yes, this is good news.
> 
> 					-Alex
> 
>