More issues found on kernel.org

More issues found on kernel.org

[J.H.]

Not that the current discussion on IMA, and the recent problems found
with XFS were enough, I've started seeing, rather regularly, what I've
reported in bugzilla

https://bugzilla.kernel.org/show_bug.cgi?id=20702

It looks like a double free is happening somewhere, and the issue
*SEEMS* to be limited to the dynamic web boxes (bugzilla, wiki's, etc)
and those are the only boxes I have running drbd and ocfs2.

Once the initial problem hits, the box more or less grinds to a halt and
will eventually kernel panic and reboot.  Explicitly trying to reboot
the box results in a solidly hung box requiring a hard reset.

Anyone have any thoughts, any extra debugging if/when this should happen
again that would be useful?

- John 'Warthog9' Hawley

Re: More issues found on kernel.org

[Joel Becker]

On Mon, Oct 18, 2010 at 12:15:19PM -0700, J.H. wrote:
> Not that the current discussion on IMA, and the recent problems found
> with XFS were enough, I've started seeing, rather regularly, what I've
> reported in bugzilla
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=20702
> 
> It looks like a double free is happening somewhere, and the issue
> *SEEMS* to be limited to the dynamic web boxes (bugzilla, wiki's, etc)
> and those are the only boxes I have running drbd and ocfs2.

	Obviously with no ocfs2 in the stack traces, it's hard to say
anything from that perspective.  Do you have any idea what file snmpd is
closing?

Joel

-- 

"I'm so tired of being tired,
 Sure as night will follow day.
 Most things I worry about
 Never happen anyway."

Joel Becker
Consulting Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: More issues found on kernel.org

[J.H.]

On 10/18/2010 03:15 PM, Joel Becker wrote:
> On Mon, Oct 18, 2010 at 12:15:19PM -0700, J.H. wrote:
>> Not that the current discussion on IMA, and the recent problems found
>> with XFS were enough, I've started seeing, rather regularly, what I've
>> reported in bugzilla
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=20702
>>
>> It looks like a double free is happening somewhere, and the issue
>> *SEEMS* to be limited to the dynamic web boxes (bugzilla, wiki's, etc)
>> and those are the only boxes I have running drbd and ocfs2.
> 
> 	Obviously with no ocfs2 in the stack traces, it's hard to say
> anything from that perspective.  Do you have any idea what file snmpd is
> closing?

Wasn't pointing the finger at ocfs2, or drbd for that matter, was noting
that was running on the box as those are the only two boxes with it, and
those are the boxes having issues right now.  I'm at the point where I
have no idea *WHAT* was causing the problem just trying to get as much
info out there for debugging as possible.

As to what files snmpd was closing, no idea.  I'm using snmpd both for
monitoring of the boxes, but HP's utilities are using it for a pile of
things as well, including disk monitoring and such.  Could have been
just about anything unfortunately and I'm not sure there's a good way to
trap that if/when it happens again.

If I get to that state again is there anything that would be useful
(from a debugging perspective) to snag before the box falls over, I
might be able to get some sysrq requests back if anyone would find that
helpful, and might be able to poke around a bit, not sure how far I can
get before it becomes unusable yet.

- John 'Warthog9' Hawley

Re: More issues found on kernel.org

[Robin Holt]

On Mon, Oct 18, 2010 at 12:15:19PM -0700, J.H. wrote:
> Not that the current discussion on IMA, and the recent problems found
> with XFS were enough, I've started seeing, rather regularly, what I've
> reported in bugzilla
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=20702
> 
> It looks like a double free is happening somewhere, and the issue
> *SEEMS* to be limited to the dynamic web boxes (bugzilla, wiki's, etc)
> and those are the only boxes I have running drbd and ocfs2.
> 
> Once the initial problem hits, the box more or less grinds to a halt and
> will eventually kernel panic and reboot.  Explicitly trying to reboot
> the box results in a solidly hung box requiring a hard reset.
> 
> Anyone have any thoughts, any extra debugging if/when this should happen
> again that would be useful?

If you are building your own kernel, I would turn on slab debugging.
Nearly all the double frees I have come across in the last years have
been tracked down using slab debug which traps double frees.  It does
not always point at the culprit, but at least narrows the field.

Robin