Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Stefan Richter]

Holger Hoffstaette wrote at LKML:
> On Mon, 08 Dec 2008 08:34:22 +0100, Rafael J. Wysocki wrote:
> 
>> On Monday, 8 of December 2008, Holger Hoffstaette wrote:
>>> Hi,
>>> 
>>> I just encountered a nasty symptom for the second time that has started to
>>> occur after updating my home server from vanilla 2.6.27.7 to .8 (same
>>> config).
>>> 
>>> A while after disconnecting a samba client, the smbd samba server
>>> process goes crazy and consumes 100% CPU. From that time on it is
>>> unkillable (kill -9 returns but the process continues to run). The only
>>> recourse is reboot, which works without problem (i.e. unmounting the
>>> served filesystems is apparently possible?). I tried to attach to the
>>> process with gdb but that just hung.
>>> 
>>> The system is a generic old single-core P4 box with a single SATA drive,
>>> Gentoo userland and Samba is 3.0.33 (in async mode). The kernel has no
>>> patches or binary drivers. It has been rock solid before the update and
>>> shows no other signs of weirdness in logs or otherwise. I downgraded to .7
>>> for now and will see what happens, but since it worked before I am certain
>>> that this is a regression in the .8 release.
>>> 
>>> The only commonality is a log entry by samba that seems to correlate with
>>> both occurrences:
>>> 
>>> [2008/12/08 01:02:52, 0] lib/util_sock.c:read_data(534)
>>>   read_data: read failure for 4 bytes to client 192.168.100.128. Error = No route to host
>>> 
>>> .128 is the Windows client machine (connected via a stable GigE link),
>>> which I shut down pretty much exactly 30 minutes before that (any 30
>>> minute timeouts in the kernel/network stack?). Both instances of these log
>>> entries correlate with the CPU spikes which I noticed in my MRTG graphs.
>>> 
>>> Any suspects or ideas?
>>> 
>>> thanks
>>> Holger
>> 
>> Please bisect.
> 
> I would love to try, but this is my "production server" (i.e. I need it
> for real work) and I'll be traveling the next few days. I will try to
> bisect after that (if nobody else has any ideas) but will have to make
> sure the bug is actually reproducible after the timeout - for now I only
> observed it by accident (via mrtg).
> In the meantime maybe someone else will observe it as well.
> 
> thanks
> Holger
> 

Added Cc: netdev, readded all other Cc's, quoted in full for netdev.
Good luck,
-- 
Stefan Richter
-=====-==--- ==-- -=---
http://arcgraph.de/sr/

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Stefan Richter]

>>> On Monday, 8 of December 2008, Holger Hoffstaette wrote:
>>>> The system is a generic old single-core P4 box with a single SATA drive,
>>>> Gentoo userland and Samba is 3.0.33 (in async mode). The kernel has no
>>>> patches or binary drivers.

Holger, it may be unrelated to the issue, but to be sure:  Which network
card driver do you use?
-- 
Stefan Richter
-=====-==--- ==-- -=---
http://arcgraph.de/sr/

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Holger Hoffstaette]

On Mon, 08 Dec 2008 20:19:37 +0100, Stefan Richter wrote:

>>>> On Monday, 8 of December 2008, Holger Hoffstaette wrote:
>>>>> The system is a generic old single-core P4 box with a single SATA
>>>>> drive, Gentoo userland and Samba is 3.0.33 (in async mode). The
>>>>> kernel has no patches or binary drivers.
> 
> Holger, it may be unrelated to the issue, but to be sure:  Which network
> card driver do you use?

e1000 with the older PCI/PCI-X 82545GM rev.04 card in a PCI slot.

thanks,
Holger

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Jan Rekorajski]

On Mon, 08 Dec 2008, Stefan Richter wrote:

> >>> On Monday, 8 of December 2008, Holger Hoffstaette wrote:
> >>>> The system is a generic old single-core P4 box with a single SATA drive,
> >>>> Gentoo userland and Samba is 3.0.33 (in async mode). The kernel has no
> >>>> patches or binary drivers.
> 
> Holger, it may be unrelated to the issue, but to be sure:  Which network
> card driver do you use?

I think you can safely rule out NIC, I'm also seeing this behaviour on a
brand new server with imap hanging in some busy-loop.
Network card in my case:
Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

What I observer was one CPU doing 100% system work, and the number of
timer interrupts went from 1k per second to 4k (for the whole system).

I didn't report it because I thought it's one of patches I have to blame.
Oh, and, unfortunately, I can't bisect, I'm seeing this only on one machine
that has to be running.

Jan
-- 
Jan Rekorajski            |  ALL SUSPECTS ARE GUILTY. PERIOD!
baggins<at>mimuw.edu.pl   |  OTHERWISE THEY WOULDN'T BE SUSPECTS, WOULD THEY?
BOFH, MANIAC              |                   -- TROOPS by Kevin Rubio

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Chuck Ebbert]

On Mon, 8 Dec 2008 23:22:46 +0100
Jan Rekorajski <baggins@sith.mimuw.edu.pl> wrote:

> On Mon, 08 Dec 2008, Stefan Richter wrote:
> 
> > >>> On Monday, 8 of December 2008, Holger Hoffstaette wrote:
> > >>>> The system is a generic old single-core P4 box with a single SATA drive,
> > >>>> Gentoo userland and Samba is 3.0.33 (in async mode). The kernel has no
> > >>>> patches or binary drivers.
> > 
> > Holger, it may be unrelated to the issue, but to be sure:  Which network
> > card driver do you use?
> 
> I think you can safely rule out NIC, I'm also seeing this behaviour on a
> brand new server with imap hanging in some busy-loop.
> Network card in my case:
> Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
> 
> What I observer was one CPU doing 100% system work, and the number of
> timer interrupts went from 1k per second to 4k (for the whole system).
> 

Try reverting the idr patch that went into 2.6.27.8. It broke DRM in the
Fedora kernel at least.

http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git;a=blob_plain;f=releases/2.6.27.8/lib-idr.c-fix-rcu-related-race-with-idr_find.patch;h=b1145766fb9460a0c0285350b49216355c5b4ad8

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Manfred Spraul]

Chuck Ebbert wrote:
> Try reverting the idr patch that went into 2.6.27.8. It broke DRM in the
> Fedora kernel at least.
>
>   
What happens?
Does it oops, does one of the BUG() statements trigger?

--
    Manfred

Re: Nasty regression from .27.7 to .27.8: idle samba goes crazy

[Chuck Ebbert]

On Tue, 09 Dec 2008 20:16:34 +0100
Manfred Spraul <manfred@colorfullife.com> wrote:

> Chuck Ebbert wrote:
> > Try reverting the idr patch that went into 2.6.27.8. It broke DRM in the
> > Fedora kernel at least.
> >
> >   
> What happens?
> Does it oops, does one of the BUG() statements trigger?
> 

It fails in strange ways, e.g. trying to open a DRM device causes it to
disappear. (And DRM is a heavy user of idr.)