[2.4] Server stops serving nfs files (possibly ext3 or quota related)

[2.4] Server stops serving nfs files (possibly ext3 or quota related)

[Dale Blount]

Hello List,

I'm not sure where to start tracking this down so this seems like the
best place.  I have a decently loaded server with the following
configuration:

2.4.22 custom compiled kernel (all drivers static)
2 x 1.3Ghz P3 (ServerWorks Chipset)
2GB ECC DDR ram
Adaptec aacraid controller card
7 SCSI disks in a hardware array
D-link gigabit ethernet
330GB ext3 filesystem with 128MB journal (enlarged after problem
	appeared)
OS (RH7.3 currently) is on separate software MD devices and SCSI disks.
/aac0/users/home mounted on /home using -o bind

Every so often (this last time had a span of just over a week) the
server will stop writing/reading nfs files to its clients (approx 8). I
couldn't test this most recent time, but as of last time files could
still be written locally to the nfs exported array.

This has happened since install using 2.4.18-acX (-ac needed for new
style quotas at the time) and is still happening as of 2.4.22.  The
server has been upgraded to 2.4.23 but has only been up a few hours.

I'm attaching a pslist at the time of reboot.  The loadavg was somewhere
in the mid 30s.  If you'll notice, quite a few scripts are stuck in D
state.  find_over_quota.sh (spawns repquota) runs every quarter hour,
and remove_old_junkmail.sh (spawns the find stuck in D) runs 4 times per
day.  It seems heavy disk IO can cause it, but it's hard to tell.

Please do not hesitate to contact me for additional information.

Regards,

Dale Blount

Re: [2.4] Server stops serving nfs files (possibly ext3 or quota related)

[Dale Blount]

> I'm attaching a pslist at the time of reboot. 

It's been brought to my attention that I forgot to do so.  Here is a url
to the pslist.  http://dale.us/pslist

Regards,

Dale Blount

Re: [2.4] Server stops serving nfs files (possibly ext3 or quota related)

[Dale Blount]

On Sat, 2003-12-20 at 14:36, Dale Blount wrote:
> > I'm attaching a pslist at the time of reboot. 
> 
> It's been brought to my attention that I forgot to do so.  Here is a url
> to the pslist.  http://dale.us/pslist
> 
> Regards,
> 
> Dale Blount
> 

Since no one seems to have any ideas, could someone please tell me (or
link to the docs) of what I need to capture next time this happens? 
It's been pretty consistent from 2.4.18-ac4 to 2.4.22 so I'm figuring
the problem still isn't resolved in 2.4.23.  I've been checking
changelogs and nothing looks relevant.

Thanks,

Dale Blount

Re: [2.4] Server stops serving nfs files (possibly ext3 or quota related)

[Mike Fedyk]

On Wed, Dec 31, 2003 at 10:13:01AM -0500, Dale Blount wrote:
> Since no one seems to have any ideas, could someone please tell me (or
> link to the docs) of what I need to capture next time this happens? 
> It's been pretty consistent from 2.4.18-ac4 to 2.4.22 so I'm figuring
> the problem still isn't resolved in 2.4.23.  I've been checking
> changelogs and nothing looks relevant.

First I'd cc ext3-users@redhat.com

If your processes are waiting on something you'll need to show what they're
waiting in (ps axl).

Do you get any other error messages in your kernel logs?  Does your network
connection keep working when NFS serving stops?

run sysrq+t at the time you're having the problem, and run it through
ksymoops.

Oh, and for small lists it's ok to just include it in the email body.

Mike