x86 SMP and RPC/NFS problems

x86 SMP and RPC/NFS problems

[Alex Kerkhove]

Hi,


We're running a quite busy mailserver (50.000 mailboxes, 170000+ msgs a
day) with maildir 'mailboxes' on an NFS volume. The server was running
redhat 7.1 with i686 2.4.3-12smp kernel.

Ever since the machine came into full production we've had big problems
on our dell 2540 dual p3-733, 1Gb RAM machine. At least twice a day we
would see nfs server timeouts, followed by "can't get request slot"
messages completeley hanging the machine and only a reboot could get the
system going again. We've tried every cure known to man to fix this
problem (changing nics, mount params, interal buffers, etc) no luck.


But when I switched to a Single processor kernel (RH 2.4.3-12) on the
same machine the problems where instantly solved! (13 days without
problems so far)


So my (blunt?) conclusion is that there must be some serious problems
with RPC/NFS (I guess RPC) and 2.4 SMP kernels! (and lots of processes
doing NFS stuff)


Anyone any thoughts on this?  My kernel hacking knowledge is limited,
but I'm willing to test patches :)

Thanks,

Alex


Please CC: me as I'm not subscribed to this list.

Re: x86 SMP and RPC/NFS problems

[Andrew Morton]

Alex Kerkhove wrote:
> 
> Hi,
> 
> We're running a quite busy mailserver (50.000 mailboxes, 170000+ msgs a
> day) with maildir 'mailboxes' on an NFS volume. The server was running
> redhat 7.1 with i686 2.4.3-12smp kernel.
> 
> Ever since the machine came into full production we've had big problems
> on our dell 2540 dual p3-733, 1Gb RAM machine. At least twice a day we
> would see nfs server timeouts, followed by "can't get request slot"
> messages completeley hanging the machine and only a reboot could get the
> system going again. We've tried every cure known to man to fix this
> problem (changing nics, mount params, interal buffers, etc) no luck.

There were some SMP bugs in the NFS client code a while back but I
don't recall any on the server side.

Is it reproducible with 2.4.7?

What NICs have you tried?  If they were ne2k and/or 3com then
you've been bitten by the APIC bug.

-

Re: x86 SMP and RPC/NFS problems

[Trond Myklebust]

>>>>> " " == Alex Kerkhove <alex.kerkhove@staff.zeelandnet.nl> writes:

     > So my (blunt?) conclusion is that there must be some serious
     > problems with RPC/NFS (I guess RPC) and 2.4 SMP kernels! (and
     > lots of processes doing NFS stuff)


     > Anyone any thoughts on this?  My kernel hacking knowledge is
     > limited, but I'm willing to test patches :)

Could you try out the patch

  http://www.fys.uio.no/~trondmy/src/2.4.3/linux-2.4.3-rpc_smpfixes.dif

and see if it changes things?
The latter straightens out a number of iffy locking issues in the
net/sunrpc/xprt.c
Most of it will only hit you if you're doing NFS over TCP though...

Cheers,
  Trond