fh_verify: no root_squashed access hundreds of times a second again

fh_verify: no root_squashed access hundreds of times a second again

[Andrew Gray]

I'm not subscribed to the linux-kernel list, I would appreciate a CC on
any replies, but I will be watching the list as well.  I'm reposting
this message in the hope someone will answer - neither I nor the mailing
list got any replies last time.

I am using kernel 2.4.24 on a heavily-used NFS server. I am receiving
hundreds of messages like:

"kernel: fh_verify: no root_squashed access at sessions/lastsession."

in my messages log, usually accompanied by a "last message repeated 6497
times" a minute or so later. I'm gathering it is just reporting it is
denying root access to a share, which is fine and exactly what I asked
for. Is there anyway to shut this logging off without just wiping the
line from fs/nfsd/nfsfh.c? I really can't afford to be rebooting the box
to install a new kernel right now. I've searched google, linux-kernel,
and other resources, and while I've found others with the same problem,
no solutions have been posted.

-- 
Andrew Gray
Systems Administrator
College of Engineering
University of Nevada, Las Vegas

Re: fh_verify: no root_squashed access hundreds of times a second again

[Helge Hafting]

On Fri, Feb 13, 2004 at 09:18:32AM -0800, Andrew Gray wrote:
> I'm not subscribed to the linux-kernel list, I would appreciate a CC on
> any replies, but I will be watching the list as well.  I'm reposting
> this message in the hope someone will answer - neither I nor the mailing
> list got any replies last time.
> 
> I am using kernel 2.4.24 on a heavily-used NFS server. I am receiving
> hundreds of messages like:
> 
> "kernel: fh_verify: no root_squashed access at sessions/lastsession."
> 
> in my messages log, usually accompanied by a "last message repeated 6497
> times" a minute or so later. I'm gathering it is just reporting it is
> denying root access to a share, which is fine and exactly what I asked
> for. Is there anyway to shut this logging off without just wiping the
> line from fs/nfsd/nfsfh.c? 

How about tracking down whoever is trying to do all these illegal
accesses and stop them?  6000 attempts per minute seems a
waste of resources, whether malicious or ill-configured. 

Helge Hafting

Re: fh_verify: no root_squashed access hundreds of times a second again

[Andrew Gray]

> How about tracking down whoever is trying to do all these illegal
> accesses and stop them?  6000 attempts per minute seems a
> waste of resources, whether malicious or ill-configured. 

I am, of course, working this angle as well.  However, I was hoping to
find out what, specifically, is causing these messages.  Preferably, how
to stop them or mute them for the short-term.  Having the log server
rotate 4GB log files every 24 hours because the kernel is almost DOSing
itself is counterproductive.

Re: fh_verify: no root_squashed access hundreds of times a second again

[Andrew Gray]

Solution found.  At least to my particular situation.

We have a bunch of SunBlades.  As part of their login in, the dtlogin
program tries to access <homedir>/.dt/sessions/lastsession.  It is
operating as root at this point, i.e. unauthenticated as the user.

For some reason, when it makes this request, if the user's home
directory doesn't allow access, the NFS server returns that it's a NFS
Stale Handle.  For some other odd reason, when dtlogin gets this, it
just immediately retries.  This leads to the hundreds of times per
second worth of accesses we were seeing. 

Combine this with having about a dozen machines all doing it at the same
time yields the problem we were seeing.

The solution in our case was to allow world-execute permissions on the
user home directories so dt could get at the file.

While this prevents the problem from occuring, further investigation is
probably needed as to why this interaction between Solaris and the Linux
kernel NFS daemon occurs.

Posted here in hopes that it will help someone else out with this
problem.