Re: [RFC][PATCH 0/5] NFS: trace points added to mounting path

Chuck Lever
2009-01-23T18:17:15Z

On Jan 21, 2009, at Jan 21, 2009, 5:56 PM, J. Bruce Fields wrote:
> On Thu, Jan 22, 2009 at 09:36:53AM +1100, Greg Banks wrote:
>> Chuck Lever wrote:
>>>
>>>
>>> I think we need to visit this issue on a case-by-case basis.
>>> Sometimes dprintk is appropriate. Sometimes printk(KERN_ERR).
>>> Sometimes a performance metric.
>> Well said.
>>
>>> Trond has always maintained that dprintk() is best for developers,
>>> but
>>> probably inappropriate for field debugging,
>> It's not a perfect tool but it beats nothing at all.
>>> and I think that may also
>>> apply to trace points.
>> It depends on whether distros can be convinced to enable it by
>> default,
>> and install by default any necessary userspace infrastructure. The
>> most important thing for field debugging is Just Knowing that you
>> have
>> all the bits necessary to perform useful debugging without having to
>> find some RPM that matches the kernel that the machine is actually
>> running now, and not the one that was present when the machine was
>> installed.
>
> On the mount case specifically: How far are we from the idea of a
> mount
> program that can identify most problems itself? I know its error
> reporting has gotten better....

> I suppose the main feedback mount gets right now is an error code from
> the mount system call, and that may be too narrow an interface to
> cover
> most problems. Is there some way we can give mount a real interface
> it
> can use to find out this stuff instead of just dumping more strings
> into
> the logs?

A main reason it does this rather than generate error messages on the
terminal is that mount has to run in "background" environments.
Mounts done at boot time do not have a controlling terminal. A bg
mount can drop into the background, and thus it loses its controlling
terminal. Automounter doesn't have a controlling terminal to begin
with.

My feeling is that, as mount is a system tool, it should report its
problems in the system log. If there's a controlling terminal, report
it there too. But by and large it is a tool that is run most often
without direct human intervention or monitoring.

In addition there are a lot of cases it can (and does) handle by
itself. Renegotiating mount option settings is one of these things.

It's a narrow interface, but I'm not sure yet it's entirely inadequate.

> My main obstacle to judging a solution is that I don't have in mind a
> good list of (say) the top 10 problems that can cause the first
> mount to
> fail. Hm:
>
> - dns lookup of the server fails
> - server isn't reachable
> - server isn't running nfs
> - requested path isn't known to server or isn't exported
> - export is there, but requires more security
> - user doesn't have gss credentials
> - file permissions on the export are wrong
> ...

- tcp_wrappers or iptables blocking access
- network routing problems
- v2/v3 server not running rpcbind or lockd

This is exactly why I want to start with some real world examples.
Without examples we are much more likely to design something that
isn't useful to anyone. History (or e-mail archives) gives us a lot
of information about what might be common problems.

I think we handle some of these cases reasonably well today, though
they could probably stand some polish; others, like security
configuration, are still a little new and kind of a low priority (for
good or bad reasons) and so it is still a bit confusing.

But really, if mount can report a clear error message and suggest a
course of corrective action, I don't think a dprintk or trace point or
SystemTap will be of any greater help.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com