Caching negative lookups

Caching negative lookups

[Mike K]

[-- Attachment #1.1: Type: text/plain, Size: 1289 bytes --]

Hello,

Based on what I have found, autofs5 supports caching negative lookups, but
it doesn't appear to be working for me.  Although my understanding of how it
is supposed to work is very limited, so I may just be mistaken.

My client system:  Red Hat 5.2, autofs-5.0.1-0.rc2.88.x86_64,
kernel-2.6.18-92.el5.x86_64, with automounter maps in a NIS+ server (client
in niscompat mode).
NFS Server: A popular NAS appliance

auto_master snippet:
/home/dev auto_dev

By default (and a recommended setting), the NAS appliance requires that all
mounts from uid 0 come from a reserved port (0-1024) on the client.  On our
client system, repeated yp lookups for entries which don't exist will
consume all ports in 0-1024, causing other legit mount attempts to fail
since they are denied by the NAS appliance.  This does not happen on Red Hat
4 or 3 systems.

Reproducer: 'while true; do cd /home/dev/example; done'.  "watch -n 1
'netstat -an | grep TIME_WAIT | wc -l'" will show the number of sockets in
TIME_WAIT increase rapidly.

In this case, each 'cd /home/dev/example' causes a yp lookup for a directory
that doesn't exist in the NIS table, consuming a socket and leaving it in
TIME_WAIT for 60 seconds.

Is this situation supposed to be prevented with caching negative lookups?

Thanks,
Mike

[-- Attachment #1.2: Type: text/html, Size: 1425 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: Caching negative lookups

[Ian Kent]

On Mon, 2008-11-03 at 10:03 -0500, Mike K wrote:
> Hello,
> 
> Based on what I have found, autofs5 supports caching negative lookups,
> but it doesn't appear to be working for me.  Although my understanding
> of how it is supposed to work is very limited, so I may just be
> mistaken.
> 
> My client system:  Red Hat 5.2, autofs-5.0.1-0.rc2.88.x86_64,
> kernel-2.6.18-92.el5.x86_64, with automounter maps in a NIS+ server
> (client in niscompat mode).
> NFS Server: A popular NAS appliance
> 
> auto_master snippet:
> /home/dev auto_dev

Is this meant to convey meaningful information?
Is auto_dev a simple indirect map?

> 
> By default (and a recommended setting), the NAS appliance requires
> that all mounts from uid 0 come from a reserved port (0-1024) on the
> client.  On our client system, repeated yp lookups for entries which
> don't exist will consume all ports in 0-1024, causing other legit
> mount attempts to fail since they are denied by the NAS appliance.
> This does not happen on Red Hat 4 or 3 systems.
> 
> Reproducer: 'while true; do cd /home/dev/example; done'.  "watch -n 1
> 'netstat -an | grep TIME_WAIT | wc -l'" will show the number of
> sockets in TIME_WAIT increase rapidly.
> 
> In this case, each 'cd /home/dev/example' causes a yp lookup for a
> directory that doesn't exist in the NIS table, consuming a socket and
> leaving it in TIME_WAIT for 60 seconds.
> 
> Is this situation supposed to be prevented with caching negative
> lookups?

If auto_dev is a simple indirect map there should be one lookup every
negative timeout seconds.

> 
> Thanks,
> Mike
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs

Re: Caching negative lookups

[Mike K]

[-- Attachment #1.1: Type: text/plain, Size: 2334 bytes --]

Ian,

Sorry for the confusion.  Yes, auto_dev is a simple indirect map.  Since it
is a negative lookup, the entry we are looking up doesn't exist in the
auto_dev map, so I didn't bother showing any of its contents.

So for some reason it appears that negative caching does not work for the
example I previous described since there is one lookup for every "cd
/home/dev/example", not one lookup for every negative timeout seconds.

- Mike

On Thu, Nov 13, 2008 at 9:08 PM, Ian Kent <raven@themaw.net> wrote:

> On Mon, 2008-11-03 at 10:03 -0500, Mike K wrote:
> > Hello,
> >
> > Based on what I have found, autofs5 supports caching negative lookups,
> > but it doesn't appear to be working for me.  Although my understanding
> > of how it is supposed to work is very limited, so I may just be
> > mistaken.
> >
> > My client system:  Red Hat 5.2, autofs-5.0.1-0.rc2.88.x86_64,
> > kernel-2.6.18-92.el5.x86_64, with automounter maps in a NIS+ server
> > (client in niscompat mode).
> > NFS Server: A popular NAS appliance
> >
> > auto_master snippet:
> > /home/dev auto_dev
>
> Is this meant to convey meaningful information?
> Is auto_dev a simple indirect map?
>
> >
> > By default (and a recommended setting), the NAS appliance requires
> > that all mounts from uid 0 come from a reserved port (0-1024) on the
> > client.  On our client system, repeated yp lookups for entries which
> > don't exist will consume all ports in 0-1024, causing other legit
> > mount attempts to fail since they are denied by the NAS appliance.
> > This does not happen on Red Hat 4 or 3 systems.
> >
> > Reproducer: 'while true; do cd /home/dev/example; done'.  "watch -n 1
> > 'netstat -an | grep TIME_WAIT | wc -l'" will show the number of
> > sockets in TIME_WAIT increase rapidly.
> >
> > In this case, each 'cd /home/dev/example' causes a yp lookup for a
> > directory that doesn't exist in the NIS table, consuming a socket and
> > leaving it in TIME_WAIT for 60 seconds.
> >
> > Is this situation supposed to be prevented with caching negative
> > lookups?
>
> If auto_dev is a simple indirect map there should be one lookup every
> negative timeout seconds.
>
> >
> > Thanks,
> > Mike
> > _______________________________________________
> > autofs mailing list
> > autofs@linux.kernel.org
> > http://linux.kernel.org/mailman/listinfo/autofs
>
>

[-- Attachment #1.2: Type: text/html, Size: 3096 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: Caching negative lookups

[Ian Kent]

On Fri, 2008-11-14 at 08:27 -0500, Mike K wrote:
> Ian, 
> 
> Sorry for the confusion.  Yes, auto_dev is a simple indirect map.
> Since it is a negative lookup, the entry we are looking up doesn't
> exist in the auto_dev map, so I didn't bother showing any of its
> contents.
> 
> So for some reason it appears that negative caching does not work for
> the example I previous described since there is one lookup for every
> "cd /home/dev/example", not one lookup for every negative timeout
> seconds.

Provide a complete debug log showing the problem.
See http://people.redhat.com/jmoyer for setup info.

> 
> - Mike
> 
> On Thu, Nov 13, 2008 at 9:08 PM, Ian Kent <raven@themaw.net> wrote:
>         On Mon, 2008-11-03 at 10:03 -0500, Mike K wrote:
>         > Hello,
>         >
>         > Based on what I have found, autofs5 supports caching
>         negative lookups,
>         > but it doesn't appear to be working for me.  Although my
>         understanding
>         > of how it is supposed to work is very limited, so I may just
>         be
>         > mistaken.
>         >
>         > My client system:  Red Hat 5.2,
>         autofs-5.0.1-0.rc2.88.x86_64,
>         > kernel-2.6.18-92.el5.x86_64, with automounter maps in a NIS+
>         server
>         > (client in niscompat mode).
>         > NFS Server: A popular NAS appliance
>         >
>         > auto_master snippet:
>         > /home/dev auto_dev
>         
>         
>         Is this meant to convey meaningful information?
>         Is auto_dev a simple indirect map?
>         
>         >
>         > By default (and a recommended setting), the NAS appliance
>         requires
>         > that all mounts from uid 0 come from a reserved port
>         (0-1024) on the
>         > client.  On our client system, repeated yp lookups for
>         entries which
>         > don't exist will consume all ports in 0-1024, causing other
>         legit
>         > mount attempts to fail since they are denied by the NAS
>         appliance.
>         > This does not happen on Red Hat 4 or 3 systems.
>         >
>         > Reproducer: 'while true; do cd /home/dev/example; done'.
>          "watch -n 1
>         > 'netstat -an | grep TIME_WAIT | wc -l'" will show the number
>         of
>         > sockets in TIME_WAIT increase rapidly.
>         >
>         > In this case, each 'cd /home/dev/example' causes a yp lookup
>         for a
>         > directory that doesn't exist in the NIS table, consuming a
>         socket and
>         > leaving it in TIME_WAIT for 60 seconds.
>         >
>         > Is this situation supposed to be prevented with caching
>         negative
>         > lookups?
>         
>         
>         If auto_dev is a simple indirect map there should be one
>         lookup every
>         negative timeout seconds.
>         
>         >
>         > Thanks,
>         > Mike
>         > _______________________________________________
>         > autofs mailing list
>         > autofs@linux.kernel.org
>         > http://linux.kernel.org/mailman/listinfo/autofs
>         
> 

Re: Caching negative lookups

[Ian Kent]

On Fri, 2008-11-14 at 22:44 +0900, Ian Kent wrote:
> On Fri, 2008-11-14 at 08:27 -0500, Mike K wrote:
> > Ian, 
> > 
> > Sorry for the confusion.  Yes, auto_dev is a simple indirect map.
> > Since it is a negative lookup, the entry we are looking up doesn't
> > exist in the auto_dev map, so I didn't bother showing any of its
> > contents.
> > 
> > So for some reason it appears that negative caching does not work for
> > the example I previous described since there is one lookup for every
> > "cd /home/dev/example", not one lookup for every negative timeout
> > seconds.
> 
> Provide a complete debug log showing the problem.
> See http://people.redhat.com/jmoyer for setup info.

No need.
I see what I've done wrong.
The cache ins't updated if the entry never existed.
Give some time to sort this out.

> 
> > 
> > - Mike
> > 
> > On Thu, Nov 13, 2008 at 9:08 PM, Ian Kent <raven@themaw.net> wrote:
> >         On Mon, 2008-11-03 at 10:03 -0500, Mike K wrote:
> >         > Hello,
> >         >
> >         > Based on what I have found, autofs5 supports caching
> >         negative lookups,
> >         > but it doesn't appear to be working for me.  Although my
> >         understanding
> >         > of how it is supposed to work is very limited, so I may just
> >         be
> >         > mistaken.
> >         >
> >         > My client system:  Red Hat 5.2,
> >         autofs-5.0.1-0.rc2.88.x86_64,
> >         > kernel-2.6.18-92.el5.x86_64, with automounter maps in a NIS+
> >         server
> >         > (client in niscompat mode).
> >         > NFS Server: A popular NAS appliance
> >         >
> >         > auto_master snippet:
> >         > /home/dev auto_dev
> >         
> >         
> >         Is this meant to convey meaningful information?
> >         Is auto_dev a simple indirect map?
> >         
> >         >
> >         > By default (and a recommended setting), the NAS appliance
> >         requires
> >         > that all mounts from uid 0 come from a reserved port
> >         (0-1024) on the
> >         > client.  On our client system, repeated yp lookups for
> >         entries which
> >         > don't exist will consume all ports in 0-1024, causing other
> >         legit
> >         > mount attempts to fail since they are denied by the NAS
> >         appliance.
> >         > This does not happen on Red Hat 4 or 3 systems.
> >         >
> >         > Reproducer: 'while true; do cd /home/dev/example; done'.
> >          "watch -n 1
> >         > 'netstat -an | grep TIME_WAIT | wc -l'" will show the number
> >         of
> >         > sockets in TIME_WAIT increase rapidly.
> >         >
> >         > In this case, each 'cd /home/dev/example' causes a yp lookup
> >         for a
> >         > directory that doesn't exist in the NIS table, consuming a
> >         socket and
> >         > leaving it in TIME_WAIT for 60 seconds.
> >         >
> >         > Is this situation supposed to be prevented with caching
> >         negative
> >         > lookups?
> >         
> >         
> >         If auto_dev is a simple indirect map there should be one
> >         lookup every
> >         negative timeout seconds.
> >         
> >         >
> >         > Thanks,
> >         > Mike
> >         > _______________________________________________
> >         > autofs mailing list
> >         > autofs@linux.kernel.org
> >         > http://linux.kernel.org/mailman/listinfo/autofs
> >         
> > 

Re: Caching negative lookups

[Ondrej Valousek]

I do not understand it - I have heard that using indirect maps can cause
unwanted NFS chattering on the network and that negative lookups are
supposed to handle this.
But this chattering I can only imagine when using wildcards like
/home auto.home
with auto.home containing something like:
* nfsserver:/vol/vol0/users/&

This way, if an application check for the existence of say /home/file -
the automounter must ask nfsserver for existence of this file every time
the application asks. But if there are no wildcards in the indirect map
and all valid entries are explicitly listed, no nfs chattering occurs as
autofs knows directly that the mount attempt for  /home/file is invalid
as there is no "file" record in the auto.home map.

Am I right? If yes, that would be a serious argument against using
wildcards in the automount maps.....
Ondrej

Re: Caching negative lookups

[Ian Kent]

On Fri, 2008-11-14 at 16:36 +0100, Ondrej Valousek wrote:
> I do not understand it - I have heard that using indirect maps can cause
> unwanted NFS chattering on the network and that negative lookups are
> supposed to handle this.
> But this chattering I can only imagine when using wildcards like
> /home auto.home
> with auto.home containing something like:
> * nfsserver:/vol/vol0/users/&
> 
> This way, if an application check for the existence of say /home/file -
> the automounter must ask nfsserver for existence of this file every time
> the application asks. But if there are no wildcards in the indirect map
> and all valid entries are explicitly listed, no nfs chattering occurs as
> autofs knows directly that the mount attempt for  /home/file is invalid
> as there is no "file" record in the auto.home map.
> 
> Am I right? If yes, that would be a serious argument against using
> wildcards in the automount maps.....

Not really.

The issue that Mike is concerned about is the actual check against the
yp map for the key consuming sockets. Given that when an application
attempts to access a directory in an indirect map the kernel is
obligated to send a request to the daemon. If the daemon isn't properly
recording previous mount fails (and in this case also recording requests
for non existent keys) and continually trying to mount or find the key
then there can be quite a bit of traffic. This is the whole idea of the
timeout for negative (failed) requests

Ian

Re: Caching negative lookups

[Ian Kent]

On Mon, 2008-11-03 at 10:03 -0500, Mike K wrote:
> Hello,
> 
> Based on what I have found, autofs5 supports caching negative lookups,
> but it doesn't appear to be working for me.  Although my understanding
> of how it is supposed to work is very limited, so I may just be
> mistaken.
> 
> My client system:  Red Hat 5.2, autofs-5.0.1-0.rc2.88.x86_64,
> kernel-2.6.18-92.el5.x86_64, with automounter maps in a NIS+ server
> (client in niscompat mode).
> NFS Server: A popular NAS appliance
> 
> auto_master snippet:
> /home/dev auto_dev
> 
> By default (and a recommended setting), the NAS appliance requires
> that all mounts from uid 0 come from a reserved port (0-1024) on the
> client.  On our client system, repeated yp lookups for entries which
> don't exist will consume all ports in 0-1024, causing other legit
> mount attempts to fail since they are denied by the NAS appliance.
> This does not happen on Red Hat 4 or 3 systems.
> 
> Reproducer: 'while true; do cd /home/dev/example; done'.  "watch -n 1
> 'netstat -an | grep TIME_WAIT | wc -l'" will show the number of
> sockets in TIME_WAIT increase rapidly.
> 
> In this case, each 'cd /home/dev/example' causes a yp lookup for a
> directory that doesn't exist in the NIS table, consuming a socket and
> leaving it in TIME_WAIT for 60 seconds.
> 
> Is this situation supposed to be prevented with caching negative
> lookups?

I have committed a patch for this to the autofs git repository and
uploaded the patch to kernel.org. So it is against version 5.0.4 of
autofs.

You may be able to view RHEL bug 469387, in which case you could
contribute to the resolution of this issue in RHEL.

Ian

Re: Caching negative lookups

[Ian Kent]

On Tue, 2008-12-09 at 15:11 +0900, Ian Kent wrote:
> > 
> > Is this situation supposed to be prevented with caching negative
> > lookups?
> 
> I have committed a patch for this to the autofs git repository and
> uploaded the patch to kernel.org. So it is against version 5.0.4 of
> autofs.
> 
> You may be able to view RHEL bug 469387, in which case you could
> contribute to the resolution of this issue in RHEL.

Oh .. sorry, you have already done so.

> 
> Ian
>