exporting to a list of IPs

exporting to a list of IPs

[Jason Keltz]

Hi.

Why is it that you cannot NFS export to a list of IPs and have exportfs 
leave the list of IPs in etab without converting over to FQDN?

In Changelog:

2001-09-20 NeilBrown <neilb@cse.unsw.edu.au>
             Anne Milicia <milicia@missioncriticallinux.com>

         * support/export/client.c (client_lookup) call gethostbyadd
         to make sure that we have a canonical hostname, even for
         dotted-quads

Why does mountd need to repeat the gethostbyname() lookup on every host 
even though exportfs already converted over to using hostname?

Thanks,

Jason.

Re: exporting to a list of IPs

[Neil Brown]

On Tue, 03 Aug 2010 11:53:07 -0400
Jason Keltz <jas@cse.yorku.ca> wrote:

> Hi.
> 
> Why is it that you cannot NFS export to a list of IPs and have exportfs 
> leave the list of IPs in etab without converting over to FQDN?

My memory is that if you only list IP addresses in /etc/exports then it will
do just want you want.  But if you list any host names or net groups then it
has to do a DNS lookup on everything to see if either of those matches.

> 
> In Changelog:
> 
> 2001-09-20 NeilBrown <neilb@cse.unsw.edu.au>
>              Anne Milicia <milicia@missioncriticallinux.com>
> 
>          * support/export/client.c (client_lookup) call gethostbyadd
>          to make sure that we have a canonical hostname, even for
>          dotted-quads

I think this patch only related to netgroup lookups.

> 
> Why does mountd need to repeat the gethostbyname() lookup on every host 
> even though exportfs already converted over to using hostname?

exportfs doesn't convert over host hostnames.  It essentially
copies /etc/exports to /var/lib/nfs/etab with minor formatting changes.

What do you actually put in /etc/exports, what do you find
in /var/lib/nfs/etab, and what exactly is the problem?

NeilBrown


> 
> Thanks,
> 
> Jason.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: exporting to a list of IPs

[Jason Keltz]

Neil Brown wrote:
> On Tue, 03 Aug 2010 11:53:07 -0400
> Jason Keltz <jas@cse.yorku.ca> wrote:
> 
>> Hi.
>>
>> Why is it that you cannot NFS export to a list of IPs and have exportfs 
>> leave the list of IPs in etab without converting over to FQDN?
> 
> My memory is that if you only list IP addresses in /etc/exports then it will
> do just want you want.  But if you list any host names or net groups then it
> has to do a DNS lookup on everything to see if either of those matches.

Hi Neil.

Thanks for your response!

Actually, if I list ONLY IPs in /etc/exports, and nothing else, then 
etab gets converted over to using hostnames:

For example:

# cat /etc/exports
/test 130.63.92.24(ro,sync)

# cat /var/lib/nfs/etab
(it's empty)
# exportfs -a
# cat /var/lib/nfs/etab
/test 
gold.cs.yorku.ca(ro,sync,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2)

In fact, I see two problems here.  First, exportfs shouldn't convert 
etab over to using FQDN.  Second, if it does do this, I don't see why 
rpc.mountd needs to RE-resolve each hostname in etab.    During this 
time (right after exportfs exits), all of my NFS shares hang, and an 
strace of rpc.mountd shows that it is re-resolving all the hostnames 
from etab.  When it finishes, all activity continues.  On one system 
with a total of 30,000 hostnames listed, this ends up in a 30 second 
hang time for all NFS exports!  I was able to shrink this time slightly 
by putting a caching name server on the NFS server, so that the NFS 
server wasn't killing the DNS, but this didn't help enough.  If exportfs 
truly has to convert IPs over to hostname, I can live with that, but 
then rpc.mountd shouldn't re-resolve the names.  If both can live with 
IPs, I'm good with that as well.

Now, albeit, I'm using an older nfs-utils that comes with RHEL4 
installation.  Compiling a later version is a bit tricky because some 
libraries have changed.  That being said, reviewing the source (given 
that I don't really know it that well) for the newest nfs-utils, I don't 
see how this behavior would really be any different.  For example, in 
client_lookup in support/export/client.c, adding some printfs, I can see 
that the IPs always get into the ...

   if (htype == MCL_FQDN && !canonical) {
... where there's a call to gethostbyname.

This is the same in nfs-utils-1.0.6 as it is in 1.2.2.

>> Why does mountd need to repeat the gethostbyname() lookup on every host 
>> even though exportfs already converted over to using hostname?
> 
> exportfs doesn't convert over host hostnames.  It essentially
> copies /etc/exports to /var/lib/nfs/etab with minor formatting changes.
> 
> What do you actually put in /etc/exports, what do you find
> in /var/lib/nfs/etab, and what exactly is the problem?
> 
> NeilBrown

See above.. thanks :)

Jason.

Re: exporting to a list of IPs

[Neil Brown]

On Wed, 04 Aug 2010 09:47:45 -0400
Jason Keltz <jas@cse.yorku.ca> wrote:

> Neil Brown wrote:
> > On Tue, 03 Aug 2010 11:53:07 -0400
> > Jason Keltz <jas@cse.yorku.ca> wrote:
> > 
> >> Hi.
> >>
> >> Why is it that you cannot NFS export to a list of IPs and have exportfs 
> >> leave the list of IPs in etab without converting over to FQDN?
> > 
> > My memory is that if you only list IP addresses in /etc/exports then it will
> > do just want you want.  But if you list any host names or net groups then it
> > has to do a DNS lookup on everything to see if either of those matches.
> 
> Hi Neil.
> 
> Thanks for your response!
> 
> Actually, if I list ONLY IPs in /etc/exports, and nothing else, then 
> etab gets converted over to using hostnames:
> 
> For example:
> 
> # cat /etc/exports
> /test 130.63.92.24(ro,sync)
> 
> # cat /var/lib/nfs/etab
> (it's empty)
> # exportfs -a
> # cat /var/lib/nfs/etab
> /test 
> gold.cs.yorku.ca(ro,sync,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2)
> 
> In fact, I see two problems here.  First, exportfs shouldn't convert 
> etab over to using FQDN.  Second, if it does do this, I don't see why 
> rpc.mountd needs to RE-resolve each hostname in etab.    During this 
> time (right after exportfs exits), all of my NFS shares hang, and an 
> strace of rpc.mountd shows that it is re-resolving all the hostnames 
> from etab.  When it finishes, all activity continues.  On one system 
> with a total of 30,000 hostnames listed, this ends up in a 30 second 
> hang time for all NFS exports!  I was able to shrink this time slightly 
> by putting a caching name server on the NFS server, so that the NFS 
> server wasn't killing the DNS, but this didn't help enough.  If exportfs 
> truly has to convert IPs over to hostname, I can live with that, but 
> then rpc.mountd shouldn't re-resolve the names.  If both can live with 
> IPs, I'm good with that as well.
> 
> Now, albeit, I'm using an older nfs-utils that comes with RHEL4 
> installation.  Compiling a later version is a bit tricky because some 
> libraries have changed.  That being said, reviewing the source (given 
> that I don't really know it that well) for the newest nfs-utils, I don't 
> see how this behavior would really be any different.  For example, in 
> client_lookup in support/export/client.c, adding some printfs, I can see 
> that the IPs always get into the ...
> 
>    if (htype == MCL_FQDN && !canonical) {
> ... where there's a call to gethostbyname.
> 
> This is the same in nfs-utils-1.0.6 as it is in 1.2.2.

True, but client_gettype is different.
In 1.0.6, w.x.y.z is treated as MCL_FQDN
In 1.2.2, w.x.y.z is treated as MCL_SUBNETWORK

If you use w.x.y.z/32 then it will be treated as MCL_SUBNETWORK and should do
what you want.

NeilBrown

> 
> >> Why does mountd need to repeat the gethostbyname() lookup on every host 
> >> even though exportfs already converted over to using hostname?
> > 
> > exportfs doesn't convert over host hostnames.  It essentially
> > copies /etc/exports to /var/lib/nfs/etab with minor formatting changes.
> > 
> > What do you actually put in /etc/exports, what do you find
> > in /var/lib/nfs/etab, and what exactly is the problem?
> > 
> > NeilBrown
> 
> See above.. thanks :)
> 
> Jason.

Re: exporting to a list of IPs

[Jason Keltz]

On Thu, 5 Aug 2010, Neil Brown wrote:

> On Wed, 04 Aug 2010 09:47:45 -0400
> Jason Keltz <jas@cse.yorku.ca> wrote:
>
>> Neil Brown wrote:
>>> On Tue, 03 Aug 2010 11:53:07 -0400
>>> Jason Keltz <jas@cse.yorku.ca> wrote:
>>>
>>>> Hi.
>>>>
>>>> Why is it that you cannot NFS export to a list of IPs and have exportfs
>>>> leave the list of IPs in etab without converting over to FQDN?
>>>
>>> My memory is that if you only list IP addresses in /etc/exports then it will
>>> do just want you want.  But if you list any host names or net groups then it
>>> has to do a DNS lookup on everything to see if either of those matches.
>>
>> Hi Neil.
>>
>> Thanks for your response!
>>
>> Actually, if I list ONLY IPs in /etc/exports, and nothing else, then
>> etab gets converted over to using hostnames:
>>
>> For example:
>>
>> # cat /etc/exports
>> /test 130.63.92.24(ro,sync)
>>
>> # cat /var/lib/nfs/etab
>> (it's empty)
>> # exportfs -a
>> # cat /var/lib/nfs/etab
>> /test
>> gold.cs.yorku.ca(ro,sync,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2)
>>
>> In fact, I see two problems here.  First, exportfs shouldn't convert
>> etab over to using FQDN.  Second, if it does do this, I don't see why
>> rpc.mountd needs to RE-resolve each hostname in etab.    During this
>> time (right after exportfs exits), all of my NFS shares hang, and an
>> strace of rpc.mountd shows that it is re-resolving all the hostnames
>> from etab.  When it finishes, all activity continues.  On one system
>> with a total of 30,000 hostnames listed, this ends up in a 30 second
>> hang time for all NFS exports!  I was able to shrink this time slightly
>> by putting a caching name server on the NFS server, so that the NFS
>> server wasn't killing the DNS, but this didn't help enough.  If exportfs
>> truly has to convert IPs over to hostname, I can live with that, but
>> then rpc.mountd shouldn't re-resolve the names.  If both can live with
>> IPs, I'm good with that as well.
>>
>> Now, albeit, I'm using an older nfs-utils that comes with RHEL4
>> installation.  Compiling a later version is a bit tricky because some
>> libraries have changed.  That being said, reviewing the source (given
>> that I don't really know it that well) for the newest nfs-utils, I don't
>> see how this behavior would really be any different.  For example, in
>> client_lookup in support/export/client.c, adding some printfs, I can see
>> that the IPs always get into the ...
>>
>>    if (htype == MCL_FQDN && !canonical) {
>> ... where there's a call to gethostbyname.
>>
>> This is the same in nfs-utils-1.0.6 as it is in 1.2.2.
>
> True, but client_gettype is different.
> In 1.0.6, w.x.y.z is treated as MCL_FQDN
> In 1.2.2, w.x.y.z is treated as MCL_SUBNETWORK
>
> If you use w.x.y.z/32 then it will be treated as MCL_SUBNETWORK and should do
> what you want.

Hi Neil,

I tried this, and it worked wonderfully! Well, sort of.  It appears that 
mountd doesn't care about the mask, since I can set it to /1, /24, or /32 
and it works...  I'm disappointed to say that after exportfs exits, my NFS 
shares still hang for around 20 seconds.  Now it seems like this is 
happening while rpc.mountd is doing a whole bunch of lstat64 calls! I 
guess that comes after what used to be the name lookups.  I just don't get 
it.  I wrote a small program to loop through 38,000 lstat64 calls in a 
matter of a second... if you could look at the lstat output here 
(http://www.cse.yorku.ca/~jas/lstat.txt), it would be really helpful .. i 
added the times so you can see how long all the lstats take.  It's crazy.

Jason.

Re: exporting to a list of IPs

[Neil Brown]

On Thu, 5 Aug 2010 21:42:09 -0400 (EDT)
Jason Keltz <jas@cse.yorku.ca> wrote:

> On Thu, 5 Aug 2010, Neil Brown wrote:
> 
> > On Wed, 04 Aug 2010 09:47:45 -0400
> > Jason Keltz <jas@cse.yorku.ca> wrote:
> >
> >> Neil Brown wrote:
> >>> On Tue, 03 Aug 2010 11:53:07 -0400
> >>> Jason Keltz <jas@cse.yorku.ca> wrote:
> >>>
> >>>> Hi.
> >>>>
> >>>> Why is it that you cannot NFS export to a list of IPs and have exportfs
> >>>> leave the list of IPs in etab without converting over to FQDN?
> >>>
> >>> My memory is that if you only list IP addresses in /etc/exports then it will
> >>> do just want you want.  But if you list any host names or net groups then it
> >>> has to do a DNS lookup on everything to see if either of those matches.
> >>
> >> Hi Neil.
> >>
> >> Thanks for your response!
> >>
> >> Actually, if I list ONLY IPs in /etc/exports, and nothing else, then
> >> etab gets converted over to using hostnames:
> >>
> >> For example:
> >>
> >> # cat /etc/exports
> >> /test 130.63.92.24(ro,sync)
> >>
> >> # cat /var/lib/nfs/etab
> >> (it's empty)
> >> # exportfs -a
> >> # cat /var/lib/nfs/etab
> >> /test
> >> gold.cs.yorku.ca(ro,sync,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2)
> >>
> >> In fact, I see two problems here.  First, exportfs shouldn't convert
> >> etab over to using FQDN.  Second, if it does do this, I don't see why
> >> rpc.mountd needs to RE-resolve each hostname in etab.    During this
> >> time (right after exportfs exits), all of my NFS shares hang, and an
> >> strace of rpc.mountd shows that it is re-resolving all the hostnames
> >> from etab.  When it finishes, all activity continues.  On one system
> >> with a total of 30,000 hostnames listed, this ends up in a 30 second
> >> hang time for all NFS exports!  I was able to shrink this time slightly
> >> by putting a caching name server on the NFS server, so that the NFS
> >> server wasn't killing the DNS, but this didn't help enough.  If exportfs
> >> truly has to convert IPs over to hostname, I can live with that, but
> >> then rpc.mountd shouldn't re-resolve the names.  If both can live with
> >> IPs, I'm good with that as well.
> >>
> >> Now, albeit, I'm using an older nfs-utils that comes with RHEL4
> >> installation.  Compiling a later version is a bit tricky because some
> >> libraries have changed.  That being said, reviewing the source (given
> >> that I don't really know it that well) for the newest nfs-utils, I don't
> >> see how this behavior would really be any different.  For example, in
> >> client_lookup in support/export/client.c, adding some printfs, I can see
> >> that the IPs always get into the ...
> >>
> >>    if (htype == MCL_FQDN && !canonical) {
> >> ... where there's a call to gethostbyname.
> >>
> >> This is the same in nfs-utils-1.0.6 as it is in 1.2.2.
> >
> > True, but client_gettype is different.
> > In 1.0.6, w.x.y.z is treated as MCL_FQDN
> > In 1.2.2, w.x.y.z is treated as MCL_SUBNETWORK
> >
> > If you use w.x.y.z/32 then it will be treated as MCL_SUBNETWORK and should do
> > what you want.
> 
> Hi Neil,
> 
> I tried this, and it worked wonderfully! Well, sort of.  It appears that 
> mountd doesn't care about the mask, since I can set it to /1, /24, or /32 
> and it works...  I'm disappointed to say that after exportfs exits, my NFS 
> shares still hang for around 20 seconds.  Now it seems like this is 
> happening while rpc.mountd is doing a whole bunch of lstat64 calls! I 
> guess that comes after what used to be the name lookups.  I just don't get 
> it.  I wrote a small program to loop through 38,000 lstat64 calls in a 
> matter of a second... if you could look at the lstat output here 
> (http://www.cse.yorku.ca/~jas/lstat.txt), it would be really helpful .. i 
> added the times so you can see how long all the lstats take.  It's crazy.
> 
>

I think it does care about the mask.

192.168.1.1/1  means any address where the first 1 bit is the same as this
address,
192.168.1.1/16 means the first 16 - so 192.168.*.*
192.168.1.1/32 means exactly 192.168.1.1, nothing else.

Your /var/lib/nfs/etab is 3 megabytes and appears to have about 18000 entries
and this takes about 24 seconds to process so 750 per second on average.
Of course it isn't just stating the files, it is also building up an internal
data structure to hold it all, and quite possibly doing linear searches to
make sure there aren't any duplicates resulting in quadratic behaviour.
If you look at the number of stats per second it starts at over 2000 and by
then end it is around 300.

Do you really need that many entries?  I note there are 22 different
directories being exported each to 858 ip addresses.  Are they really all
distinct addresses or can you use a larger subnet mask to group some of them
together?

NeilBrown