Debugging mount failure with netgroup

Debugging mount failure with netgroup

[Tom Haynes]

Trying to mount:

[root@wont ~]# mount -o vers=3,hard,intr adept:/fooper /t/adept
mount.nfs: access denied by server while mounting adept:/fooper
[root@wont ~]# uname -a
Linux wont.internal.excfb.com 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@wont ~]# showmount -e adept
Export list for adept:
/fooper @adeptya

The server tells us:

[root@adept log]# more /etc/exports 
/fooper   @titan(rw)
[root@adept log]# vi /etc/exports 
[root@adept log]# more /etc/exports 
/fooper   @adeptya(rw)
[root@adept log]# exportfs -au
[root@adept log]# exportfs -a
[root@adept log]# exportfs
/fooper       	@adeptya
[root@adept log]# ypmatch adeptya netgroup
(adept,,) (adept.internal.excfb.com,,) (wont,,) (wont.internal.excfb.com,,)
[root@adept log]# host wont.internal.excfb.com
wont.internal.excfb.com has address 192.168.2.105
[root@adept log]# host 192.168.2.105
105.2.168.192.in-addr.arpa domain name pointer wont.internal.excfb.com.
[root@adept log]# tail messages
Feb  7 13:33:41 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:34:58 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:36:02 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:36:32 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:37:14 adept mountd[13050]: authenticated mount request from 192.168.2.105:698 for /fooper (/fooper)
Feb  7 13:38:12 adept mountd[13050]: authenticated unmount request from 192.168.2.105:657 for /fooper (/fooper)
Feb  7 13:39:06 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:43:52 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:45:24 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:48:17 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host

The success at 13:37:14 was from when I changed the export to:

[root@adept ~]# exportfs
/fooper       	192.168.2.0/255.255.255.0

to verify that NFS was working. At that point I was able to mount,
create a file, etc.

Oh yeah, the NFS server is also the NIS and DNS server, so there
shouldn't be any connectivity issues.

What can I do to further triage this?


Thanks!




-- 
Tom Haynes
ex-cfb

Re: Debugging mount failure with netgroup

[J. Bruce Fields]

On Mon, Feb 07, 2011 at 01:56:09PM -0600, Tom Haynes wrote:
> Trying to mount:
> 
> [root@wont ~]# mount -o vers=3,hard,intr adept:/fooper /t/adept
> mount.nfs: access denied by server while mounting adept:/fooper
> [root@wont ~]# uname -a
> Linux wont.internal.excfb.com 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
> [root@wont ~]# showmount -e adept
> Export list for adept:
> /fooper @adeptya
> 
> The server tells us:
> 
> [root@adept log]# more /etc/exports 
> /fooper   @titan(rw)
> [root@adept log]# vi /etc/exports 
> [root@adept log]# more /etc/exports 
> /fooper   @adeptya(rw)
> [root@adept log]# exportfs -au
> [root@adept log]# exportfs -a
> [root@adept log]# exportfs
> /fooper       	@adeptya
> [root@adept log]# ypmatch adeptya netgroup
> (adept,,) (adept.internal.excfb.com,,) (wont,,) (wont.internal.excfb.com,,)
> [root@adept log]# host wont.internal.excfb.com
> wont.internal.excfb.com has address 192.168.2.105
> [root@adept log]# host 192.168.2.105
> 105.2.168.192.in-addr.arpa domain name pointer wont.internal.excfb.com.
> [root@adept log]# tail messages
> Feb  7 13:33:41 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:34:58 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:36:02 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:36:32 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:37:14 adept mountd[13050]: authenticated mount request from 192.168.2.105:698 for /fooper (/fooper)
> Feb  7 13:38:12 adept mountd[13050]: authenticated unmount request from 192.168.2.105:657 for /fooper (/fooper)
> Feb  7 13:39:06 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:43:52 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:45:24 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> Feb  7 13:48:17 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
> 
> The success at 13:37:14 was from when I changed the export to:
> 
> [root@adept ~]# exportfs
> /fooper       	192.168.2.0/255.255.255.0
> 
> to verify that NFS was working. At that point I was able to mount,
> create a file, etc.
> 
> Oh yeah, the NFS server is also the NIS and DNS server, so there
> shouldn't be any connectivity issues.
> 
> What can I do to further triage this?

Alas, I've never used netgroups, much less tried to debug problems with
them.... Have you tried looking at the nis traffic over "lo" to see if
mountd's doing the right queries?  You could also try seeing if adding
"-dall" to the rpc.mountd commandline turns up any useful debugging.

After that, I don't know, I suppose I'd start looking at the mountd
source....

--b.

Re: Debugging mount failure with netgroup

[Tom Haynes]

On Mon, Feb 07, 2011 at 03:12:02PM -0500, J. Bruce Fields wrote:
> > What can I do to further triage this?
> 
> Alas, I've never used netgroups, much less tried to debug problems with
> them.... Have you tried looking at the nis traffic over "lo" to see if
> mountd's doing the right queries?  You could also try seeing if adding
> "-dall" to the rpc.mountd commandline turns up any useful debugging.

[tdh@adept ~]> sudo tshark -i lo
Running as user "root" and group "root". This could be dangerous.
Capturing on lo
  0.000000 192.168.2.108 -> 192.168.2.108 DNS Standard query PTR 105.2.168.192.in-addr.arpa
  0.002862 192.168.2.108 -> 192.168.2.108 DNS Standard query response PTR wont.internal.excfb.com
  0.003455 192.168.2.108 -> 192.168.2.108 DNS Standard query A wont.internal.excfb.com
  0.003499 192.168.2.108 -> 192.168.2.108 DNS Standard query AAAA wont.internal.excfb.com
  0.003819 192.168.2.108 -> 192.168.2.108 DNS Standard query response A 192.168.2.105
  0.003939 192.168.2.108 -> 192.168.2.108 DNS Standard query response
  0.004427 192.168.2.108 -> 192.168.2.108 DNS Standard query A wont.internal.excfb.com
  0.004761 192.168.2.108 -> 192.168.2.108 DNS Standard query response A 192.168.2.105
  7.472292 192.168.2.108 -> 192.168.2.108 YPSERV V2 DOMAIN Call
  7.472413 192.168.2.108 -> 192.168.2.108 YPSERV V2 DOMAIN Reply (Call In 9)

The above seems to suggest that the host lookup went fine.

-dall:

Feb  7 14:35:27 adept mountd[13285]: check_default: access by 192.168.2.105 ALLOWED (cached)
Feb  7 14:35:27 adept mountd[13285]: Received NULL request from 192.168.2.105
Feb  7 14:35:27 adept mountd[13285]: check_default: access by 192.168.2.105 ALLOWED (cached)
Feb  7 14:35:27 adept mountd[13285]: Received NULL request from 192.168.2.105
Feb  7 14:35:27 adept mountd[13285]: check_default: access by 192.168.2.105 ALLOWED (cached)
Feb  7 14:35:27 adept mountd[13285]: Received MNT3(/fooper) request from 192.168.2.105
Feb  7 14:35:27 adept mountd[13285]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host



> 
> After that, I don't know, I suppose I'd start looking at the mountd
> source....

Yeah, I'm not sure if I can do that. :->

I did google and did not find much on netgroups - but I did find
a bunch on "unmatched host".

It seems to be black magic from some of the answers.

And I agree:

1) Modified the export:
[root@adept /]# more /etc/exports 
/fooper   @adeptya(rw,insecure,no_root_squash,sync)

2) Did other mumbo-jumbo

3) Really, really did not want to reboot.

4) service nfs stop ; exportfs ; service nfs start

And that last one worked!

Feb  7 14:44:31 adept mountd[13712]: Received MNT3(/fooper) request from 192.168.2.105
Feb  7 14:44:31 adept mountd[13712]: authenticated mount request from wont.internal.excfb.com:972 for /fooper (/fooper)
Feb  7 14:44:31 adept mountd[13712]: nfsd_fh: inbuf '@adeptya 7 \x0100060000000000a9a3be796f4c43878b99ea75025e5384'
Feb  7 14:44:31 adept mountd[13712]: nfsd_fh: found 0x20da950 path /fooper

Thanks

> 
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Tom Haynes
ex-cfb

Re: Debugging mount failure with netgroup

[J. Bruce Fields]

On Mon, Feb 07, 2011 at 02:53:06PM -0600, Tom Haynes wrote:
> I did google and did not find much on netgroups - but I did find
> a bunch on "unmatched host".
> 
> It seems to be black magic from some of the answers.
> 
> And I agree:
> 
> 1) Modified the export:
> [root@adept /]# more /etc/exports 
> /fooper   @adeptya(rw,insecure,no_root_squash,sync)

So it was probably the "insecure"?

Are we returning the right error to the mount client in that case?

And I suppose the logged error should be more helpful.  I'm not really
sure mountd should be logging every mount failure by default, but if you
request it, it should at be able to tell you the reason for the failure.

--b.

> 
> 2) Did other mumbo-jumbo
> 
> 3) Really, really did not want to reboot.
> 
> 4) service nfs stop ; exportfs ; service nfs start
> 
> And that last one worked!
> 
> Feb  7 14:44:31 adept mountd[13712]: Received MNT3(/fooper) request from 192.168.2.105
> Feb  7 14:44:31 adept mountd[13712]: authenticated mount request from wont.internal.excfb.com:972 for /fooper (/fooper)
> Feb  7 14:44:31 adept mountd[13712]: nfsd_fh: inbuf '@adeptya 7 \x0100060000000000a9a3be796f4c43878b99ea75025e5384'
> Feb  7 14:44:31 adept mountd[13712]: nfsd_fh: found 0x20da950 path /fooper
> 
> Thanks
> 
> > 
> > --b.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -- 
> Tom Haynes
> ex-cfb

Re: Debugging mount failure with netgroup

[Tom Haynes]

On Mon, Feb 07, 2011 at 04:19:17PM -0500, J. Bruce Fields wrote:
> On Mon, Feb 07, 2011 at 02:53:06PM -0600, Tom Haynes wrote:
> > I did google and did not find much on netgroups - but I did find
> > a bunch on "unmatched host".
> > 
> > It seems to be black magic from some of the answers.
> > 
> > And I agree:
> > 
> > 1) Modified the export:
> > [root@adept /]# more /etc/exports 
> > /fooper   @adeptya(rw,insecure,no_root_squash,sync)
> 
> So it was probably the "insecure"?

Sorry, I forgot to mention after every step that I tried
the mount and it failed. It was only restarting nfs
that worked.

I'd have to check to see what port number the requests
are coming in on, but I don't think it is this.

And I've actually got the system horked up right now -
I restarted one time too many. Frustrating as there isn't
anything logged beyond "unmatched host".

> 
> Are we returning the right error to the mount client in that case?
> 
> And I suppose the logged error should be more helpful.  I'm not really
> sure mountd should be logging every mount failure by default, but if you
> request it, it should at be able to tell you the reason for the failure.
> 
> --b.
> 

-- 
Tom Haynes
ex-cfb

Re: Debugging mount failure with netgroup

[J. Bruce Fields]

On Mon, Feb 07, 2011 at 03:53:44PM -0600, Tom Haynes wrote:
> On Mon, Feb 07, 2011 at 04:19:17PM -0500, J. Bruce Fields wrote:
> > On Mon, Feb 07, 2011 at 02:53:06PM -0600, Tom Haynes wrote:
> > > I did google and did not find much on netgroups - but I did find
> > > a bunch on "unmatched host".
> > > 
> > > It seems to be black magic from some of the answers.
> > > 
> > > And I agree:
> > > 
> > > 1) Modified the export:
> > > [root@adept /]# more /etc/exports 
> > > /fooper   @adeptya(rw,insecure,no_root_squash,sync)
> > 
> > So it was probably the "insecure"?
> 
> Sorry, I forgot to mention after every step that I tried
> the mount and it failed. It was only restarting nfs
> that worked.

Changes to /etc/exports aren't automatically detected, so you need a
server restart or an exportfs -ra after any change.

> I'd have to check to see what port number the requests
> are coming in on, but I don't think it is this.
> 
> And I've actually got the system horked up right now -
> I restarted one time too many. Frustrating as there isn't
> anything logged beyond "unmatched host".

I forgot you said that was the error.  And looking at the code we
actually do have a different message for the port number case ("illegal
port").

OK, so maybe the problem was just that mountd never found out about your
original /etc/exports file?

--b.

> 
> > 
> > Are we returning the right error to the mount client in that case?
> > 
> > And I suppose the logged error should be more helpful.  I'm not really
> > sure mountd should be logging every mount failure by default, but if you
> > request it, it should at be able to tell you the reason for the failure.
> > 
> > --b.
> > 
> 
> -- 
> Tom Haynes
> ex-cfb

Re: Debugging mount failure with netgroup

[Tom Haynes]

On Mon, Feb 07, 2011 at 04:59:35PM -0500, J. Bruce Fields wrote:
> 
> Changes to /etc/exports aren't automatically detected, so you need a
> server restart or an exportfs -ra after any change.

Yep:

exportfs -au  # unload old rules
exportfs      # verify it is empty
exportfs -a   # load it up
exportfs      # verify it loaded

So I would have expected it to have reloaded.


> 
> OK, so maybe the problem was just that mountd never found out about your
> original /etc/exports file?

That wouldn't explain why it has stopped working now.

Neither the exportfs commands above nor a service nfs stop/start
yields a working mount from the client.

Restarting nfsd with an export of:

[root@adept internal.excfb.com]# exportfs
/fooper       	192.168.2.0/255.255.255.0
[root@adept internal.excfb.com]# uname -a
Linux adept.internal.excfb.com 2.6.34.7-66.fc13.i686.PAE #1 SMP Wed Dec 15 07:21:49 UTC 2010 i686 i686 i386 GNU/Linux

Does yield a mount.

> 
> --b.
> 

-- 
Tom Haynes
ex-cfb

Re: Debugging mount failure with netgroup

[J. Bruce Fields]

On Mon, Feb 07, 2011 at 05:21:10PM -0600, Tom Haynes wrote:
> On Mon, Feb 07, 2011 at 04:59:35PM -0500, J. Bruce Fields wrote:
> > 
> > Changes to /etc/exports aren't automatically detected, so you need a
> > server restart or an exportfs -ra after any change.
> 
> Yep:
> 
> exportfs -au  # unload old rules
> exportfs      # verify it is empty
> exportfs -a   # load it up
> exportfs      # verify it loaded
> 
> So I would have expected it to have reloaded.

Sorry, and you said that in your original email.  OK.

> > OK, so maybe the problem was just that mountd never found out about your
> > original /etc/exports file?
> 
> That wouldn't explain why it has stopped working now.
> 
> Neither the exportfs commands above nor a service nfs stop/start
> yields a working mount from the client.
> 
> Restarting nfsd with an export of:
> 
> [root@adept internal.excfb.com]# exportfs
> /fooper       	192.168.2.0/255.255.255.0
> [root@adept internal.excfb.com]# uname -a
> Linux adept.internal.excfb.com 2.6.34.7-66.fc13.i686.PAE #1 SMP Wed Dec 15 07:21:49 UTC 2010 i686 i686 i386 GNU/Linux
> 
> Does yield a mount.

And it's failing in the same way?  ("unmatched host"?)

Like I say, I suppose my next step would be to get out the nfs-utils
source and figure out where it's failing (probably somewhere in
auth_authenticate_newcache -> get_client_hostname -> client_compose ->
client_check -> check_netgroup()).

If it's reporting "unmatched host", and if you can see it making the
right NIS request (and getting the right response), then I'm out of
other debugging ideas....

--b.