Re: Can't mount NFSv4 with kerberos on Debian Wheezy

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

I'm able to use NFSv4 just fine using AUTH_SYS, but when I turn on 
sec=krb5 I can't mount at all. I'm using Debian Wheezy.

I'm able to use Kerberos just fine for other things (like ssh), and 
forward and reverse DNS appears to be working correctly per the host 
command. However, the NFS mount command fails differently when I add my 
host's IP address to /etc/hosts (the same host is both client and 
server). Specifically, when the address is in /etc/hosts the NFS server 
fails immediately with a "permission denied" error, while if the address 
is not present in /etc/hosts the mount command hangs forever and never 
returns. This makes it seem like mount.nfs or rpc.gssd can't find the 
host in DNS even though other programs can. How can this be?

In /var/log/syslog I see this:

|Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntd
Jun 11 20:28:12 tbm rpc.idmapd[8954]: Stale client: d
Jun 11 20:28:12 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clntd/idmap
Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:12 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntc
Jun 11 20:28:12 tbm rpc.idmapd[8954]: Stale client: c
Jun 11 20:28:12 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clntc/idmap
Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:13 tbm rpc.idmapd[8954]: New client: e
Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e4570 data 0x7fffbc4e4440
Jun 11 20:28:13 tbm rpc.idmapd[8954]: Opened /var/lib/nfs/rpc_pipefs/nfs/clnte/idmap
Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e4570 data 0x7fffbc4e4440
Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:13 tbm rpc.idmapd[8954]: New client: f
Jun 11 20:28:13 tbm rpc.gssd[8959]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnte)
Jun 11 20:28:13 tbm rpc.gssd[8959]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
Jun 11 20:28:13 tbm rpc.gssd[8959]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnte)
Jun 11 20:28:13 tbm rpc.gssd[8959]: process_krb5_upcall: service is '<null>'
Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known while getting full hostname for 'tbm.enterprise.local'
Jun
  11 20:28:23 tbm rpc.gssd[8959]: ERROR:
gssd_refresh_krb5_machine_credential: no usable keytab entry found in
keytab /etc/krb5.keytab for connection with host tbm.enterprise.local
Jun 11 20:28:23 tbm rpc.gssd[8959]: ERROR: No credentials found for connection to server tbm.enterprise.local
Jun 11 20:28:23 tbm rpc.gssd[8959]: doing error downcall
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.idmapd[8954]: Stale client: f
Jun 11 20:28:23 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clntf/idmap
Jun 11 20:28:23 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntf
Jun 11 20:28:23 tbm rpc.idmapd[8954]: Stale client: e
Jun 11 20:28:23 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clnte/idmap
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
Jun 11 20:28:23 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnte|

Can anyone point me in the right direction for getting this working?


John Haiducek

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[Chuck Lever]

On Jun 14, 2013, at 1:57 AM, John Haiducek <jhaiduce@gmail.com> wrote:

> I'm able to use NFSv4 just fine using AUTH_SYS, but when I turn on sec=krb5 I can't mount at all. I'm using Debian Wheezy.
> 
> I'm able to use Kerberos just fine for other things (like ssh), and forward and reverse DNS appears to be working correctly per the host command. However, the NFS mount command fails differently when I add my host's IP address to /etc/hosts (the same host is both client and server). Specifically, when the address is in /etc/hosts the NFS server fails immediately with a "permission denied" error, while if the address is not present in /etc/hosts the mount command hangs forever and never returns. This makes it seem like mount.nfs or rpc.gssd can't find the host in DNS even though other programs can. How can this be?
> 
> In /var/log/syslog I see this:
> 
> |Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntd
> Jun 11 20:28:12 tbm rpc.idmapd[8954]: Stale client: d
> Jun 11 20:28:12 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clntd/idmap
> Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:12 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntc
> Jun 11 20:28:12 tbm rpc.idmapd[8954]: Stale client: c
> Jun 11 20:28:12 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clntc/idmap
> Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:13 tbm rpc.idmapd[8954]: New client: e
> Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e4570 data 0x7fffbc4e4440
> Jun 11 20:28:13 tbm rpc.idmapd[8954]: Opened /var/lib/nfs/rpc_pipefs/nfs/clnte/idmap
> Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e4570 data 0x7fffbc4e4440
> Jun 11 20:28:13 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:13 tbm rpc.idmapd[8954]: New client: f
> Jun 11 20:28:13 tbm rpc.gssd[8959]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnte)
> Jun 11 20:28:13 tbm rpc.gssd[8959]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
> Jun 11 20:28:13 tbm rpc.gssd[8959]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnte)
> Jun 11 20:28:13 tbm rpc.gssd[8959]: process_krb5_upcall: service is '<null>'
> Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known while getting full hostname for 'tbm.enterprise.local'

gssd thinks your client's hostname is "tbm.enterprise.local," which has no DNS entry.

> Jun 11 20:28:23 tbm rpc.gssd[8959]: ERROR:
> gssd_refresh_krb5_machine_credential: no usable keytab entry found in
> keytab /etc/krb5.keytab for connection with host tbm.enterprise.local
> Jun 11 20:28:23 tbm rpc.gssd[8959]: ERROR: No credentials found for connection to server tbm.enterprise.local

This suggests you don't have a keytab on your client, or you have one, but it doesn't have an entry that can be used as root's credential.  Could be a result of the DNS lookup problem above.

> Jun 11 20:28:23 tbm rpc.gssd[8959]: doing error downcall
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.idmapd[8954]: Stale client: f
> Jun 11 20:28:23 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clntf/idmap
> Jun 11 20:28:23 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clntf
> Jun 11 20:28:23 tbm rpc.idmapd[8954]: Stale client: e
> Jun 11 20:28:23 tbm rpc.idmapd[8954]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clnte/idmap
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.gssd[8959]: dir_notify_handler: sig 37 si 0x7fffbc4e9570 data 0x7fffbc4e9440
> Jun 11 20:28:23 tbm rpc.gssd[8959]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnte|
> 
> Can anyone point me in the right direction for getting this working?

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

On 06/14/2013 02:13 PM, Chuck Lever wrote:
>
> On Jun 14, 2013, at 3:49 PM, John Haiducek <jhaiduce@gmail.com 
> <mailto:jhaiduce@gmail.com>> wrote:
>
>>
>> On Jun 14, 2013 11:05 AM, "Chuck Lever" <chuck.lever@oracle.com 
>> <mailto:chuck.lever@oracle.com>> wrote:
>> >
>> >
>> > On Jun 14, 2013, at 1:57 AM, John Haiducek <jhaiduce@gmail.com 
>> <mailto:jhaiduce@gmail.com>> wrote:
>> >
>> > > Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known 
>> while getting full hostname for 'tbm.enterprise.local'
>> >
>> > gssd thinks your client's hostname is "tbm.enterprise.local," which 
>> has no DNS entry.
>>
>> That is the correct client hostname, and according to the 'host' 
>> command it is in dns. What would cause the host command to find it 
>> when gssd can't?
>>
>
> The error message is from utils/gssd/krb5_util.c:get_full_hostname(). 
>  If get_full_hostname() fails, then gssd can't search your client's 
> keytab.
>
> Figure out why that getaddrinfo(3) call is failing to find a canonical 
> name for "tbm.enterprise.local" -- that could be a client system 
> configuration problem as much as a DNS misconfiguration.

Ok, I think I fixed the DNS problem. I was running avahi, and apparently 
you can't use avahi and also have a DNS server with a domain ending in 
.local. Shutting down avahi fixed it, although if I wanted to keep avahi 
working I could probably fix this by changing my domain to end in 
something other than .local.

But now the mount command hangs and never returns. I get this in 
/var/log/syslog:

Jun 15 09:19:36 tbm rpc.idmapd[16253]: New client: 24
Jun 15 09:19:36 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 
0x7fffb0fb3330 data 0x7fffb0fb3200
Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 
0x7fffb0fb3330 data 0x7fffb0fb3200
Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 
0x7fffb0fb3330 data 0x7fffb0fb3200
Jun 15 09:19:37 tbm rpc.gssd[16258]: destroying client 
/var/lib/nfs/rpc_pipefs/nfs/clnt24
Jun 15 09:19:37 tbm rpc.idmapd[16253]: Stale client: 24
Jun 15 09:19:37 tbm rpc.idmapd[16253]: #011-> closed 
/var/lib/nfs/rpc_pipefs/nfs/clnt24/idmap
Jun 15 09:19:53 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 
0x7fffb0fb3330 data 0x7fffb0fb3200
Jun 15 09:19:53 tbm rpc.idmapd[16253]: New client: 25

I might be missing something, but none of these entries look like 
errors. Where else should I look?

John Haiducek

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[Chuck Lever]

On Jun 15, 2013, at 11:24 AM, John Haiducek <jhaiduce@gmail.com> wrote:

> On 06/14/2013 02:13 PM, Chuck Lever wrote:
>> 
>> On Jun 14, 2013, at 3:49 PM, John Haiducek <jhaiduce@gmail.com <mailto:jhaiduce@gmail.com>> wrote:
>> 
>>> 
>>> On Jun 14, 2013 11:05 AM, "Chuck Lever" <chuck.lever@oracle.com <mailto:chuck.lever@oracle.com>> wrote:
>>> >
>>> >
>>> > On Jun 14, 2013, at 1:57 AM, John Haiducek <jhaiduce@gmail.com <mailto:jhaiduce@gmail.com>> wrote:
>>> >
>>> > > Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known while getting full hostname for 'tbm.enterprise.local'
>>> >
>>> > gssd thinks your client's hostname is "tbm.enterprise.local," which has no DNS entry.
>>> 
>>> That is the correct client hostname, and according to the 'host' command it is in dns. What would cause the host command to find it when gssd can't?
>>> 
>> 
>> The error message is from utils/gssd/krb5_util.c:get_full_hostname().  If get_full_hostname() fails, then gssd can't search your client's keytab.
>> 
>> Figure out why that getaddrinfo(3) call is failing to find a canonical name for "tbm.enterprise.local" -- that could be a client system configuration problem as much as a DNS misconfiguration.
> 
> Ok, I think I fixed the DNS problem. I was running avahi, and apparently you can't use avahi and also have a DNS server with a domain ending in .local. Shutting down avahi fixed it, although if I wanted to keep avahi working I could probably fix this by changing my domain to end in something other than .local.
> 
> But now the mount command hangs and never returns. I get this in /var/log/syslog:
> 
> Jun 15 09:19:36 tbm rpc.idmapd[16253]: New client: 24
> Jun 15 09:19:36 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
> Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
> Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
> Jun 15 09:19:37 tbm rpc.gssd[16258]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt24
> Jun 15 09:19:37 tbm rpc.idmapd[16253]: Stale client: 24
> Jun 15 09:19:37 tbm rpc.idmapd[16253]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt24/idmap
> Jun 15 09:19:53 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
> Jun 15 09:19:53 tbm rpc.idmapd[16253]: New client: 25
> 
> I might be missing something, but none of these entries look like errors. Where else should I look?

You can boost the verbosity of the debugging messages from gssd.  Start it with "-vv" or "-vvv".

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

On 06/15/2013 10:27 AM, Chuck Lever wrote:
> On Jun 15, 2013, at 11:24 AM, John Haiducek<jhaiduce@gmail.com>  wrote:
>
>> On 06/14/2013 02:13 PM, Chuck Lever wrote:
>>> On Jun 14, 2013, at 3:49 PM, John Haiducek<jhaiduce@gmail.com<mailto:jhaiduce@gmail.com>>  wrote:
>>>
>>>> On Jun 14, 2013 11:05 AM, "Chuck Lever"<chuck.lever@oracle.com<mailto:chuck.lever@oracle.com>>  wrote:
>>>>>
>>>>> On Jun 14, 2013, at 1:57 AM, John Haiducek<jhaiduce@gmail.com<mailto:jhaiduce@gmail.com>>  wrote:
>>>>>
>>>>>> Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known while getting full hostname for 'tbm.enterprise.local'
>>>>> gssd thinks your client's hostname is "tbm.enterprise.local," which has no DNS entry.
>>>> That is the correct client hostname, and according to the 'host' command it is in dns. What would cause the host command to find it when gssd can't?
>>>>
>>> The error message is from utils/gssd/krb5_util.c:get_full_hostname().  If get_full_hostname() fails, then gssd can't search your client's keytab.
>>>
>>> Figure out why that getaddrinfo(3) call is failing to find a canonical name for "tbm.enterprise.local" -- that could be a client system configuration problem as much as a DNS misconfiguration.
>> Ok, I think I fixed the DNS problem. I was running avahi, and apparently you can't use avahi and also have a DNS server with a domain ending in .local. Shutting down avahi fixed it, although if I wanted to keep avahi working I could probably fix this by changing my domain to end in something other than .local.
>>
>> But now the mount command hangs and never returns. I get this in /var/log/syslog:
>>
>> Jun 15 09:19:36 tbm rpc.idmapd[16253]: New client: 24
>> Jun 15 09:19:36 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>> Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>> Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>> Jun 15 09:19:37 tbm rpc.gssd[16258]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt24
>> Jun 15 09:19:37 tbm rpc.idmapd[16253]: Stale client: 24
>> Jun 15 09:19:37 tbm rpc.idmapd[16253]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clnt24/idmap
>> Jun 15 09:19:53 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>> Jun 15 09:19:53 tbm rpc.idmapd[16253]: New client: 25
>>
>> I might be missing something, but none of these entries look like errors. Where else should I look?
> You can boost the verbosity of the debugging messages from gssd.  Start it with "-vv" or "-vvv".
>

Already have it gssd running with -vvv.

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[Chuck Lever]

On Jun 15, 2013, at 12:28 PM, John Haiducek <jhaiduce@gmail.com> wrote:

> On 06/15/2013 10:27 AM, Chuck Lever wrote:
>> On Jun 15, 2013, at 11:24 AM, John Haiducek<jhaiduce@gmail.com>  wrote:
>> 
>>> On 06/14/2013 02:13 PM, Chuck Lever wrote:
>>>> On Jun 14, 2013, at 3:49 PM, John Haiducek<jhaiduce@gmail.com<mailto:jhaiduce@gmail.com>>  wrote:
>>>> 
>>>>> On Jun 14, 2013 11:05 AM, "Chuck Lever"<chuck.lever@oracle.com<mailto:chuck.lever@oracle.com>>  wrote:
>>>>>> 
>>>>>> On Jun 14, 2013, at 1:57 AM, John Haiducek<jhaiduce@gmail.com<mailto:jhaiduce@gmail.com>>  wrote:
>>>>>> 
>>>>>>> Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known while getting full hostname for 'tbm.enterprise.local'
>>>>>> gssd thinks your client's hostname is "tbm.enterprise.local," which has no DNS entry.
>>>>> That is the correct client hostname, and according to the 'host' command it is in dns. What would cause the host command to find it when gssd can't?
>>>>> 
>>>> The error message is from utils/gssd/krb5_util.c:get_full_hostname().  If get_full_hostname() fails, then gssd can't search your client's keytab.
>>>> 
>>>> Figure out why that getaddrinfo(3) call is failing to find a canonical name for "tbm.enterprise.local" -- that could be a client system configuration problem as much as a DNS misconfiguration.
>>> Ok, I think I fixed the DNS problem. I was running avahi, and apparently you can't use avahi and also have a DNS server with a domain ending in .local. Shutting down avahi fixed it, although if I wanted to keep avahi working I could probably fix this by changing my domain to end in something other than .local.
>>> 
>>> But now the mount command hangs and never returns. I get this in /var/log/syslog:
>>> 
>>> Jun 15 09:19:36 tbm rpc.idmapd[16253]: New client: 24
>>> Jun 15 09:19:36 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>>> Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>>> Jun 15 09:19:37 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>>> Jun 15 09:19:37 tbm rpc.gssd[16258]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt24
>>> Jun 15 09:19:37 tbm rpc.idmapd[16253]: Stale client: 24
>>> Jun 15 09:19:37 tbm rpc.idmapd[16253]: #011->  closed /var/lib/nfs/rpc_pipefs/nfs/clnt24/idmap
>>> Jun 15 09:19:53 tbm rpc.gssd[16258]: dir_notify_handler: sig 37 si 0x7fffb0fb3330 data 0x7fffb0fb3200
>>> Jun 15 09:19:53 tbm rpc.idmapd[16253]: New client: 25
>>> 
>>> I might be missing something, but none of these entries look like errors. Where else should I look?
>> You can boost the verbosity of the debugging messages from gssd.  Start it with "-vv" or "-vvv".
>> 
> 
> Already have it gssd running with -vvv.

On the client,

  "sudo rpcdebug -m nfs -s all"

And try the mount.  Look in /var/log/messages for output.

Did you tell us what kernel release you are running on client and server?  "uname -a"

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

On 06/15/2013 10:31 AM, Chuck Lever wrote:
> On the client, "sudo rpcdebug -m nfs -s all" And try the mount. Look 
> in /var/log/messages for output. Did you tell us what kernel release 
> you are running on client and server? "uname -a" 

Ok, now in /var/log messages I have this:

Jun 15 09:13:21 tbm kernel: [307264.414216] NFSD: starting 90-second 
grace period
Jun 15 09:23:34 tbm kernel: [307877.856060] nfs: server 
tbm.enterprise.local not responding, still trying
Jun 15 09:29:35 tbm kernel: [308238.304057] nfs: server 
tbm.enterprise.local not responding, still trying
Jun 15 10:33:27 tbm kernel: [312070.654766] NFS: nfs mount 
opts='addr=192.168.1.7,clientaddr=192.168.1.7'
Jun 15 10:33:27 tbm kernel: [312070.654780] NFS:   parsing nfs mount 
option 'addr=192.168.1.7'
Jun 15 10:33:27 tbm kernel: [312070.654795] NFS:   parsing nfs mount 
option 'clientaddr=192.168.1.7'
Jun 15 10:33:27 tbm kernel: [312070.654807] NFS: MNTPATH: '/export/nfstest'
Jun 15 10:33:27 tbm kernel: [312070.654810] --> nfs4_try_mount()
Jun 15 10:33:27 tbm kernel: [312070.654822] --> nfs4_create_server()
Jun 15 10:33:27 tbm kernel: [312070.654854] --> nfs4_init_server()
Jun 15 10:33:27 tbm kernel: [312070.654857] --> nfs4_set_client()
Jun 15 10:33:27 tbm kernel: [312070.654862] --> nfs_get_client(tbm,v4)
Jun 15 10:33:27 tbm kernel: [312070.654867] --> nfs_get_client() = 
ffff88021bcd3c00 [share]
Jun 15 10:33:27 tbm kernel: [312070.654872] <-- nfs4_set_client() = 0 
[new ffff88021bcd3c00]
Jun 15 10:33:27 tbm kernel: [312070.655071] <-- nfs4_init_server() = 0
Jun 15 10:33:27 tbm kernel: [312070.655241] --> nfs4_get_rootfh()
Jun 15 10:33:27 tbm kernel: [312070.655279] encode_compound: tag=
Jun 15 10:33:44 tbm kernel: [312087.264052] encode_compound: tag=
Jun 15 10:34:27 tbm kernel: [312130.784203] encode_compound: tag=
Jun 15 10:34:27 tbm kernel: [312130.784290] encode_compound: tag=
Jun 15 10:34:27 tbm kernel: [312130.784541] encode_compound: tag=
Jun 15 10:34:27 tbm kernel: [312130.784587] encode_compound: tag=


The first 3 lines are from a previous mount attempt before I ran the 
rpcdebug command.

Client and server are the same machine. uname -a prints this:

Linux tbm 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux


John

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[Chuck Lever]

On Jun 15, 2013, at 12:38 PM, John Haiducek <jhaiduce@gmail.com> wrote:

> On 06/15/2013 10:31 AM, Chuck Lever wrote:
>> On the client, "sudo rpcdebug -m nfs -s all" And try the mount. Look in /var/log/messages for output. Did you tell us what kernel release you are running on client and server? "uname -a" 
> 
> Ok, now in /var/log messages I have this:
> 
> Jun 15 09:13:21 tbm kernel: [307264.414216] NFSD: starting 90-second grace period
> Jun 15 09:23:34 tbm kernel: [307877.856060] nfs: server tbm.enterprise.local not responding, still trying
> Jun 15 09:29:35 tbm kernel: [308238.304057] nfs: server tbm.enterprise.local not responding, still trying
> Jun 15 10:33:27 tbm kernel: [312070.654766] NFS: nfs mount opts='addr=192.168.1.7,clientaddr=192.168.1.7'
> Jun 15 10:33:27 tbm kernel: [312070.654780] NFS:   parsing nfs mount option 'addr=192.168.1.7'
> Jun 15 10:33:27 tbm kernel: [312070.654795] NFS:   parsing nfs mount option 'clientaddr=192.168.1.7'
> Jun 15 10:33:27 tbm kernel: [312070.654807] NFS: MNTPATH: '/export/nfstest'
> Jun 15 10:33:27 tbm kernel: [312070.654810] --> nfs4_try_mount()
> Jun 15 10:33:27 tbm kernel: [312070.654822] --> nfs4_create_server()
> Jun 15 10:33:27 tbm kernel: [312070.654854] --> nfs4_init_server()
> Jun 15 10:33:27 tbm kernel: [312070.654857] --> nfs4_set_client()
> Jun 15 10:33:27 tbm kernel: [312070.654862] --> nfs_get_client(tbm,v4)
> Jun 15 10:33:27 tbm kernel: [312070.654867] --> nfs_get_client() = ffff88021bcd3c00 [share]
> Jun 15 10:33:27 tbm kernel: [312070.654872] <-- nfs4_set_client() = 0 [new ffff88021bcd3c00]
> Jun 15 10:33:27 tbm kernel: [312070.655071] <-- nfs4_init_server() = 0
> Jun 15 10:33:27 tbm kernel: [312070.655241] --> nfs4_get_rootfh()
> Jun 15 10:33:27 tbm kernel: [312070.655279] encode_compound: tag=
> Jun 15 10:33:44 tbm kernel: [312087.264052] encode_compound: tag=
> Jun 15 10:34:27 tbm kernel: [312130.784203] encode_compound: tag=
> Jun 15 10:34:27 tbm kernel: [312130.784290] encode_compound: tag=
> Jun 15 10:34:27 tbm kernel: [312130.784541] encode_compound: tag=
> Jun 15 10:34:27 tbm kernel: [312130.784587] encode_compound: tag=
> 
> 
> The first 3 lines are from a previous mount attempt before I ran the rpcdebug command.
> 
> Client and server are the same machine. uname -a prints this:
> 
> Linux tbm 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux

Is rpc.svcgssd running?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

On 06/17/2013 08:23 AM, Chuck Lever wrote:
>  Is rpc.svcgssd running?

Yes, and with the same verbosity as rpc.gssd (-vvv).

John

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

On 06/14/2013 02:13 PM, Chuck Lever wrote:
>
> On Jun 14, 2013, at 3:49 PM, John Haiducek <jhaiduce@gmail.com 
> <mailto:jhaiduce@gmail.com>> wrote:
>
>>
>> On Jun 14, 2013 11:05 AM, "Chuck Lever" <chuck.lever@oracle.com 
>> <mailto:chuck.lever@oracle.com>> wrote:
>> >
>> >
>> > On Jun 14, 2013, at 1:57 AM, John Haiducek <jhaiduce@gmail.com 
>> <mailto:jhaiduce@gmail.com>> wrote:
>> >
>> > > Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known 
>> while getting full hostname for 'tbm.enterprise.local'
>> >
>> > gssd thinks your client's hostname is "tbm.enterprise.local," which 
>> has no DNS entry.
>>
>> That is the correct client hostname, and according to the 'host' 
>> command it is in dns. What would cause the host command to find it 
>> when gssd can't?
>>
>
> The error message is from utils/gssd/krb5_util.c:get_full_hostname(). 
>  If get_full_hostname() fails, then gssd can't search your client's 
> keytab.
>
> Figure out why that getaddrinfo(3) call is failing to find a canonical 
> name for "tbm.enterprise.local" -- that could be a client system 
> configuration problem as much as a DNS misconfiguration.

Incidentally, I have a dual-stack network (ipv4 and ipv6). My best guess 
is that ipv6 is not part of the problem; the client and server had 
missing AAAA records in DNS; when I added those the NFS mount switched 
automatically to using ipv6 for the connection but it hangs the same as 
before.

John

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[Chuck Lever]

On Jun 17, 2013, at 10:58 AM, John Haiducek <jhaiduce@gmail.com> wrote:

> 
> On 06/14/2013 02:13 PM, Chuck Lever wrote:
>> 
>> On Jun 14, 2013, at 3:49 PM, John Haiducek <jhaiduce@gmail.com <mailto:jhaiduce@gmail.com>> wrote:
>> 
>>> 
>>> On Jun 14, 2013 11:05 AM, "Chuck Lever" <chuck.lever@oracle.com <mailto:chuck.lever@oracle.com>> wrote:
>>> >
>>> >
>>> > On Jun 14, 2013, at 1:57 AM, John Haiducek <jhaiduce@gmail.com <mailto:jhaiduce@gmail.com>> wrote:
>>> >
>>> > > Jun 11 20:28:23 tbm rpc.gssd[8959]: Name or service not known while getting full hostname for 'tbm.enterprise.local'
>>> >
>>> > gssd thinks your client's hostname is "tbm.enterprise.local," which has no DNS entry.
>>> 
>>> That is the correct client hostname, and according to the 'host' command it is in dns. What would cause the host command to find it when gssd can't?
>>> 
>> 
>> The error message is from utils/gssd/krb5_util.c:get_full_hostname().  If get_full_hostname() fails, then gssd can't search your client's keytab.
>> 
>> Figure out why that getaddrinfo(3) call is failing to find a canonical name for "tbm.enterprise.local" -- that could be a client system configuration problem as much as a DNS misconfiguration.
> 
> Incidentally, I have a dual-stack network (ipv4 and ipv6). My best guess is that ipv6 is not part of the problem; the client and server had missing AAAA records in DNS; when I added those the NFS mount switched automatically to using ipv6 for the connection but it hangs the same as before.

It looks to me like a network connectivity issue at this point.  The next step is to capture a network trace.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[John Haiducek]

Chuck,

I upgraded my system to Debian experimental yesterday and that fixed my 
NFS problem. Downgraded again to Wheezy and NFS still works. (I had put 
NFS on the back burner; the upgrade and downgrade was part of an attempt 
to install Skype, which eventually proved successful.) So apparently the 
problem was a broken installation of one of the NFS packages or a 
dependency. Presumably I could have fixed it by re-installing the one or 
two offending packages, but the "reinstall almost everything on the 
system" approach seems to have worked too. Thank you again for all your 
help!

John

Re: Can't mount NFSv4 with kerberos on Debian Wheezy

[Sven Geggus]

John Haiducek <jhaiduce@gmail.com> wrote:

> I upgraded my system to Debian experimental yesterday and that fixed my 
> NFS problem.

NFS userland is newer in testing/unstable 1.2.8 vs. 1.2.6 so did you
downgrade those as well?

Sven

-- 
"Das ist halt der Unterschied: Unix ist ein Betriebssystem mit Tradition,
 die anderen sind einfach von sich aus unlogisch."
                          (Anselm Lingnau in de.comp.os.unix.discussion)
/me ist giggls@ircnet, http://sven.gegg.us/ im WWW