CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

[NFS Traumatized Linux User]

Hi there,

sorry for bothering here, but I did not fing anything useful on 
Google... occasianally searching for serveral months.

I'm running a CentOs 8.2.2004 file server on KVM, kernel version 
4.18.0-193.6.3.el8_2. That file server VM is part of a FreeeIPA domain. 
Domain users (=extended family) can log in via SSH: this works very 
reliable.

At home, I have set up some CentOS 8 clients, two for my kids and one 
for myself. I'll focus on my own client for now the kids one's are the 
same/similar. My personal client is a HP z420 with 12 cores and 64GB of 
RAM, so pleanty of resources. Kernel 4.18.0-193.14-2. This machine has a 
1GBit link to the file server.

I registered the client machine in the domain: id <domainuser> works 
just fine: the userid and the associated groups are returned as would be 
expected.

There is no clock skew between the client and the NFS & Kerberos 
servers: I have a Stratum 1 Meinberg NTP appliance providing accurate 
GPS time to the entire network.

I export the home directories on the NFS server to the local network 
(rw,sec=krb5p,root_squash,async,subtree_check) and configured autofs on 
the clients to mount the respective user home directory when a user logs 
into a client:

/etc/auto.master:
/home /etc/auto.home --timeout=600

/etc/auto.home
* -rw,nfs4,nfsvers=4.1,soft fileserver:/home/&

When a user logs into a client, the user automatically gets a 24h 
Kerberos ticket that can be displayed with klist.

Upon login, AutoFS does its thing by mounting the user's home directory 
in the expected location. ID mappiung works as expected: ls -l therefore 
shows usernames/groupnames instead of numeric UIDs/GIDs. Write access to 
the exported FS works: For instance

   touch test
   rm test

in the user's home all just work fine.

BUT starting a normal GNOME session after accepting the login 
credentials takes VERY long (2-3 minutes). Opening a simple GNOME 
terminal takes long (~1 minute), starting Nautilus takes ages (firing it 
up from the terminal until an empty window appear takes ~1 minute, then 
I get a spinning wheel in the lower right corner saying "Loading" for 
another couple of minutes before the home directory contents is 
displayed in the window). Lauching programs in GNOME frequently makes 
the mouse&keyboard freeze for several minutes.

SELinux does not seem to be the culprit: I tried with "setenforce 0" on 
both the server and the client without success.

My feeling is that something with NFS is wrong here, but I don't see 
what it might be. Both, the client and the server run on quite powerful 
hardware, so I would expect sec=krb5p not to be an issue.

I have a few questions:

   * What is the right approach to debug this?
   * Are there any known issues with kerberized NFS and exported home 
directories similar to what I described above?

Any pointers / help would be greatly appreciated.

--
Cheers,
Ray

Re: CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

[Benjamin Coddington]

On 12 Aug 2020, at 16:05, NFS Traumatized Linux User wrote:

> Hi there,
..
> Any pointers / help would be greatly appreciated.

Delays in the UI on the order of several minutes that are suspected to be
NFS related should be visible in a wirecapture.  Have you tried using
vanilla krb5 and if the delays are still there, what does a wirecapture of
the NFS traffic look like?  Are there long delays between each RPC call and
reply?  That's the place I would start.

I can say that this problem sounds a bit like a name resolution problem
somewhere.  Long delays that produce no errors at random times often turn
out to be systems repeatedly trying to use a name server that doesn't reply.
Perhaps either the client or server's name services are trying to look up an
identifier and there's a DNS server record that isn't replying - but that's
a wild guess.

Ben

Re: CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

[J. Bruce Fields]

On Wed, Aug 12, 2020 at 10:05:08PM +0200, NFS Traumatized Linux User wrote:
> I export the home directories on the NFS server to the local network
> (rw,sec=krb5p,root_squash,async,subtree_check) and configured autofs

Unrelated to your problem, but I wouldn't recommend "async".

Like Ben I'd start by watching the traffic in wireshark.

--b.