Re: Trouble with multiple kerberos ticket caches

[Orion Poplawski <orion@nwra.com>]

[-- Attachment #1: Type: text/plain, Size: 5214 bytes --]

On 5/2/25 11:29, Orion Poplawski wrote:
> One of our users is struggling with multiple kerberos ticket caches impacting
> access to NFS sec=krb5 mounts.
> 
> Because home directories are NFS mounted, we use GSSAPI auth to forward a
> ticket.  But then we need to kinit to have a long-term renewable ticket.
> 
> But we seem to be seeing that new ssh connections which create a new ticket
> cache break access to the NFS mounts, resulting in "permission denied" or
> "Stale file handle" messages.  Switching back to a renewable ticket cache
> seems to resolve the issue.
> 
> Any suggestions?  Is this expected?  I would have thought that the nfs access
> would work with any valid ticket.
> 
> NAME="AlmaLinux"
> VERSION="8.10 (Cerulean Leopard)"
> nfs-utils-2.3.3-59.el8.x86_64
> 4.18.0-553.50.1.el8_10.x86_64
> 

More details.  The issue seems to arise when doing gssapi delegation from a
macOS client to the Linux box.  If I have ticket on macOS:

Ticket cache: API:427671FC-DB63-442F-ACA4-13A9194F4398
Default principal: user@AD.NWRA.COM

Valid starting     Expires            Service principal
05/06/25 13:29:54  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
        renew until 05/13/25 13:29:50
05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/NWRA.COM@AD.NWRA.COM
05/06/25 13:30:30  05/06/25 23:29:54  host/host@NWRA.COM

and ssh to the Linux box, I can't access the nfs mount:

-bash: /home/user/.bash_profile: Permission denied

Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
Default principal: user@AD.NWRA.COM

Valid starting     Expires            Service principal
05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM

I also notice that the ticket is non-renewable.

If I then kinit I can access the home directory fine.  Other than the new
ticket being renewable I don't see any difference:

Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
Default principal: user@AD.NWRA.COM

Valid starting     Expires            Service principal
05/06/25 13:31:27  05/06/25 23:31:27  nfs/server@NWRA.COM
        renew until 05/13/25 13:31:22
05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/NWRA.COM@AD.NWRA.COM
        renew until 05/13/25 13:31:22
05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/AD.NWRA.COM@AD.NWRA.COM
        renew until 05/13/25 13:31:22

Actually, I also notice now that there is a krbtgt/NWRA.COM principal as well.
 I wonder if that is the difference.


On the initial connection I see:

May 06 13:16:22 rpc.gssd[1539533]:
                                   handle_gssd_upcall(0x7f9c73b2fc80):
'mech=krb5 uid=30657 enctypes=18,17,16,23,3,1,2' (nfs/clnt2c)
May 06 13:16:22 rpc.gssd[1539533]: start_upcall_thread(0x7f9c73b2fc80):
created thread id 0x7f9c69d07700
May 06 13:16:22 rpc.gssd[1539533]: krb5_not_machine_creds(0x7f9c69d07700): uid
30657 tgtname (null)
May 06 13:16:22 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c69d07700):
creating tcp client for server server
May 06 13:16:22 rpc.gssd[1539533]: DEBUG: port already set to 2049
May 06 13:16:22 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c69d07700):
creating context with server nfs@server
May 06 13:16:22 rpc.gssd[1539533]: WARNING: Failed to create krb5 context for
user with uid 30657 for server nfs@server
May 06 13:16:22 rpc.gssd[1539533]: looking for client creds with uid 30657 for
server serverin /tmp
May 06 13:16:22 rpc.gssd[1539533]: CC '/tmp/krb5ccmachine_NWRA.COM' being
considered, with preferred realm 'NWRA.COM'
May 06 13:16:22 rpc.gssd[1539533]: CC '/tmp/krb5ccmachine_NWRA.COM' owned by
0, not 30657
May 06 13:16:22 rpc.gssd[1539533]: looking for client creds with uid 30657 for
server server in /run/user/%U
May 06 13:16:22 rpc.gssd[1539533]: do_error_downcall(0x7f9c69d07700): uid
30657 err -13


after kinit:

May 06 13:31:27 rpc.gssd[1539533]:
                                   handle_gssd_upcall(0x7f9c73b2fc80):
'mech=krb5 uid=30657 enctypes=18,17,16,23,3,1,2' (nfs/
clnt2c)
May 06 13:31:27 rpc.gssd[1539533]: start_upcall_thread(0x7f9c73b2fc80):
created thread id 0x7f9c63fff700
May 06 13:31:27 rpc.gssd[1539533]: krb5_not_machine_creds(0x7f9c63fff700): uid
30657 tgtname (null)
May 06 13:31:27 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c63fff700):
creating tcp client for server server
May 06 13:31:27 rpc.gssd[1539533]: DEBUG: port already set to 2049
May 06 13:31:27 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c63fff700):
creating context with server nfs@server
May 06 13:31:27 rpc.gssd[1539533]: DEBUG: serialize_krb5_ctx: lucid version!
May 06 13:31:27 rpc.gssd[1539533]: prepare_krb5_rfc4121_buffer: protocol 1
May 06 13:31:27 rpc.gssd[1539533]: prepare_krb5_rfc4121_buffer: serializing
key with enctype 18 and size 32
May 06 13:31:27 rpc.gssd[1539533]: do_downcall(0x7f9c63fff700):
lifetime_rec=10h:0m:0s acceptor=nfs@server


-- 
Orion Poplawski
he/him/his  - surely the least important thing about me
Manager of IT Systems                      720-772-5637
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       orion@nwra.com
Boulder, CO 80301                 https://www.nwra.com/

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4087 bytes --]