Trouble with multiple kerberos ticket caches

Trouble with multiple kerberos ticket caches

[Orion Poplawski]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

One of our users is struggling with multiple kerberos ticket caches impacting
access to NFS sec=krb5 mounts.

Because home directories are NFS mounted, we use GSSAPI auth to forward a
ticket.  But then we need to kinit to have a long-term renewable ticket.

But we seem to be seeing that new ssh connections which create a new ticket
cache break access to the NFS mounts, resulting in "permission denied" or
"Stale file handle" messages.  Switching back to a renewable ticket cache
seems to resolve the issue.

Any suggestions?  Is this expected?  I would have thought that the nfs access
would work with any valid ticket.

NAME="AlmaLinux"
VERSION="8.10 (Cerulean Leopard)"
nfs-utils-2.3.3-59.el8.x86_64
4.18.0-553.50.1.el8_10.x86_64

-- 
Orion Poplawski
he/him/his  - surely the least important thing about me
Manager of IT Systems                      720-772-5637
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       orion@nwra.com
Boulder, CO 80301                 https://www.nwra.com/


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4087 bytes --]

Re: Trouble with multiple kerberos ticket caches

[Orion Poplawski]

[-- Attachment #1: Type: text/plain, Size: 5214 bytes --]

On 5/2/25 11:29, Orion Poplawski wrote:
> One of our users is struggling with multiple kerberos ticket caches impacting
> access to NFS sec=krb5 mounts.
> 
> Because home directories are NFS mounted, we use GSSAPI auth to forward a
> ticket.  But then we need to kinit to have a long-term renewable ticket.
> 
> But we seem to be seeing that new ssh connections which create a new ticket
> cache break access to the NFS mounts, resulting in "permission denied" or
> "Stale file handle" messages.  Switching back to a renewable ticket cache
> seems to resolve the issue.
> 
> Any suggestions?  Is this expected?  I would have thought that the nfs access
> would work with any valid ticket.
> 
> NAME="AlmaLinux"
> VERSION="8.10 (Cerulean Leopard)"
> nfs-utils-2.3.3-59.el8.x86_64
> 4.18.0-553.50.1.el8_10.x86_64
> 

More details.  The issue seems to arise when doing gssapi delegation from a
macOS client to the Linux box.  If I have ticket on macOS:

Ticket cache: API:427671FC-DB63-442F-ACA4-13A9194F4398
Default principal: user@AD.NWRA.COM

Valid starting     Expires            Service principal
05/06/25 13:29:54  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
        renew until 05/13/25 13:29:50
05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/NWRA.COM@AD.NWRA.COM
05/06/25 13:30:30  05/06/25 23:29:54  host/host@NWRA.COM

and ssh to the Linux box, I can't access the nfs mount:

-bash: /home/user/.bash_profile: Permission denied

Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
Default principal: user@AD.NWRA.COM

Valid starting     Expires            Service principal
05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM

I also notice that the ticket is non-renewable.

If I then kinit I can access the home directory fine.  Other than the new
ticket being renewable I don't see any difference:

Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
Default principal: user@AD.NWRA.COM

Valid starting     Expires            Service principal
05/06/25 13:31:27  05/06/25 23:31:27  nfs/server@NWRA.COM
        renew until 05/13/25 13:31:22
05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/NWRA.COM@AD.NWRA.COM
        renew until 05/13/25 13:31:22
05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/AD.NWRA.COM@AD.NWRA.COM
        renew until 05/13/25 13:31:22

Actually, I also notice now that there is a krbtgt/NWRA.COM principal as well.
 I wonder if that is the difference.


On the initial connection I see:

May 06 13:16:22 rpc.gssd[1539533]:
                                   handle_gssd_upcall(0x7f9c73b2fc80):
'mech=krb5 uid=30657 enctypes=18,17,16,23,3,1,2' (nfs/clnt2c)
May 06 13:16:22 rpc.gssd[1539533]: start_upcall_thread(0x7f9c73b2fc80):
created thread id 0x7f9c69d07700
May 06 13:16:22 rpc.gssd[1539533]: krb5_not_machine_creds(0x7f9c69d07700): uid
30657 tgtname (null)
May 06 13:16:22 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c69d07700):
creating tcp client for server server
May 06 13:16:22 rpc.gssd[1539533]: DEBUG: port already set to 2049
May 06 13:16:22 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c69d07700):
creating context with server nfs@server
May 06 13:16:22 rpc.gssd[1539533]: WARNING: Failed to create krb5 context for
user with uid 30657 for server nfs@server
May 06 13:16:22 rpc.gssd[1539533]: looking for client creds with uid 30657 for
server serverin /tmp
May 06 13:16:22 rpc.gssd[1539533]: CC '/tmp/krb5ccmachine_NWRA.COM' being
considered, with preferred realm 'NWRA.COM'
May 06 13:16:22 rpc.gssd[1539533]: CC '/tmp/krb5ccmachine_NWRA.COM' owned by
0, not 30657
May 06 13:16:22 rpc.gssd[1539533]: looking for client creds with uid 30657 for
server server in /run/user/%U
May 06 13:16:22 rpc.gssd[1539533]: do_error_downcall(0x7f9c69d07700): uid
30657 err -13


after kinit:

May 06 13:31:27 rpc.gssd[1539533]:
                                   handle_gssd_upcall(0x7f9c73b2fc80):
'mech=krb5 uid=30657 enctypes=18,17,16,23,3,1,2' (nfs/
clnt2c)
May 06 13:31:27 rpc.gssd[1539533]: start_upcall_thread(0x7f9c73b2fc80):
created thread id 0x7f9c63fff700
May 06 13:31:27 rpc.gssd[1539533]: krb5_not_machine_creds(0x7f9c63fff700): uid
30657 tgtname (null)
May 06 13:31:27 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c63fff700):
creating tcp client for server server
May 06 13:31:27 rpc.gssd[1539533]: DEBUG: port already set to 2049
May 06 13:31:27 rpc.gssd[1539533]: create_auth_rpc_client(0x7f9c63fff700):
creating context with server nfs@server
May 06 13:31:27 rpc.gssd[1539533]: DEBUG: serialize_krb5_ctx: lucid version!
May 06 13:31:27 rpc.gssd[1539533]: prepare_krb5_rfc4121_buffer: protocol 1
May 06 13:31:27 rpc.gssd[1539533]: prepare_krb5_rfc4121_buffer: serializing
key with enctype 18 and size 32
May 06 13:31:27 rpc.gssd[1539533]: do_downcall(0x7f9c63fff700):
lifetime_rec=10h:0m:0s acceptor=nfs@server


-- 
Orion Poplawski
he/him/his  - surely the least important thing about me
Manager of IT Systems                      720-772-5637
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       orion@nwra.com
Boulder, CO 80301                 https://www.nwra.com/

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4087 bytes --]

Re: Trouble with multiple kerberos ticket caches

[Daniel Kobras]

Hi!

Am 06.05.25 um 21:54 schrieb Orion Poplawski:
> More details.  The issue seems to arise when doing gssapi delegation from a
> macOS client to the Linux box.  If I have ticket on macOS:
> 
> Ticket cache: API:427671FC-DB63-442F-ACA4-13A9194F4398
> Default principal: user@AD.NWRA.COM
> 
> Valid starting     Expires            Service principal
> 05/06/25 13:29:54  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>          renew until 05/13/25 13:29:50
> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/NWRA.COM@AD.NWRA.COM
> 05/06/25 13:30:30  05/06/25 23:29:54  host/host@NWRA.COM
> 
> and ssh to the Linux box, I can't access the nfs mount:
> 
> -bash: /home/user/.bash_profile: Permission denied
> 
> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
> Default principal: user@AD.NWRA.COM
> 
> Valid starting     Expires            Service principal
> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
> 
> I also notice that the ticket is non-renewable.
> 
> If I then kinit I can access the home directory fine.  Other than the new
> ticket being renewable I don't see any difference:
> 
> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
> Default principal: user@AD.NWRA.COM
> 
> Valid starting     Expires            Service principal
> 05/06/25 13:31:27  05/06/25 23:31:27  nfs/server@NWRA.COM
>          renew until 05/13/25 13:31:22
> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/NWRA.COM@AD.NWRA.COM
>          renew until 05/13/25 13:31:22
> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>          renew until 05/13/25 13:31:22
> 
> Actually, I also notice now that there is a krbtgt/NWRA.COM principal as well.
>   I wonder if that is the difference.

Can you please check and compare the output of 'klist -ef' (which 
includes additional info on enctypes and flags) in both cases? It sounds 
like the macOS client is forwarding a ticket that is not accepted by the 
Linux client's krb5 library. This could happen if eg. the 
default_tgs_enctypes configured in krb5.conf on the macOS side is 
incompatible with the permitted_enctypes in krb5.conf on the Linux side.

Kind regards,

Daniel
-- 
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de

-- 
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft: Eisenbahnstraße 1, 72072 
Tübingen

Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer: 
Lukas Kallies, Daniel Kobras, Mark Pröhl

Re: Trouble with multiple kerberos ticket caches

[Orion Poplawski]

[-- Attachment #1: Type: text/plain, Size: 3980 bytes --]

On 5/7/25 10:57, Daniel Kobras wrote:
> Hi!
> 
> Am 06.05.25 um 21:54 schrieb Orion Poplawski:
>> More details.  The issue seems to arise when doing gssapi delegation from a
>> macOS client to the Linux box.  If I have ticket on macOS:
>>
>> Ticket cache: API:427671FC-DB63-442F-ACA4-13A9194F4398
>> Default principal: user@AD.NWRA.COM
>>
>> Valid starting     Expires            Service principal
>> 05/06/25 13:29:54  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>          renew until 05/13/25 13:29:50
>> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/NWRA.COM@AD.NWRA.COM
>> 05/06/25 13:30:30  05/06/25 23:29:54  host/host@NWRA.COM
>>
>> and ssh to the Linux box, I can't access the nfs mount:
>>
>> -bash: /home/user/.bash_profile: Permission denied
>>
>> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
>> Default principal: user@AD.NWRA.COM
>>
>> Valid starting     Expires            Service principal
>> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>
>> I also notice that the ticket is non-renewable.
>>
>> If I then kinit I can access the home directory fine.  Other than the new
>> ticket being renewable I don't see any difference:
>>
>> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
>> Default principal: user@AD.NWRA.COM
>>
>> Valid starting     Expires            Service principal
>> 05/06/25 13:31:27  05/06/25 23:31:27  nfs/server@NWRA.COM
>>          renew until 05/13/25 13:31:22
>> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/NWRA.COM@AD.NWRA.COM
>>          renew until 05/13/25 13:31:22
>> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>          renew until 05/13/25 13:31:22
>>
>> Actually, I also notice now that there is a krbtgt/NWRA.COM principal as well.
>>   I wonder if that is the difference.
> 
> Can you please check and compare the output of 'klist -ef' (which includes
> additional info on enctypes and flags) in both cases? It sounds like the macOS
> client is forwarding a ticket that is not accepted by the Linux client's krb5
> library. This could happen if eg. the default_tgs_enctypes configured in
> krb5.conf on the macOS side is incompatible with the permitted_enctypes in
> krb5.conf on the Linux side.

That does indeed seem to be the issue - but it seems strange.

On the mac:

Valid starting       Expires              Service principal
05/07/2025 10:50:16  05/07/2025 20:50:16  krbtgt/AD.NWRA.COM@AD.NWRA.COM
       Flags: FPIA, Etype (skey, tkt): aes256-cts-hmac-sha1-96,
aes256-cts-hmac-sha1-96

On the Linux box:

Valid starting       Expires              Service principal
05/07/2025 11:17:25  05/07/2025 20:50:16  krbtgt/AD.NWRA.COM@AD.NWRA.COM
        Flags: FfPA, Etype (skey, tkt): DEPRECATED:arcfour-hmac,
aes256-cts-hmac-sha1-96


The linux box has crypto-policies:

[libdefaults]
permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
camellia128-cts-cmac

Shouldn't that prevent it from ending up with arcfour-hmac in the first place?

I tried adding this to the mac without any change:

[libdefaults]
permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
camellia128-cts-cmac
default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
camellia128-cts-cmac

Thanks for the reply.

-- 
Orion Poplawski
he/him/his  - surely the least important thing about me
Manager of IT Systems                      720-772-5637
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       orion@nwra.com
Boulder, CO 80301                 https://www.nwra.com/

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4087 bytes --]

Re: Trouble with multiple kerberos ticket caches

[Daniel Kobras]

Hi!

Am 07.05.25 um 19:39 schrieb Orion Poplawski:
> On 5/7/25 10:57, Daniel Kobras wrote:
>> Am 06.05.25 um 21:54 schrieb Orion Poplawski:
>>> More details.  The issue seems to arise when doing gssapi delegation from a
>>> macOS client to the Linux box.  If I have ticket on macOS:
>>>
>>> Ticket cache: API:427671FC-DB63-442F-ACA4-13A9194F4398
>>> Default principal: user@AD.NWRA.COM
>>>
>>> Valid starting     Expires            Service principal
>>> 05/06/25 13:29:54  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>>           renew until 05/13/25 13:29:50
>>> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/NWRA.COM@AD.NWRA.COM
>>> 05/06/25 13:30:30  05/06/25 23:29:54  host/host@NWRA.COM
>>>
>>> and ssh to the Linux box, I can't access the nfs mount:
>>>
>>> -bash: /home/user/.bash_profile: Permission denied
>>>
>>> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
>>> Default principal: user@AD.NWRA.COM
>>>
>>> Valid starting     Expires            Service principal
>>> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>>
>>> I also notice that the ticket is non-renewable.
>>>
>>> If I then kinit I can access the home directory fine.  Other than the new
>>> ticket being renewable I don't see any difference:
>>>
>>> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
>>> Default principal: user@AD.NWRA.COM
>>>
>>> Valid starting     Expires            Service principal
>>> 05/06/25 13:31:27  05/06/25 23:31:27  nfs/server@NWRA.COM
>>>           renew until 05/13/25 13:31:22
>>> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/NWRA.COM@AD.NWRA.COM
>>>           renew until 05/13/25 13:31:22
>>> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>>           renew until 05/13/25 13:31:22
>>>
>>> Actually, I also notice now that there is a krbtgt/NWRA.COM principal as well.
>>>    I wonder if that is the difference.
>>
>> Can you please check and compare the output of 'klist -ef' (which includes
>> additional info on enctypes and flags) in both cases? It sounds like the macOS
>> client is forwarding a ticket that is not accepted by the Linux client's krb5
>> library. This could happen if eg. the default_tgs_enctypes configured in
>> krb5.conf on the macOS side is incompatible with the permitted_enctypes in
>> krb5.conf on the Linux side.
> 
> That does indeed seem to be the issue - but it seems strange.
> 
> On the mac:
> 
> Valid starting       Expires              Service principal
> 05/07/2025 10:50:16  05/07/2025 20:50:16  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>         Flags: FPIA, Etype (skey, tkt): aes256-cts-hmac-sha1-96,
> aes256-cts-hmac-sha1-96
> 
> On the Linux box:
> 
> Valid starting       Expires              Service principal
> 05/07/2025 11:17:25  05/07/2025 20:50:16  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>          Flags: FfPA, Etype (skey, tkt): DEPRECATED:arcfour-hmac,
> aes256-cts-hmac-sha1-96

Can you please re-check the 'klist -ef' output on the mac once you've 
ssh'ed to the Linux box, ie. once also the host ticket is present in the 
ccache? The TGT that ends up on the Linux system gets requested by the 
mac, which doesn't know about the krb5 configuration on the Linux side,
and therefore needs to guess the supported enctypes. MIT's libkrb5 
derives the enctype to request for the forwarded TGT from the ticket for 
the target service, ie. the host service in the case of ssh. On macOS, 
you're likely using a Heimdal-based libkrb5, but I assume it uses a 
similar heuristic.

If the ccache shows that related host ticket also uses RC4 enctypes, 
this could be either due to the default set of ticket enctypes 
configured on the mac, or because the AD computer account for your
Linux system does not allow any stronger enctypes (see 
msDS-supportedEncryptionTypes attribute).

Either case would explain the weak session key in the forwarded TGT. But 
then I wonder why the ssh login works at all, given that your Linux 
system ought to reject the RC4 host ticket due to its restricted set of
permitted_enctypes. So something still doesn't add up.

> The linux box has crypto-policies:
> 
> [libdefaults]
> permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
> camellia128-cts-cmac
> 
> Shouldn't that prevent it from ending up with arcfour-hmac in the first place?
> 
> I tried adding this to the mac without any change:
> 
> [libdefaults]
> permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
> camellia128-cts-cmac
> default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
> camellia128-cts-cmac

Those are options for MIT's libkrb5. Unless you're using a non-default 
stack on the mac, you probably want to use Heimdal's default_etypes, or 
the more specific default_as_etypes/default_tgs_etypes instead.

Kind regards,

Daniel
-- 
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de

-- 
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft: Eisenbahnstraße 1, 72072 
Tübingen

Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer: 
Lukas Kallies, Daniel Kobras, Mark Pröhl

Re: Trouble with kerberos encryption types

[Orion Poplawski]

[-- Attachment #1: Type: text/plain, Size: 9178 bytes --]

On 5/9/25 07:21, Daniel Kobras wrote:
> Hi!
> 
> Am 07.05.25 um 19:39 schrieb Orion Poplawski:
>> On 5/7/25 10:57, Daniel Kobras wrote:
>>> Am 06.05.25 um 21:54 schrieb Orion Poplawski:
>>>> More details.  The issue seems to arise when doing gssapi delegation from a
>>>> macOS client to the Linux box.  If I have ticket on macOS:
>>>>
>>>> Ticket cache: API:427671FC-DB63-442F-ACA4-13A9194F4398
>>>> Default principal: user@AD.NWRA.COM
>>>>
>>>> Valid starting     Expires            Service principal
>>>> 05/06/25 13:29:54  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>>>           renew until 05/13/25 13:29:50
>>>> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/NWRA.COM@AD.NWRA.COM
>>>> 05/06/25 13:30:30  05/06/25 23:29:54  host/host@NWRA.COM
>>>>
>>>> and ssh to the Linux box, I can't access the nfs mount:
>>>>
>>>> -bash: /home/user/.bash_profile: Permission denied
>>>>
>>>> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
>>>> Default principal: user@AD.NWRA.COM
>>>>
>>>> Valid starting     Expires            Service principal
>>>> 05/06/25 13:30:30  05/06/25 23:29:54  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>>>
>>>> I also notice that the ticket is non-renewable.
>>>>
>>>> If I then kinit I can access the home directory fine.  Other than the new
>>>> ticket being renewable I don't see any difference:
>>>>
>>>> Ticket cache: KEYRING:persistent:30657:krb_ccache_efgrZpc
>>>> Default principal: user@AD.NWRA.COM
>>>>
>>>> Valid starting     Expires            Service principal
>>>> 05/06/25 13:31:27  05/06/25 23:31:27  nfs/server@NWRA.COM
>>>>           renew until 05/13/25 13:31:22
>>>> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/NWRA.COM@AD.NWRA.COM
>>>>           renew until 05/13/25 13:31:22
>>>> 05/06/25 13:31:27  05/06/25 23:31:27  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>>>           renew until 05/13/25 13:31:22
>>>>
>>>> Actually, I also notice now that there is a krbtgt/NWRA.COM principal as
>>>> well.
>>>>    I wonder if that is the difference.
>>>
>>> Can you please check and compare the output of 'klist -ef' (which includes
>>> additional info on enctypes and flags) in both cases? It sounds like the macOS
>>> client is forwarding a ticket that is not accepted by the Linux client's krb5
>>> library. This could happen if eg. the default_tgs_enctypes configured in
>>> krb5.conf on the macOS side is incompatible with the permitted_enctypes in
>>> krb5.conf on the Linux side.
>>
>> That does indeed seem to be the issue - but it seems strange.
>>
>> On the mac:
>>
>> Valid starting       Expires              Service principal
>> 05/07/2025 10:50:16  05/07/2025 20:50:16  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>         Flags: FPIA, Etype (skey, tkt): aes256-cts-hmac-sha1-96,
>> aes256-cts-hmac-sha1-96
>>
>> On the Linux box:
>>
>> Valid starting       Expires              Service principal
>> 05/07/2025 11:17:25  05/07/2025 20:50:16  krbtgt/AD.NWRA.COM@AD.NWRA.COM
>>          Flags: FfPA, Etype (skey, tkt): DEPRECATED:arcfour-hmac,
>> aes256-cts-hmac-sha1-96
> 
> Can you please re-check the 'klist -ef' output on the mac once you've ssh'ed
> to the Linux box, ie. once also the host ticket is present in the ccache? The
> TGT that ends up on the Linux system gets requested by the mac, which doesn't
> know about the krb5 configuration on the Linux side,
> and therefore needs to guess the supported enctypes. MIT's libkrb5 derives the
> enctype to request for the forwarded TGT from the ticket for the target
> service, ie. the host service in the case of ssh. On macOS, you're likely
> using a Heimdal-based libkrb5, but I assume it uses a similar heuristic.
> 
> If the ccache shows that related host ticket also uses RC4 enctypes, this
> could be either due to the default set of ticket enctypes configured on the
> mac, or because the AD computer account for your
> Linux system does not allow any stronger enctypes (see msDS-
> supportedEncryptionTypes attribute).

It looks fine on the Mac:

Valid starting     Expires            Service principal
05/09/25 09:05:31  05/09/25 18:32:46  krbtgt/AD.NWRA.COM@AD.NWRA.COM
        renew until 05/15/25 18:11:23, Flags: FfRA
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
05/09/25 09:10:05  05/09/25 18:32:46  krbtgt/NWRA.COM@AD.NWRA.COM
        Flags: FfA, Etype (skey, tkt): aes256-cts-hmac-sha1-96,
aes256-cts-hmac-sha1-96
05/09/25 09:10:05  05/09/25 18:32:46  host/LINUXBOX@NWRA.COM
        Flags: FfAT, Etype (skey, tkt): aes256-cts-hmac-sha1-96,
aes256-cts-hmac-sha1-96


> Either case would explain the weak session key in the forwarded TGT. But then
> I wonder why the ssh login works at all, given that your Linux system ought to
> reject the RC4 host ticket due to its restricted set of
> permitted_enctypes. So something still doesn't add up.

Things are definitely not adding up!

>> The linux box has crypto-policies:
>>
>> [libdefaults]
>> permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>> camellia128-cts-cmac
>>
>> Shouldn't that prevent it from ending up with arcfour-hmac in the first place?
>>
>> I tried adding this to the mac without any change:
>>
>> [libdefaults]
>> permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>> camellia128-cts-cmac
>> default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>> camellia128-cts-cmac
> 
> Those are options for MIT's libkrb5. Unless you're using a non-default stack
> on the mac, you probably want to use Heimdal's default_etypes, or the more
> specific default_as_etypes/default_tgs_etypes instead.

I ended up slimming down to:

  permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
  default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
  default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96

but those are the option names from man krb5.conf on the mac.

I've set msDS-SupportedEncryptionTypes for the AD user to only allow aes to no
avail.

This seems like arcfour is disabled:

$ kvno host/LINUXBOX -e arcfour-hmac
kvno: KDC has no support for encryption type while getting credentials for
host/LINUXBOX
$ kvno krbtgt/AD.NWRA.COM@AD.NWRA.COM -e arcfour-hmac
kvno: KDC has no support for encryption type while getting credentials for
krbtgt/AD.NWRA.COM@AD.NWRA.COM

I tried enabling KRB5_TRACE during the ssh command, but it does not seem useful:

2025-05-09T09:15:21 set-error: -1765328234: entypes not supported
2025-05-09T09:15:21 set-error: -1765328243: Did not find credential for
krb5_ccache_conf_data/realm-config@X-CACHECONF: in cache
FILE:/tmp/krb5cc_1504398909_0a9FHZRC0q
2025-05-09T09:15:21 set-error: -1765328243: Did not find credential for
krb5_ccache_conf_data/realm-config@X-CACHECONF: in cache
FILE:/tmp/krb5cc_1504398909_0a9FHZRC0q
2025-05-09T09:15:21 set-error: -1765328243: Did not find credential for
krb5_ccache_conf_data/time-offset/host\134/LINUXBOX\134@AD.NWRA.COM@X-CACHECONF:
in cache FILE:/tmp/krb5cc_1504398909_0a9FHZRC0q
2025-05-09T09:15:21 Setting up PFS for auth context
2025-05-09T09:15:21 set-error: -1765328234: Encryption type
des-cbc-md5-deprecated not supported
2025-05-09T09:15:21 set-error: -1765328234: Encryption type
des-cbc-md4-deprecated not supported
2025-05-09T09:15:21 set-error: -1765328234: Encryption type
des-cbc-crc-deprecated not supported
2025-05-09T09:15:21 set-error: -1765328234: entypes not supported
2025-05-09T09:15:21 set-error: -1765328243: Did not find credential for
krb5_ccache_conf_data/realm-config@X-CACHECONF: in cache
FILE:/tmp/krb5cc_1504398909_0a9FHZRC0q
2025-05-09T09:15:21 set-error: -1765328243: Did not find credential for
krb5_ccache_conf_data/realm-config@X-CACHECONF: in cache
FILE:/tmp/krb5cc_1504398909_0a9FHZRC0q
2025-05-09T09:15:21 set-error: -1765328243: Did not find credential for
krb5_ccache_conf_data/time-offset/host\134/LINUXBOX\134@AD.NWRA.COM@X-CACHECONF:
in cache FILE:/tmp/krb5cc_1504398909_0a9FHZRC0q
2025-05-09T09:15:21 Setting up PFS for auth context
2025-05-09T09:15:21 set-error: -1765328234: Encryption type
des-cbc-md5-deprecated not supported
2025-05-09T09:15:21 set-error: -1765328234: Encryption type
des-cbc-md4-deprecated not supported
2025-05-09T09:15:21 set-error: -1765328234: Encryption type
des-cbc-crc-deprecated not supported
2025-05-09T09:15:22 krb5_rd_rep not using PFS

-- 
Orion Poplawski
he/him/his  - surely the least important thing about me
Manager of IT Systems                      720-772-5637
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       orion@nwra.com
Boulder, CO 80301                 https://www.nwra.com/

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4087 bytes --]

Re: Trouble with kerberos encryption types

[Orion Poplawski]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

On 5/9/25 13:55, Orion Poplawski wrote:
> On 5/9/25 07:21, Daniel Kobras wrote:
>> Hi!
>>
>> Am 07.05.25 um 19:39 schrieb Orion Poplawski:
>>> I tried adding this to the mac without any change:
>>>
>>> [libdefaults]
>>> permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>>> camellia128-cts-cmac
>>> default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>>> camellia128-cts-cmac
>>
>> Those are options for MIT's libkrb5. Unless you're using a non-default stack
>> on the mac, you probably want to use Heimdal's default_etypes, or the more
>> specific default_as_etypes/default_tgs_etypes instead.
> 
> I ended up slimming down to:
> 
>   permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
>   default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
>   default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
> 
> but those are the option names from man krb5.conf on the mac.

I should have trusted you :) - I finally came across this:

https://services.dartmouth.edu/TDClient/1806/Portal/KB/ArticleDet?ID=89203

which has:

        default_etypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96

and setting that does fix the skey encryption type.

I'm still stuck with the non-renewable ticket that they mention as well, so it
seems like GSSAPI auth from a mac is not very useful.

Thank you very much for your help.

-- 
Orion Poplawski
he/him/his  - surely the least important thing about me
Manager of IT Systems                      720-772-5637
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       orion@nwra.com
Boulder, CO 80301                 https://www.nwra.com/

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4087 bytes --]

Re: Trouble with kerberos encryption types

[Daniel Kobras]

Hi!

Am 09.05.25 um 23:03 schrieb Orion Poplawski:
> On 5/9/25 13:55, Orion Poplawski wrote:
>> On 5/9/25 07:21, Daniel Kobras wrote:
>>> Hi!
>>>
>>> Am 07.05.25 um 19:39 schrieb Orion Poplawski:
>>>> I tried adding this to the mac without any change:
>>>>
>>>> [libdefaults]
>>>> permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>>>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>>>> camellia128-cts-cmac
>>>> default_tgs_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
>>>> aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac
>>>> camellia128-cts-cmac
>>>
>>> Those are options for MIT's libkrb5. Unless you're using a non-default stack
>>> on the mac, you probably want to use Heimdal's default_etypes, or the more
>>> specific default_as_etypes/default_tgs_etypes instead.
>>
>> I ended up slimming down to:
>>
>>    permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
>>    default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
>>    default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
>>
>> but those are the option names from man krb5.conf on the mac.
> 
> I should have trusted you :) - I finally came across this:
> 
> https://services.dartmouth.edu/TDClient/1806/Portal/KB/ArticleDet?ID=89203
> 
> which has:
> 
>          default_etypes = aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96
> 
> and setting that does fix the skey encryption type.

Thanks for the update. So the heuristic to derive the enctype for the 
forwarded TGT from the service ticket seems to be specific to MIT, 
whereas Heimdal apparently just uses a client-side default.

> I'm still stuck with the non-renewable ticket that they mention as well, so it
> seems like GSSAPI auth from a mac is not very useful.

That's another Heimdal-specific behavior, which means that the renewal 
of delegated tickets needs to be driven from the delegating side. With 
OpenSSH, this is possible with the following settings:

ssh server:
   GSSAPIKeyExchange yes
   GSSAPIStoreCredentialsOnRekey yes

ssh client:
   GSSAPIKeyExchange yes
   GSSAPIRenewalForcesRekey yes


If this is not sufficient because the NFS client requires tickets beyond 
the lifetime of the ssh session, then maybe a different approach like 
gssproxy's impersonation feature (cf. 
<https://github.com/gssapi/gssproxy/blob/main/docs/NFS.md#user-impersonation-via-constrained-delegation>) 
is more suitable?

Kind regards,

Daniel
-- 
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de

-- 
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft: Eisenbahnstraße 1, 72072 
Tübingen

Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer: 
Lukas Kallies, Daniel Kobras, Mark Pröhl