Re: ls input/output error ("NFS: readdir(/) returns -5") on krb5 NFSv4 client using SHA2

Scott Mayhew
2025-11-18T17:53:02Z

On Tue, 18 Nov 2025, Tyler W. Ross wrote:

> On 11/17/25 3:54 PM, Scott Mayhew wrote:
> > FWIW I have both Debian Trixie and Sid/Forky VMs, and krb5{,i,p} is
> > working across the board for me. Normally I just use a plain MIT KDC,
> > so I tried IPA and that works fine too.
>
> Did you confirm the enctype used?

Yes. This is how I was testing:

root@forky:~# uname -r
6.17.7+deb14+1-amd64
root@forky:~# systemctl restart rpc-gssd
root@forky:~# klist -ce /tmp/krb5ccmachine_SMAYHEW.TEST
klist: No credentials cache found (filename: /tmp/krb5ccmachine_SMAYHEW.TEST)
root@forky:~# for serv in forky trixie rawhide rhel10 rhel9; do for flav in krb5 krb5i krb5p; do mount -o v4.2,sec=$flav $serv.smayhew.test:/export /mnt/t; ls -lR /mnt/t >/dev/null; umount /mnt/t; done; done
root@forky:~# klist -ce /tmp/krb5ccmachine_SMAYHEW.TEST
Ticket cache: FILE:/tmp/krb5ccmachine_SMAYHEW.TEST
Default principal: nfs/forky.smayhew.test@SMAYHEW.TEST

Valid starting Expires Service principal
11/14/25 14:53:03 11/15/25 14:53:03 krbtgt/SMAYHEW.TEST@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
11/14/25 14:53:03 11/15/25 14:53:03 nfs/forky.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
11/14/25 14:53:03 11/15/25 14:53:03 nfs/trixie.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
11/14/25 14:53:03 11/15/25 14:53:03 nfs/rawhide.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
11/14/25 14:53:04 11/15/25 14:53:03 nfs/rhel10.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
11/14/25 14:53:05 11/15/25 14:53:03 nfs/rhel9.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192

>
> My repro steps, from initial mounted state:
> kinit
> kvno -e aes256-cts-hmac-sha384-192 <nfs spn>
> ls /mnt/example
>
> On my Debian Sid VM, if I do kinit and then immediately ls, the issue
> does not occur. klist shows the acquired service ticket has an
> aes256-cts-hmac-sha1-96 session key.

Oh! I see the problem. If the automatically acquired service ticket
for a normal user is using aes256-cts-hmac-sha1-96, then I'm assuming
the machine credential is also using aes256-cts-hmac-sha1-96.
Run 'klist -ce /tmp/krb5ccmachine_IPA.TWRLAB.NET' to check. You can't
use 'kvno -e' to choose a different encryption type. Why are you doing
that? Is it because you want to use the stronger encryption types? In
that case, the proper way to do this would be to manually add this line
to the "[libdefaults]" stanza of your /etc/krb5.conf:

permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96

and get rid of allowed-enctypes settings that you may have added to
/etc/nfs.conf. Then unmount, run 'systemctl restart rpc-gssd', remount,
etc. and your system should be using aes256-cts-hmac-sha384-192 by default.

RHEL/CentOS/Fedora all ship a package called "crypto-policies" that
include system-wide configurations for various crypto packages. For
kerberos, it drops a config snippet in /etc/krb5.conf.d similar to what
I have above. AFAICT Suse has this package too, but it appears Debian
does not.

Without the permitted_enctypes setting, the kerberos library will fall
back to the default settings, which according to krb5.conf(5)

---8<---
permitted_enctypes
Identifies the encryption types that servers will permit for ses‐
sion keys and for ticket and authenticator encryption, ordered by
preference from highest to lowest. Starting in release 1.18,
this tag also acts as the default value for default_tgs_enctypes
and default_tkt_enctypes. The default value for this tag is
aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128
des3-cbc-sha1 arcfour-hmac-md5 camellia256-cts-cmac camel‐
lia128-cts-cmac.
---8<---

If I remove that line from my krb5.conf and use 'kvno -e' like your
test, then I can reproduce the behavior you're seeing:

root@forky:~# systemctl restart rpc-gssd
root@forky:~# mount -o v4.2,sec=krb5 trixie.smayhew.test:/export /mnt/t
root@forky:~# klist -ce /tmp/krb5ccmachine_SMAYHEW.TEST
Ticket cache: FILE:/tmp/krb5ccmachine_SMAYHEW.TEST
Default principal: nfs/forky.smayhew.test@SMAYHEW.TEST

Valid starting Expires Service principal
11/18/25 17:41:29 11/19/25 17:15:04 krbtgt/SMAYHEW.TEST@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha1-96, camellia256-cts-cmac
11/18/25 17:41:29 11/19/25 17:15:04 nfs/trixie.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha384-192
root@forky:~# su - smayhew
smayhew@forky:~$ kinit
Password for smayhew@SMAYHEW.TEST:
smayhew@forky:~$ kvno -e aes256-cts-hmac-sha384-192 nfs/trixie.smayhew.test
nfs/trixie.smayhew.test@SMAYHEW.TEST: kvno = 1
smayhew@forky:~$ klist -ce
Ticket cache: KEYRING:persistent:1052000003:1052000003
Default principal: smayhew@SMAYHEW.TEST

Valid starting Expires Service principal
11/18/25 17:41:53 11/19/25 17:20:27 nfs/trixie.smayhew.test@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
11/18/25 17:41:39 11/19/25 17:20:27 krbtgt/SMAYHEW.TEST@SMAYHEW.TEST
Etype (skey, tkt): aes256-cts-hmac-sha1-96, camellia256-cts-cmac
smayhew@forky:~$ ls /mnt/t
ls: reading directory '/mnt/t': Input/output error
smayhew@forky:~$
logout
root@forky:~# grep overflow /sys/kernel/debug/tracing/trace
ls-2032 [002] ..... 3025.593816: rpc_xdr_overflow: task:00000009@00000006 nfsv4 READDIR requested=8 p=00000000dfba8950 end=00000000b97e329e xdr=[00000000389cc91a,132]/4008/[00000000b97e329e,4]/988

-Scott
>
>
> TWR
>