Linux kernel -stable discussions
 help / color / mirror / Atom feed
* [Regression linux-6.1.y]: NFSv4 server: sec=krb5p mount fails with EIO in 6.1.192
@ 2026-11-20 13:17 Max Vozeler
  2026-11-21 10:28 ` Max Vozeler
  0 siblings, 1 reply; 3+ messages in thread
From: Max Vozeler @ 2026-11-20 13:17 UTC (permalink / raw)
  To: linux-nfs; +Cc: stable, regressions

#regzbot introduced: v6.1.190..v6.1.192

Hi,

after upgrading an NFS server (Debian bookworm) to 6.1.192
(Debian linux-image-6.1.0-55-amd64), NFSv4 clients fail to mount exports
using sec=krb5p with EIO.

Downgrading the server back to 6.1.190 (linux-image-6.1.0-54-amd64)
makes things work again.

Good: 6.1.190 (Debian linux-image-6.1.0-54-amd64)
Bad:  6.1.192 (Debian linux-image-6.1.0-55-amd64, linux-image-6.1.0-56-amd64)

sec=krb5i works on the exact same server, export, and Kerberos keytab;
only sec=krb5p fails.

# mount -vvv -t nfs4 -o sec=krb5p,vers=4.0 10.x.y.z:/<export> /mnt
mount.nfs4: timeout set for Fri Nov 20 08:10:07 2026
mount.nfs4: trying text-based options 'sec=krb5p,nfsvers=4.0,addr=10.x.y.z,clientaddr=10.x.y.a'
mount.nfs4: mount(2): Input/output error
mount.nfs4: mount system call failed

Server dmesg with rpcdebug enabled:

[42095.312947] svc: server 0000000039bc31cb, pool 0, transport 00000000cfa39484, inuse=2
[42095.312971] svc: server 0000000072d08ff5, pool 0, transport 00000000cfa39484, inuse=3
[42095.312975] svc: svc_authenticate (0)
[42095.324501] svc: server 0000000072d08ff5, pool 0, transport 00000000c60f4fcd, inuse=2
[42095.334305] svc: server 0000000072d08ff5, pool 0, transport 00000000c60f4fcd, inuse=2
[42095.334324] svc: svc_authenticate (6)
[42095.334331] svc: server 0000000039bc31cb, pool 0, transport 00000000c60f4fcd, inuse=3
[42095.334388] RPC:       xs_local_send_request(1756) = 0
[42095.338142] RPC:       krb5_encrypt returns 0
[42095.338144] RPC:       krb5_encrypt returns 0
[42095.338148] RPC:       krb5_encrypt returns 0
[42095.338148] RPC:       krb5_encrypt returns 0
[42095.338151] RPC:       krb5_encrypt returns 0
[42095.338151] RPC:       krb5_encrypt returns 0
[42095.338154] RPC:       krb5_encrypt returns 0
[42095.338154] RPC:       krb5_encrypt returns 0
[42095.338156] RPC:       krb5_encrypt returns 0
[42095.338156] RPC:       krb5_encrypt returns 0
[42095.338158] RPC:       krb5_encrypt returns 0
[42095.338158] RPC:       krb5_encrypt returns 0
[42095.338160] RPC:       gss_import_sec_context_kerberos: returning 0
[42095.338163] RPC:       gss_get_mic_v2
[42095.338546] svc: server 0000000072d08ff5, pool 0, transport 00000000cfa39484, inuse=2
[42095.338554] svc: svc_authenticate (6)
[42095.338555] RPC:       gss_verify_mic_v2
[42095.338557] RPC:       gss_get_mic_v2
[42095.338557] svc: server 0000000039bc31cb, pool 0, transport 00000000cfa39484, inuse=3
[42095.338559] RPC:       gss_verify_mic_v2
[42095.338573] nfsd4_exchange_id rqstp=0000000072d08ff5 exid=0000000096ec91a4 clname.len=27 clname.data=0000000035150242 ip_addr=10.x.y.50 flags 101, spa_how 1
[42095.338592] nfsd4_exchange_id seqid 0 flags 20001
[42095.338594] RPC:       gss_get_mic_v2
[42095.338777] svc: server 0000000072d08ff5, pool 0, transport 00000000cfa39484, inuse=2
[42095.338783] svc: svc_authenticate (6)
[42095.338784] RPC:       gss_verify_mic_v2
[42095.338786] RPC:       gss_get_mic_v2
[42095.338786] svc: server 0000000039bc31cb, pool 0, transport 00000000cfa39484, inuse=3
[42095.338787] RPC:       gss_verify_mic_v2
[42095.338790] nfsd4_exchange_id rqstp=0000000072d08ff5 exid=0000000096ec91a4 clname.len=27 clname.data=0000000035150242 ip_addr=10.x.y.50 flags 101, spa_how 1
[42095.338799] nfsd4_exchange_id seqid 0 flags 20001
[42095.338834] RPC:       gss_get_mic_v2
[42095.338953] svc: server 0000000072d08ff5, pool 0, transport 00000000cfa39484, inuse=2
[42095.338958] svc: svc_authenticate (6)
[42095.338958] RPC:       gss_verify_mic_v2
[42095.338960] RPC:       gss_get_mic_v2
[42095.338960] svc: server 0000000039bc31cb, pool 0, transport 00000000cfa39484, inuse=3
[42095.338962] RPC:       gss_verify_mic_v2
[42095.338982] check_slot_seqid enter. seqid 1 slot_seqid 0
[42095.338998] RPC:       set up xprt to 10.x.y.50 (port 0) via tcp
[42095.339057] RPC:       gss_get_mic_v2
[42095.339264] svc: server 0000000072d08ff5, pool 0, transport 00000000cfa39484, inuse=4
[42095.339267] svc: svc_authenticate (6)
[42095.339268] RPC:       gss_verify_mic_v2
[42095.339270] RPC:       gss_get_mic_v2
[42095.339271] RPC:       gss_verify_mic_v2
[42095.339271] svc: server 0000000039bc31cb, pool 0, transport 00000000cfa39484, inuse=5
[42095.339274] __find_in_sessionid_hashtbl: 1795172831:1033077859:4:0
[42095.339284] nfsd4_sequence: slotid 0
[42095.339285] check_slot_seqid enter. seqid 1 slot_seqid 0
[42095.339287] alloc_cld_upcall: allocated xid 9
[42095.340556] --> nfsd4_store_cache_entry slot 000000008b186fa8
[42095.340560] RPC:       gss_get_mic_v2
[42095.343136] svc: server 0000000072d08ff5, pool 0, transport 00000000f9548822, inuse=2
[42095.344680] svc: server 0000000072d08ff5, pool 0, transport 00000000f9548822, inuse=2
[42095.344687] svc: svc_authenticate (6)
[42095.344690] svc: server 0000000039bc31cb, pool 0, transport 00000000f9548822, inuse=3
[42095.344757] RPC:       xs_local_send_request(1756) = 0
[42095.347090] RPC:       krb5_encrypt returns 0
[42095.347092] RPC:       krb5_encrypt returns 0
[42095.347096] RPC:       krb5_encrypt returns 0
[42095.347096] RPC:       krb5_encrypt returns 0
[42095.347099] RPC:       krb5_encrypt returns 0
[42095.347099] RPC:       krb5_encrypt returns 0
[42095.347101] RPC:       krb5_encrypt returns 0
[42095.347101] RPC:       krb5_encrypt returns 0
[42095.347103] RPC:       krb5_encrypt returns 0
[42095.347104] RPC:       krb5_encrypt returns 0
[42095.347105] RPC:       krb5_encrypt returns 0
[42095.347105] RPC:       krb5_encrypt returns 0
[42095.347107] RPC:       gss_import_sec_context_kerberos: returning 0
[42095.347109] RPC:       gss_get_mic_v2
[42095.347456] svc: server 0000000072d08ff5, pool 0, transport 00000000cfa39484, inuse=4
[42095.347468] svc: svc_authenticate (6)
[42095.347469] RPC:       gss_verify_mic_v2
[42095.347471] RPC:       gss_get_mic_v2
[42095.347471] svc: server 0000000039bc31cb, pool 0, transport 00000000cfa39484, inuse=5
[42095.347473] RPC:       gss_unwrap_kerberos_v2
[42095.347646] RPC:       bc_destroy xprt 00000000488733b3
[42095.348467] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.348472] svc: svc_authenticate (6)
[42095.348474] RPC:       gss_verify_mic_v2
[42095.348481] RPC:       gss_get_mic_v2
[42095.348483] RPC:       gss_unwrap_kerberos_v2
[42095.348507] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.348713] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.348752] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.348758] svc: svc_authenticate (6)
[42095.348759] RPC:       gss_verify_mic_v2
[42095.348761] RPC:       gss_get_mic_v2
[42095.348763] RPC:       gss_unwrap_kerberos_v2
[42095.348762] svc: server 0000000039bc31cb, pool 0, transport 000000005d77f59c, inuse=3
[42095.365858] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.365867] svc: svc_authenticate (6)
[42095.365870] RPC:       gss_verify_mic_v2
[42095.365869] svc: server 0000000039bc31cb, pool 0, transport 000000005d77f59c, inuse=3
[42095.365877] RPC:       gss_get_mic_v2
[42095.365877] svc: svc_authenticate (6)
[42095.365880] RPC:       gss_delete_sec_context deleting 000000004f9c6594
[42095.365881] RPC:       gss_verify_mic_v2
[42095.365892] RPC:       gss_get_mic_v2
[42095.365894] RPC:       gss_verify_mic_v2
[42095.365900] nfsd4_destroy_session: 1795172831:1033077859:4:0
[42095.365901] __find_in_sessionid_hashtbl: 1795172831:1033077859:4:0
[42095.365903] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=3
[42095.365928] RPC:       gss_get_mic_v2
[42095.366086] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.366091] svc: svc_authenticate (6)
[42095.366092] RPC:       gss_verify_mic_v2
[42095.366094] RPC:       gss_get_mic_v2
[42095.366094] svc: server 0000000039bc31cb, pool 0, transport 000000005d77f59c, inuse=3
[42095.366095] RPC:       gss_verify_mic_v2
[42095.366101] alloc_cld_upcall: allocated xid 10
[42095.367452] RPC:       gss_get_mic_v2
[42095.367647] svc: server 0000000072d08ff5, pool 0, transport 000000005d77f59c, inuse=2
[42095.367653] svc: svc_authenticate (6)
[42095.367655] RPC:       gss_verify_mic_v2
[42095.367655] svc: server 0000000039bc31cb, pool 0, transport 000000005d77f59c, inuse=3
[42095.367658] RPC:       gss_get_mic_v2
[42095.367660] RPC:       gss_delete_sec_context deleting 0000000023931779

Let me know if there is any useful information I can provide.

Best regards,
Max

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [Regression linux-6.1.y]: NFSv4 server: sec=krb5p mount fails with EIO in 6.1.192
  2026-11-20 13:17 [Regression linux-6.1.y]: NFSv4 server: sec=krb5p mount fails with EIO in 6.1.192 Max Vozeler
@ 2026-11-21 10:28 ` Max Vozeler
  2026-11-21 12:30   ` Max Vozeler
  0 siblings, 1 reply; 3+ messages in thread
From: Max Vozeler @ 2026-11-21 10:28 UTC (permalink / raw)
  To: linux-nfs; +Cc: stable, regressions

#regzbot introduced: v6.1.190..v6.1.191

On Fri, Nov 20, 2026 at 02:17:43PM +0100, Max Vozeler wrote:
> after upgrading an NFS server (Debian bookworm) to 6.1.192
> (Debian linux-image-6.1.0-55-amd64), NFSv4 clients fail to mount exports
> using sec=krb5p with EIO.
> 
> Downgrading the server back to 6.1.190 (linux-image-6.1.0-54-amd64)
> makes things work again.
> 
> Good: 6.1.190 (Debian linux-image-6.1.0-54-amd64)
> Bad:  6.1.192 (Debian linux-image-6.1.0-55-amd64, linux-image-6.1.0-56-amd64)

Reproduced with vanilla 6.1.191 and 6.1.192

Good: 6.1.190
Bad: 6.1.191, 6.1.192

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [Regression linux-6.1.y]: NFSv4 server: sec=krb5p mount fails with EIO in 6.1.192
  2026-11-21 10:28 ` Max Vozeler
@ 2026-11-21 12:30   ` Max Vozeler
  0 siblings, 0 replies; 3+ messages in thread
From: Max Vozeler @ 2026-11-21 12:30 UTC (permalink / raw)
  To: Chuck Lever, Sasha Levin, Greg Kroah-Hartman, Chris Mason,
	Jeff Layton
  Cc: linux-nfs, stable, regressions

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

#regzbot introduced: 82cf21374fb7ee55b37be9afb6704ba0eeffb236

On Sat, Nov 21, 2026 at 11:28:53AM +0100, Max Vozeler wrote:
> On Fri, Nov 20, 2026 at 02:17:43PM +0100, Max Vozeler wrote:
> > after upgrading an NFS server (Debian bookworm) to 6.1.192
> > (Debian linux-image-6.1.0-55-amd64), NFSv4 clients fail to mount exports
> > using sec=krb5p with EIO.
> > 
> > Downgrading the server back to 6.1.190 (linux-image-6.1.0-54-amd64)
> > makes things work again.
> > 
> > Good: 6.1.190 (Debian linux-image-6.1.0-54-amd64)
> > Bad:  6.1.192 (Debian linux-image-6.1.0-55-amd64, linux-image-6.1.0-56-amd64)
> 
> Reproduced with vanilla 6.1.191 and 6.1.192
> 
> Good: 6.1.190
> Bad: 6.1.191, 6.1.192

bisect points to

commit 82cf21374fb7ee55b37be9afb6704ba0eeffb236
Author: Chuck Lever <chuck.lever@oracle.com>
Date:   Mon Jan 2 12:07:07 2023 -0500

    SUNRPC: Convert unwrap_priv_data() to use xdr_stream
    
    [ Upstream commit 42140718ea26c47fb06c679451a6aa6703545136 ]
    
    Done as part of hardening the server-side RPC header decoding path.
    
    Reviewed-by: Jeff Layton <jlayton@kernel.org>
    Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
    Stable-dep-of: a919c5c88769 ("SUNRPC: svcauth_gss: enforce krb5 token minimum length")
    Signed-off-by: Sasha Levin <sashal@kernel.org>

This seems to have been part of a larger series, only parts of which
were backported for a919c5c88769 ?

Best regards,
Max

[-- Attachment #2: bisect.log --]
[-- Type: text/plain, Size: 1637 bytes --]

git bisect start
# status: waiting for both good and bad commits
# bad: [08889840163ebcdb7e24bf7ec311bc2e6fc3aaea] Linux 6.1.191
git bisect bad 08889840163ebcdb7e24bf7ec311bc2e6fc3aaea
# status: waiting for good commit(s), bad commit known
# good: [79807e970724c1f90cee1316361d61b80db4eef6] Linux 6.1.190
git bisect good 79807e970724c1f90cee1316361d61b80db4eef6
# good: [91c2a5d201daf66fa3f42b8a383bb89174bde2c8] serial: core: fix infinite loop in handle_tx() for PORT_UNKNOWN
git bisect good 91c2a5d201daf66fa3f42b8a383bb89174bde2c8
# bad: [ab8c8e027fa8f9314340a96999dd3309b7f80d61] fbdev: ssd1307fb: defer I2C transfers from damage callbacks
git bisect bad ab8c8e027fa8f9314340a96999dd3309b7f80d61
# good: [35b46918d5deee01fade7e2f2075a7c238a2a0e8] cifs: prevent readdir from changing file size due to stale directory metadata
git bisect good 35b46918d5deee01fade7e2f2075a7c238a2a0e8
# good: [91abd8383fd0588d1fcc31120d6344150a64ba5a] net: sched: cls_api: add filter counter
git bisect good 91abd8383fd0588d1fcc31120d6344150a64ba5a
# bad: [ed126ca71d4d949ad99f40b974a8f7685e808f8b] net/packet: defer vmalloc TX_RING free until skbs finish
git bisect bad ed126ca71d4d949ad99f40b974a8f7685e808f8b
# bad: [82cf21374fb7ee55b37be9afb6704ba0eeffb236] SUNRPC: Convert unwrap_priv_data() to use xdr_stream
git bisect bad 82cf21374fb7ee55b37be9afb6704ba0eeffb236
# good: [8da6f765098d46e5b3178ef4b2f8273c9ba1d3bb] SUNRPC: Rename automatic variables in unwrap_priv_data()
git bisect good 8da6f765098d46e5b3178ef4b2f8273c9ba1d3bb
# first bad commit: [82cf21374fb7ee55b37be9afb6704ba0eeffb236] SUNRPC: Convert unwrap_priv_data() to use xdr_stream

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-21 12:30 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 13:17 [Regression linux-6.1.y]: NFSv4 server: sec=krb5p mount fails with EIO in 6.1.192 Max Vozeler
2026-11-21 10:28 ` Max Vozeler
2026-11-21 12:30   ` Max Vozeler

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox