* Duplicated RPC client initialization with flexfiles layout
@ 2026-10-02 12:33 Mkrtchyan, Tigran
0 siblings, 0 replies; only message in thread
From: Mkrtchyan, Tigran @ 2026-10-02 12:33 UTC (permalink / raw)
To: linux-nfs
[-- Attachment #1: Type: text/plain, Size: 6738 bytes --]
Dear NFS fellows,
Your users were able to trigger a weird bug in rpc/flexfiles/our server.
The behavior is that the pNFS flexfiles client tries to create connections
to 6 data servers in parallel. However, the initialization to each DS happens
twice; for that, the connection is dropped at the TCP level. The wireshark
packes of one of the DSes and bpftrace-based tracing of rpc_create_xprt
functions are below.
This is with REHL 5.14.0-687.52.1 (at the moment); I will check whether I can reproduce it with the upstream kernel as well.
Any ideas?
Best regards,
Tigran.
1 0.000000000 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 763 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321872006 TSecr=0 WS=512
2 0.000000000 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 [TCP Out-Of-Order] 763 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321872006 TSecr=0 WS=512
3 0.000043716 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 24006 → 763 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196237843 TSecr=3321872006 WS=128
4 0.000049947 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 [TCP Out-Of-Order] 24006 → 763 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196237843 TSecr=3321872006 WS=128
5 0.000101734 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 941 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321872006 TSecr=0 WS=512
6 0.000101734 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 [TCP Out-Of-Order] 941 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321872006 TSecr=0 WS=512
7 0.000130403 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 24006 → 941 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196237844 TSecr=3321872006 WS=128
8 0.000135463 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 [TCP Out-Of-Order] 24006 → 941 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196237844 TSecr=3321872006 WS=128
9 0.000134812 1xx.yy.182.55 → 1xx.yy.50.150 TCP 62 763 → 24006 [RST] Seq=1 Win=0 Len=0
10 0.000134812 1xx.yy.182.55 → 1xx.yy.50.150 TCP 62 763 → 24006 [RST] Seq=1 Win=0 Len=0
11 0.000243066 1xx.yy.182.55 → 1xx.yy.50.150 TCP 68 941 → 24006 [ACK] Seq=1 Ack=1 Win=64512 Len=0 TSval=3321872006 TSecr=196237844
12 0.000243066 1xx.yy.182.55 → 1xx.yy.50.150 TCP 68 [TCP Dup ACK 11#1] 941 → 24006 [ACK] Seq=1 Ack=1 Win=64512 Len=0 TSval=3321872006 TSecr=196237844
13 0.000479359 1xx.yy.182.55 → 1xx.yy.50.150 TCP 68 941 → 24006 [FIN, ACK] Seq=1 Ack=1 Win=64512 Len=0 TSval=3321872006 TSecr=196237844
14 0.000479359 1xx.yy.182.55 → 1xx.yy.50.150 TCP 68 [TCP Out-Of-Order] 941 → 24006 [FIN, ACK] Seq=1 Ack=1 Win=64512 Len=0 TSval=3321872006 TSecr=196237844
15 0.000595959 1xx.yy.50.150 → 1xx.yy.182.55 TCP 68 24006 → 941 [FIN, ACK] Seq=1 Ack=2 Win=65280 Len=0 TSval=196237844 TSecr=3321872006
16 0.000603322 1xx.yy.50.150 → 1xx.yy.182.55 TCP 68 [TCP Out-Of-Order] 24006 → 941 [FIN, ACK] Seq=1 Ack=2 Win=65280 Len=0 TSval=196237844 TSecr=3321872006
17 0.000688326 1xx.yy.182.55 → 1xx.yy.50.150 TCP 68 941 → 24006 [ACK] Seq=2 Ack=2 Win=64512 Len=0 TSval=3321872007 TSecr=196237844
18 0.000688326 1xx.yy.182.55 → 1xx.yy.50.150 TCP 68 [TCP Dup ACK 17#1] 941 → 24006 [ACK] Seq=2 Ack=2 Win=64512 Len=0 TSval=3321872007 TSecr=196237844
19 1.023913451 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 716 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321873030 TSecr=0 WS=512
20 1.023913451 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 [TCP Out-Of-Order] 716 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321873030 TSecr=0 WS=512
21 1.023967704 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 24006 → 716 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196238867 TSecr=3321873030 WS=128
22 1.023976295 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 [TCP Out-Of-Order] 24006 → 716 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196238867 TSecr=3321873030 WS=128
23 1.023981641 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 708 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321873030 TSecr=0 WS=512
24 1.023981641 1xx.yy.182.55 → 1xx.yy.50.150 TCP 76 [TCP Out-Of-Order] 708 → 24006 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3321873030 TSecr=0 WS=512
25 1.024021399 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 24006 → 708 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196238867 TSecr=3321873030 WS=128
26 1.024027734 1xx.yy.50.150 → 1xx.yy.182.55 TCP 76 [TCP Out-Of-Order] 24006 → 708 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=196238867 TSecr=3321873030 WS=128
27 1.024056233 1xx.yy.182.55 → 1xx.yy.50.150 TCP 62 716 → 24006 [RST] Seq=1 Win=0 Len=0
28 1.024056233 1xx.yy.182.55 → 1xx.yy.50.150 TCP 62 716 → 24006 [RST] Seq=1 Win=0 Len=0
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.51.163:24010 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.51.163:24010 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.50.147:24001 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.50.147:24001 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.50.150:24006 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.50.150:24006 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.51.218:24011 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.51.218:24011 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.50.59:24005 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.50.59:24005 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.52.47:24001 err=-512
2026-10-02 14:14:25: rpc_create_xprt failed: 1xx.yy.52.47:24001 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.51.163:24010 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.51.163:24010 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.50.147:24001 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.50.147:24001 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.50.150:24006 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.50.150:24006 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.51.218:24011 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.51.218:24011 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.50.59:24005 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.50.59:24005 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.52.47:24001 err=-512
2026-10-02 14:14:26: rpc_create_xprt failed: 1xx.yy.52.47:24001 err=-512
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2309 bytes --]
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2026-10-02 12:33 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-02 12:33 Duplicated RPC client initialization with flexfiles layout Mkrtchyan, Tigran
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.