* NFS client failure [not found] <c4f1d8bf-745b-4a98-9d38-2da4c355d691@gmail.com> @ 2024-07-29 21:56 ` marc eshel 2024-08-05 21:51 ` NFS client to pNFS DS marc eshel 0 siblings, 1 reply; 17+ messages in thread From: marc eshel @ 2024-07-29 21:56 UTC (permalink / raw) To: linux-nfs@vger.kernel.org On 7/29/24 2:50 PM, marc eshel wrote: > > Using NFS client to connect (before a read) from the pNFS DS I see the > following on the NFS client side and the Ganesha Server side (The DS) > any ideas what the client did not like. > > Thanks, Marc. > > Jul 28 10:26:04 svl-marcrh-node-1 kernel: NFS: permission(0:51/7169), > mask=0x81, res=0 > > Jul 28 10:26:04 svl-marcrh-node-1 kernel: --> nfs41_call_sync_prepare > data->seq_server 00000000ff0f7422 > > Jul 28 10:26:04 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:04 svl-marcrh-node-1 kernel: NFS: > nfs_update_inode(0:51/245760 fh_crc=0x11f91d95 ct=1 info=0x427e7f) > > Jul 28 10:26:04 svl-marcrh-node-1 kernel: NFS: dentry_delete(/file8m, > 48084c) > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: NFS: permission(0:51/7169), > mask=0x81, res=0 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: NFS: > nfs_update_inode(0:51/245760 fh_crc=0x11f91d95 ct=3 info=0x427e7f) > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: --> nfs41_call_sync_prepare > data->seq_server 00000000ff0f7422 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: <-- _nfs4_proc_getdeviceinfo > status=0 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > stripe count 1 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > ds_num 1 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: RPC: Couldn't create > auth handle (flavor 390004) > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: --> nfs4_proc_create_session > clp=00000000bf42bf91 session=000000004af72bf8 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs4_init_channel_attrs: > Fore Channel : max_rqst_sz=1049620 max_resp_sz=1049480 max_ops=8 > max_reqs=64 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs4_init_channel_attrs: > Back Channel : max_rqst_sz=4096 max_resp_sz=4096 max_resp_sz_cached=0 > max_ops=2 max_reqs=16 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs4_proc_create_session > client>seqid 2 sessionid 4:1722187461:2:0 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: <-- > nfs41_proc_reclaim_complete status=0 > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: --> nfs4_read_done > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: --> nfs4_read_done > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: --> nfs4_read_done > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: --> nfs4_read_done > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > …. > > Jul 28 10:26:05 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:06 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:06 svl-marcrh-node-1 kernel: NFS: > nfs_update_inode(0:51/245760 fh_crc=0x11f91d95 ct=2 info=0x27040) > > Jul 28 10:26:06 svl-marcrh-node-1 kernel: nfs4_close_done: ret = 0 > > Jul 28 10:26:06 svl-marcrh-node-1 kernel: NFS: dentry_delete(/file8m, > 48084c) > > Jul 28 10:26:46 svl-marcrh-node-1 kernel: <-- > nfs41_proc_async_sequence status=0 > > Jul 28 10:26:46 svl-marcrh-node-1 kernel: nfs41_sequence_process: > Error 0 free the slot > > Jul 28 10:26:46 svl-marcrh-node-1 kernel: nfs41_sequence_call_done > rpc_cred 000000008d476879 > > 5.784003013 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= > -2, anon_gid= -2, , sys) > > 5.784031320 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :default options (options=03303002/ffffffff > root_squash , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, anon_gid= -2, expire= 60, none, sys) > > 5.784048104 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :Final options (options=023021e0/ffffffff > no_root_squash, RWrw, 34-, ---, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, anon_gid= -2, expire= 60, sys) > > 5.784183909 395973 TRACE_GANESHA: [svc_15] nfs_null :NFS3 :DEBUG > :REQUEST PROCESSING: Calling NFS_NULL > > 5.784958341 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= > -2, anon_gid= -2, , sys) > > 5.784981736 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :default options (options=03303002/ffffffff > root_squash , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, anon_gid= -2, expire= 60, none, sys) > > 5.785016198 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :Final options (options=023021e0/ffffffff > no_root_squash, RWrw, 34-, ---, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, anon_gid= -2, expire= 60, sys) > > 5.785062392 389339 TRACE_GANESHA: [svc_11] nfs4_Compound :NFS4 > :DEBUG :COMPOUND: There are 1 operations, res = 0x7f2a20001be0, tag = > NO TAG > > 5.785114712 389339 TRACE_GANESHA: [svc_11] nfs4_Compound :NFS4 > :F_DBG :COMPOUND: There are 1 operations nfs4 operations {EXCHANGE_ID} > > 5.785151123 389339 TRACE_GANESHA: [svc_11] process_one_op :NFS4 > :DEBUG :Request 0: opcode 42 is OP_EXCHANGE_ID > > 5.785183182 389339 TRACE_GANESHA: [svc_11] nfs4_op_exchange_id > :CLIENT ID :DEBUG :EXCHANGE_ID pnfs_flags 0x00010000 eia_flags 0x00040101 > > 5.785309869 389339 TRACE_GANESHA: [svc_11] inc_client_record_ref > :CLIENT ID :F_DBG :Increment refcount {0x7f2a200020a0 name=(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.785345676 389339 TRACE_GANESHA: [svc_11] inc_client_id_ref > :CLIENT ID :F_DBG :Increment refcount Clientid {0x7f2a200021a0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000001} UNCONFIRMED > Client={0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 reservations=0 > refcount=1} to 1 > > 5.785368820 389339 TRACE_GANESHA: [svc_11] dec_client_record_ref > :CLIENT ID :F_DBG :Decrement refcount now=1 {0x7f2a200020a0 > name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.785382625 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.785395515 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.785416713 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :DEBUG :Status of OP_EXCHANGE_ID in position 0 = NFS4_OK, op response > size is 0 total response size is 36 > > 5.785546764 389339 TRACE_GANESHA: [svc_11] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f2a20001e30 > (resarraylen=1) > > 5.786242209 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= > -2, anon_gid= -2, , sys) > > 5.786274623 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :default options (options=03303002/ffffffff > root_squash , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, anon_gid= -2, expire= 60, none, sys) > > 5.786292745 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :Final options (options=023021e0/ffffffff > no_root_squash, RWrw, 34-, ---, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, anon_gid= -2, expire= 60, sys) > > 5.786313843 395973 TRACE_GANESHA: [svc_15] nfs4_Compound :NFS4 > :DEBUG :COMPOUND: There are 1 operations, res = 0x7f29fc001310, tag = > NO TAG > > 5.786329329 395973 TRACE_GANESHA: [svc_15] nfs4_Compound :NFS4 > :F_DBG :COMPOUND: There are 1 operations nfs4 operations {EXCHANGE_ID} > > 5.786343154 395973 TRACE_GANESHA: [svc_15] process_one_op :NFS4 > :DEBUG :Request 0: opcode 42 is OP_EXCHANGE_ID > > 5.786377249 395973 TRACE_GANESHA: [svc_15] nfs4_op_exchange_id > :CLIENT ID :DEBUG :EXCHANGE_ID pnfs_flags 0x00010000 eia_flags 0x00040101 > > 5.786413732 395973 TRACE_GANESHA: [svc_15] dec_client_id_ref > :CLIENT ID :F_DBG :Decrement refcount Clientid {0x7f2a200021a0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000001} EXPIRED > Client={0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 reservations=0 > refcount=1} refcount to 0 > > 5.786426971 395973 TRACE_GANESHA: [svc_15] dec_client_id_ref > :CLIENT ID :F_DBG :Free Clientid refcount now=0 {0x7f2a200021a0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000001} EXPIRED > Client={0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 reservations=0 > refcount=1} > > 5.786455632 395973 TRACE_GANESHA: [svc_15] dec_client_record_ref > :CLIENT ID :F_DBG :Decrement refcount now=1 {0x7f2a200020a0 > name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.786478133 395973 TRACE_GANESHA: [svc_15] inc_client_record_ref > :CLIENT ID :F_DBG :Increment refcount {0x7f2a200020a0 name=(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.786496149 395973 TRACE_GANESHA: [svc_15] inc_client_id_ref > :CLIENT ID :F_DBG :Increment refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} UNCONFIRMED > Client={0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 reservations=0 > refcount=1} to 1 > > 5.786510720 395973 TRACE_GANESHA: [svc_15] dec_client_record_ref > :CLIENT ID :F_DBG :Decrement refcount now=1 {0x7f2a200020a0 > name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.786524531 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.786537391 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.786551547 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :DEBUG :Status of OP_EXCHANGE_ID in position 0 = NFS4_OK, op response > size is 0 total response size is 36 > > 5.786678902 395973 TRACE_GANESHA: [svc_15] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f29fc001560 > (resarraylen=1) > > 5.786833586 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= > -2, anon_gid= -2, , sys) > > 5.786891569 389339 TRACE_GANESHA: [svc_11] nfs4_Compound :NFS4 > :DEBUG :COMPOUND: There are 1 operations, res = 0x7f2a20002ae0, tag = > NO TAG > > 5.786907017 389339 TRACE_GANESHA: [svc_11] nfs4_Compound :NFS4 > :F_DBG :COMPOUND: There are 1 operations nfs4 operations {CREATE_SESSION} > > 5.786938593 389339 TRACE_GANESHA: [svc_11] process_one_op :NFS4 > :DEBUG :Request 0: opcode 43 is OP_CREATE_SESSION > > 5.786996781 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :DEBUG :CREATE_SESSION client addr=::ffff:10.11.56.193 > clientid=Epoch=0x66a7e0c6 Counter=0x00000002 ------------------- > > 5.787030153 389339 TRACE_GANESHA: [svc_11] inc_client_id_ref > :CLIENT ID :F_DBG :Increment refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} UNCONFIRMED > Client={0x7f2a200020a0 nam > > e=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} > t_delta=0 reservations=0 refcount=2} to 2 > > 5.787046306 389339 TRACE_GANESHA: [svc_11] inc_client_record_ref > :CLIENT ID :F_DBG :Increment refcount {0x7f2a200020a0 name=(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.787061232 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :F_DBG :Client Record 0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=2 cr_confirmed_rec=(nil) cr_unco > > nfirmed_rec=0x7f29fc0026d0 > > 5.787074437 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :DEBUG :CREATE_SESSION clientid=Epoch=0x66a7e0c6 > Counter=0x00000002 csa_sequence=1 clientid_cs_seq=1 data_oppos=0 > > 5.787091144 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :F_DBG :Found 0x7f29fc0026d0 ClientID={Epoch=0x66a7e0c6 > Counter=0x00000002} UNCONFIRMED Client={0x7f2a200020a0 name=(44:Linux > NFSv4.1 > > svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 reservations=0 > refcount=2 > > 5.787115990 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :F_DBG :Fore Channel attributes ca_headerpadsize 0 > ca_maxrequestsize 1049620 ca_maxresponsesize 1049480 > ca_maxresponsesize_cached 758 > > 4 ca_maxoperations 8 ca_maxrequests 64 > > 5.787129437 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :F_DBG :Back Channel attributes ca_headerpadsize 0 > ca_maxrequestsize 4096 ca_maxresponsesize 4096 > ca_maxresponsesize_cached 0 ca_maxo > > perations 2 ca_maxrequests 16 > > 5.787211408 389339 TRACE_GANESHA: [svc_11] inc_client_id_ref > :CLIENT ID :F_DBG :Increment refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} UNCONFIRMED > Client={0x7f2a200020a0 nam > > e=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > t_delta=0 reservations=0 refcount=3} to 3 > > 5.787250959 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :F_DBG :Confirming new 0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} UNCONFIRMED > Client={0x7f2a200020a0 name=(44:Linu > > x NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 > reservations=0 refcount=3 > > 5.787313756 389339 TRACE_GANESHA: [svc_11] fs_create_clid_name > :CLIENT ID :DEBUG :Created client name [::ffff:10.11.56.193-(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com)] > > 5.787386559 389339 TRACE_GANESHA: [svc_11] fs_add_clid :CLIENT ID > :DEBUG :Created client dir > [/var/lib/nfs/ganesha/v4recov/node0/::ffff:10.11.56.193-(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com)] > > 5.787413463 389339 TRACE_GANESHA: [svc_11] nfs4_chk_clid_impl > :CLIENT ID :DEBUG :chk for 7397128053987475458 > > 5.787441005 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :DEBUG :Confirmed 0x7f29fc0026d0 ClientID={Epoch=0x66a7e0c6 > Counter=0x00000002} CONFIRMED Client={0x7f2a200020a0 name=(44:Linux NFSv4 > > .1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 > reservations=0 refcount=3 > > 5.787457512 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :F_DBG :Client Record 0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=2 cr_confirmed_rec=0x7f29fc0026d > > 0 cr_unconfirmed_rec=(nil) > > 5.787526325 389339 TRACE_GANESHA: [svc_11] nfs4_op_create_session > :SESSIONS :DEBUG :success session 0x7f2a20002f40 > {sessionid=(16:0x02000000c6e0a7660100000000000000)} csa_flags 0x3 > csr_flags 0x2 > > 5.787554916 389339 TRACE_GANESHA: [svc_11] dec_client_id_ref > :CLIENT ID :F_DBG :Decrement refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} CONFIRMED > Client={0x7f2a200020a0 name= > > (44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > t_delta=0 reservations=0 refcount=3} refcount to 2 > > 5.787571861 389339 TRACE_GANESHA: [svc_11] dec_client_record_ref > :CLIENT ID :F_DBG :Decrement refcount now=1 {0x7f2a200020a0 > name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.787585600 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.787598472 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.787612249 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :DEBUG :Status of OP_CREATE_SESSION in position 0 = NFS4_OK, op > response size is 112 total response size is 148 > > 5.787721136 389339 TRACE_GANESHA: [svc_11] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f2a20002590 > (resarraylen=1) > > 5.787906140 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= -2, a > > non_gid= -2, , sys) > > 5.787946036 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :default options (options=03303002/ffffffff > root_squash , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, a > > non_gid= -2, expire= 60, none, sys) > > 5.787975742 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :Final options (options=023021e0/ffffffff > no_root_squash, RWrw, 34-, ---, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, a > > non_gid= -2, expire= 60, sys) > > 5.788052517 395973 TRACE_GANESHA: [svc_15] nfs4_Compound :NFS4 > :DEBUG :COMPOUND: There are 2 operations, res = 0x7f29fc002070, tag = > NO TAG > > 5.788087408 395973 TRACE_GANESHA: [svc_15] nfs4_Compound :NFS4 > :F_DBG :COMPOUND: There are 2 operations nfs4 operations {SEQUENCE, > RECLAIM_COMPLETE} > > 5.788117846 395973 TRACE_GANESHA: [svc_15] process_one_op :NFS4 > :DEBUG :Request 0: opcode 53 is OP_SEQUENCE > > 5.788165215 395973 TRACE_GANESHA: [svc_15] > nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Get Session > sessionid=(16:0x02000000c6e0a7660100000000000000) > > 5.788187319 395973 TRACE_GANESHA: [svc_15] > nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Session > sessionid=(16:0x02000000c6e0a7660100000000000000) Found > > 5.788242777 395973 TRACE_GANESHA: [svc_15] nfs4_op_sequence > :SESSIONS :DEBUG :SEQUENCE session=0x7f2a20002f40 > > 5.788283602 395973 TRACE_GANESHA: [svc_15] > reserve_lease_or_expire :CLIENT ID :F_DBG :Reserve Lease > 0x7f29fc0026d0 ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} > CONFIRMED Client={0x7f2a200020a0 name=(44:Linux > > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} t_delta=0 > reservations=1 refcount=2 (Valid=YES 60 seconds left > > 5.788305835 395973 TRACE_GANESHA: [svc_15] nfs4_op_sequence > :SESSIONS :F_DBG :Don't use session slot 0=0x7f2a20003900 for DRC > > 5.788320442 395973 TRACE_GANESHA: [svc_15] check_resp_room :NFS4 > :F_DBG :Status of OP_SEQUENCE in position 0 is ok so far, op response > size = 40 total response size would be = 84 out of max 1049480/7584 > > 5.788352724 395973 TRACE_GANESHA: [svc_15] check_session_conn > :SESSIONS :F_DBG :Comparing addr ::ffff:10.11.56.193:996 for > OP_SEQUENCE to Session bound addr ::ffff:10.11.56.193:996 > > 5.788372492 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.788386706 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.788400303 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :DEBUG :Status of OP_SEQUENCE in position 0 = NFS4_OK, op response > size is 40 total response size is 76 > > 5.788415252 395973 TRACE_GANESHA: [svc_15] process_one_op :NFS4 > :DEBUG :Request 1: opcode 58 is OP_RECLAIM_COMPLETE > > 5.788429355 395973 TRACE_GANESHA: [svc_15] check_resp_room :NFS4 > :F_DBG :Status of OP_RECLAIM_COMPLETE in position 1 is ok so far, op > response size = 4 total response size would be = 92 out of max > 1049480/7584 > > 5.788465437 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.788479686 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.788492986 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :DEBUG :Status of OP_RECLAIM_COMPLETE in position 1 = NFS4_OK, op > response size is 4 total response size is 84 > > 5.788512857 395973 TRACE_GANESHA: [svc_15] update_lease :CLIENT > ID :F_DBG :Update Lease 0x7f29fc0026d0 ClientID={Epoch=0x66a7e0c6 > Counter=0x00000002} CONFIRMED Client={0x7f2a200020a0 name=(44:Linux > NFSv4.1 svl > > -marcrh-node-1.fyre.ibm.com) refcount=1} t_delta=0 reservations=0 > refcount=2 > > 5.788610239 395973 TRACE_GANESHA: [svc_15] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f29fc0022c0 > (resarraylen=2) > > 5.789131433 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= -2, a > > non_gid= -2, , sys) > > 5.789151102 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :default options (options=03303002/ffffffff > root_squash , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, a > > non_gid= -2, expire= 60, none, sys) > > 5.789179306 389339 TRACE_GANESHA: [svc_11] export_check_access > :EXPORT :M_DBG :Final options (options=023021e0/ffffffff > no_root_squash, RWrw, 34-, ---, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, a > > non_gid= -2, expire= 60, sys) > > 5.789243406 389339 TRACE_GANESHA: [svc_11] nfs4_Compound :NFS4 > :DEBUG :COMPOUND: There are 1 operations, res = 0x7f2a20001440, tag = > NO TAG > > 5.789270968 389339 TRACE_GANESHA: [svc_11] nfs4_Compound :NFS4 > :F_DBG :COMPOUND: There are 1 operations nfs4 operations {DESTROY_SESSION} > > 5.789292769 389339 TRACE_GANESHA: [svc_11] process_one_op :NFS4 > :DEBUG :Request 0: opcode 44 is OP_DESTROY_SESSION > > 5.789324008 389339 TRACE_GANESHA: [svc_11] > nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Get Session > sessionid=(16:0x02000000c6e0a7660100000000000000) > > 5.789347635 389339 TRACE_GANESHA: [svc_11] > nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Session > sessionid=(16:0x02000000c6e0a7660100000000000000) Found > > 5.789374279 389339 TRACE_GANESHA: [svc_11] check_session_conn > :SESSIONS :F_DBG :Comparing addr ::ffff:10.11.56.193:996 for > OP_DESTROY_SESSION to Session bound addr ::ffff:10.11.56.193:996 > > 5.789396299 389339 TRACE_GANESHA: [svc_11] dec_client_id_ref > :CLIENT ID :F_DBG :Decrement refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} CONFIRMED > Client={0x7f2a200020a0 name= > > (44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} > t_delta=0 reservations=0 refcount=2} refcount to 1 > > 5.789410060 389339 TRACE_GANESHA: [svc_11] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f29fc0025b0 > (resarraylen=2) > > 5.789450150 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.789463993 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.789463993 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.789477602 389339 TRACE_GANESHA: [svc_11] complete_op :NFS4 > :DEBUG :Status of OP_DESTROY_SESSION in position 0 = NFS4_OK, op > response size is 4 total response size is 40 > > 5.789564757 389339 TRACE_GANESHA: [svc_11] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f2a20001690 > (resarraylen=1) > > 5.789753915 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :EXPORT_DEFAULTS (options=020021e0/0701f1ff > no_root_squash, RWrw, , ---, TCP, ----, , , anon_uid= -2, a > > non_gid= -2, , sys) > > 5.789787721 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :default options (options=03303002/ffffffff > root_squash , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, a > > non_gid= -2, expire= 60, none, sys) > > 5.789818999 395973 TRACE_GANESHA: [svc_15] export_check_access > :EXPORT :M_DBG :Final options (options=023021e0/ffffffff > no_root_squash, RWrw, 34-, ---, TCP, ----, No Manage_Gids, -- Deleg, > anon_uid= -2, a > > non_gid= -2, expire= 60, sys) > > 5.789840590 395973 TRACE_GANESHA: [svc_15] nfs4_Compound :NFS4 > :DEBUG :COMPOUND: There are 1 operations, res = 0x7f29fc003150, tag = > NO TAG > > 5.789855216 395973 TRACE_GANESHA: [svc_15] nfs4_Compound :NFS4 > :F_DBG :COMPOUND: There are 1 operations nfs4 operations > {DESTROY_CLIENTID} > > 5.789874099 395973 TRACE_GANESHA: [svc_15] process_one_op :NFS4 > :DEBUG :Request 0: opcode 57 is OP_DESTROY_CLIENTID > > 5.789910130 395973 TRACE_GANESHA: [svc_15] > nfs4_op_destroy_clientid :CLIENT ID :DEBUG :DESTROY_CLIENTID > clientid=Epoch=0x66a7e0c6 Counter=0x00000002 > > 5.789930494 395973 TRACE_GANESHA: [svc_15] inc_client_id_ref > :CLIENT ID :F_DBG :Increment refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} CONFIRMED > Client={0x7f2a200020a0 name= > > (44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} > t_delta=0 reservations=0 refcount=2} to 2 > > 5.789945301 395973 TRACE_GANESHA: [svc_15] inc_client_record_ref > :CLIENT ID :F_DBG :Increment refcount {0x7f2a200020a0 name=(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.789960256 395973 TRACE_GANESHA: [svc_15] > nfs4_op_destroy_clientid :CLIENT ID :F_DBG :Client Record > 0x7f2a200020a0 name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) > refcount=2 cr_confirmed_rec=0x7f29fc00 > > 26d0 cr_unconfirmed_rec=(nil) > > 5.789978554 395973 TRACE_GANESHA: [svc_15] > nfs4_op_destroy_clientid :CLIENT ID :DEBUG :Removing confirmed > clientid 0x7f29fc0026d0 ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} > CONFIRMED Client={0x7f2a200020a0 > > name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > t_delta=0 reservations=0 refcount=2 > > 5.790117503 395973 TRACE_GANESHA: [svc_15] fs_rm_clid_impl > :CLIENT ID :DEBUG :Removed client dir > (/var/lib/nfs/ganesha/v4recov/node0/::ffff:10.11.56.193-(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com)) > > 5.790141122 395973 TRACE_GANESHA: [svc_15] dec_client_id_ref > :CLIENT ID :F_DBG :Decrement refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} EXPIRED > Client={0x7f2a200020a0 name=(4 > > 4:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} t_delta=0 > reservations=0 refcount=2} refcount to 1 > > 5.790156176 395973 TRACE_GANESHA: [svc_15] dec_client_record_ref > :CLIENT ID :F_DBG :Decrement refcount now=1 {0x7f2a200020a0 > name=(44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=2} > > 5.790172109 395973 TRACE_GANESHA: [svc_15] dec_client_id_ref > :CLIENT ID :F_DBG :Decrement refcount Clientid {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} EXPIRED > Client={0x7f2a200020a0 name=(4 > > 4:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} t_delta=0 > reservations=0 refcount=1} refcount to 0 > > 5.790185156 395973 TRACE_GANESHA: [svc_15] dec_client_id_ref > :CLIENT ID :F_DBG :Free Clientid refcount now=0 {0x7f29fc0026d0 > ClientID={Epoch=0x66a7e0c6 Counter=0x00000002} EXPIRED > Client={0x7f2a200020a0 name=( > > 44:Linux NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} t_delta=0 > reservations=0 refcount=1} > > 5.790200509 395973 TRACE_GANESHA: [svc_15] dec_client_record_ref > :CLIENT ID :F_DBG :Try to remove {0x7f2a200020a0 name=(44:Linux > NFSv4.1 svl-marcrh-node-1.fyre.ibm.com) refcount=1} > > 5.790215662 395973 TRACE_GANESHA: [svc_15] dec_client_record_ref > :CLIENT ID :F_DBG :Free {0x7f2a200020a0 name=(44:Linux NFSv4.1 > svl-marcrh-node-1.fyre.ibm.com) refcount=1} > > 5.790231007 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Current FH Len=0 (EMPTY) > > 5.790244094 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :F_DBG :Saved FH Len=0 (EMPTY) > > 5.790262649 395973 TRACE_GANESHA: [svc_15] complete_op :NFS4 > :DEBUG :Status of OP_DESTROY_CLIENTID in position 0 = NFS4_OK, op > response size is 4 total response size is 40 > > 5.790341620 395973 TRACE_GANESHA: [svc_15] > release_nfs4_res_compound :NFS4 :F_DBG :Compound Free 0x7f29fc0033a0 > (resarraylen=1) > ^ permalink raw reply [flat|nested] 17+ messages in thread
* NFS client to pNFS DS 2024-07-29 21:56 ` NFS client failure marc eshel @ 2024-08-05 21:51 ` marc eshel 2024-08-08 14:22 ` Anna Schumaker 2024-08-08 22:07 ` Olga Kornievskaia 0 siblings, 2 replies; 17+ messages in thread From: marc eshel @ 2024-08-05 21:51 UTC (permalink / raw) To: linux-nfs@vger.kernel.org, Trond Myklebust Hi Trond, Will the Linux NFS client try to us krb5i regardless of the MDS configuration? Is there any option to avoid it? Thanks, Marc. ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node stripe count 1 Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node ds_num 1 Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create auth handle (flavor 390004) ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-05 21:51 ` NFS client to pNFS DS marc eshel @ 2024-08-08 14:22 ` Anna Schumaker 2024-08-08 22:07 ` Olga Kornievskaia 1 sibling, 0 replies; 17+ messages in thread From: Anna Schumaker @ 2024-08-08 14:22 UTC (permalink / raw) To: marc eshel; +Cc: linux-nfs@vger.kernel.org, Trond Myklebust Hi Marc, On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: > > Hi Trond, > > Will the Linux NFS client try to us krb5i regardless of the MDS > configuration? My expectation is that the NFS client will use SECINFO_NO_NAME to find the preferred security flavors of the server. If krb5i is first on the list that the server returns then that's what the client will use. > > Is there any option to avoid it? Yes, you can use the '-o sec=$FLAVOR' mount option to mount with other security flavors. Valid flavors are 'none', 'sys', 'krb5', 'krb5i', and 'krb5p'. I hope this helps! Anna > > Thanks, Marc. > > ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > stripe count 1 > Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > ds_num 1 > Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > auth handle (flavor 390004) > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-05 21:51 ` NFS client to pNFS DS marc eshel 2024-08-08 14:22 ` Anna Schumaker @ 2024-08-08 22:07 ` Olga Kornievskaia 2024-08-09 13:06 ` Anna Schumaker 1 sibling, 1 reply; 17+ messages in thread From: Olga Kornievskaia @ 2024-08-08 22:07 UTC (permalink / raw) To: marc eshel; +Cc: linux-nfs@vger.kernel.org, Trond Myklebust On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: > > Hi Trond, > > Will the Linux NFS client try to us krb5i regardless of the MDS > configuration? > > Is there any option to avoid it? I was under the impression the linux client has no way of choosing a different auth_gss security flavor for the DS than the MDS. Meaning that if mount command has say sec=krb5i then both MDS and DS connections have to do krb5i and if say the DS isn't configured for Kerberos, then IO would fallback to MDS. I no longer have a pnfs server to verify whether or not what I say is true but that is what my memory tells me is the case. > > Thanks, Marc. > > ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > stripe count 1 > Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > ds_num 1 > Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > auth handle (flavor 390004) > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-08 22:07 ` Olga Kornievskaia @ 2024-08-09 13:06 ` Anna Schumaker 2024-08-09 14:29 ` marc eshel [not found] ` <8ab0fd49-0c90-42bd-a34e-9dcf63a99bd5@gmail.com> 0 siblings, 2 replies; 17+ messages in thread From: Anna Schumaker @ 2024-08-09 13:06 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: marc eshel, linux-nfs@vger.kernel.org, Trond Myklebust On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: > > On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: > > > > Hi Trond, > > > > Will the Linux NFS client try to us krb5i regardless of the MDS > > configuration? > > > > Is there any option to avoid it? > > I was under the impression the linux client has no way of choosing a > different auth_gss security flavor for the DS than the MDS. Meaning That's a good point, I completely missed that this is specifically for the DS. > that if mount command has say sec=krb5i then both MDS and DS > connections have to do krb5i and if say the DS isn't configured for > Kerberos, then IO would fallback to MDS. I no longer have a pnfs That's what I would expect, too. > server to verify whether or not what I say is true but that is what my > memory tells me is the case. > > > > > > Thanks, Marc. > > > > ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > > stripe count 1 > > Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > > ds_num 1 > > Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > > auth handle (flavor 390004) > > > > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-09 13:06 ` Anna Schumaker @ 2024-08-09 14:29 ` marc eshel [not found] ` <8ab0fd49-0c90-42bd-a34e-9dcf63a99bd5@gmail.com> 1 sibling, 0 replies; 17+ messages in thread From: marc eshel @ 2024-08-09 14:29 UTC (permalink / raw) To: Anna Schumaker, Olga Kornievskaia Cc: linux-nfs@vger.kernel.org, Trond Myklebust Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS. Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i. Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create auth handle (flavor 390004) Marc. On 8/9/24 6:06 AM, Anna Schumaker wrote: > On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: >> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: >>> Hi Trond, >>> >>> Will the Linux NFS client try to us krb5i regardless of the MDS >>> configuration? >>> >>> Is there any option to avoid it? >> I was under the impression the linux client has no way of choosing a >> different auth_gss security flavor for the DS than the MDS. Meaning > That's a good point, I completely missed that this is specifically for the DS. > >> that if mount command has say sec=krb5i then both MDS and DS >> connections have to do krb5i and if say the DS isn't configured for >> Kerberos, then IO would fallback to MDS. I no longer have a pnfs > That's what I would expect, too. > >> server to verify whether or not what I say is true but that is what my >> memory tells me is the case. >> >> >>> Thanks, Marc. >>> >>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>> stripe count 1 >>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>> ds_num 1 >>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>> auth handle (flavor 390004) >>> >>> ^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <8ab0fd49-0c90-42bd-a34e-9dcf63a99bd5@gmail.com>]
* Re: NFS client to pNFS DS [not found] ` <8ab0fd49-0c90-42bd-a34e-9dcf63a99bd5@gmail.com> @ 2024-08-09 15:15 ` Olga Kornievskaia 2024-08-09 16:26 ` marc eshel 0 siblings, 1 reply; 17+ messages in thread From: Olga Kornievskaia @ 2024-08-09 15:15 UTC (permalink / raw) To: marc eshel; +Cc: Anna Schumaker, linux-nfs@vger.kernel.org, Trond Myklebust On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: > > Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS. > > Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i. For 4.1, the client will always try to do state operations with krb5i even when sec=sys when the client detects that it's configured to do Kerberos (ie., gssd is running). This context creation is triggered regardless of whether the rpc client is used for MDS or DS. My question to you: is the MDS configured with Kerberos but the DS isn't? And also, does this lead to a failure? > Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > auth handle (flavor 390004) > > Marc. > > On 8/9/24 6:06 AM, Anna Schumaker wrote: > > On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: > > On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: > > Hi Trond, > > Will the Linux NFS client try to us krb5i regardless of the MDS > configuration? > > Is there any option to avoid it? > > I was under the impression the linux client has no way of choosing a > different auth_gss security flavor for the DS than the MDS. Meaning > > That's a good point, I completely missed that this is specifically for the DS. > > that if mount command has say sec=krb5i then both MDS and DS > connections have to do krb5i and if say the DS isn't configured for > Kerberos, then IO would fallback to MDS. I no longer have a pnfs > > That's what I would expect, too. > > server to verify whether or not what I say is true but that is what my > memory tells me is the case. > > > Thanks, Marc. > > ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > stripe count 1 > Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > ds_num 1 > Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > auth handle (flavor 390004) > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-09 15:15 ` Olga Kornievskaia @ 2024-08-09 16:26 ` marc eshel 2024-08-10 21:20 ` Olga Kornievskaia 0 siblings, 1 reply; 17+ messages in thread From: marc eshel @ 2024-08-09 16:26 UTC (permalink / raw) To: Olga Kornievskaia Cc: Anna Schumaker, linux-nfs@vger.kernel.org, Trond Myklebust On 8/9/24 8:15 AM, Olga Kornievskaia wrote: > On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: >> Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS. >> >> Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i. > For 4.1, the client will always try to do state operations with krb5i > even when sec=sys when the client detects that it's configured to do > Kerberos (ie., gssd is running). This context creation is triggered > regardless of whether the rpc client is used for MDS or DS. > > My question to you: is the MDS configured with Kerberos but the DS > isn't? And also, does this lead to a failure? Both MDS DS are configured for sec=sys and it is leading to client switching from DS to MDS so yes, it is pNFS failure. What I see on the DS is the client creating a session and than imminently destroying it before committing it. If the is something else that I can debug I will be happy to. Marc. > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >> auth handle (flavor 390004) >> >> Marc. >> >> On 8/9/24 6:06 AM, Anna Schumaker wrote: >> >> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: >> >> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: >> >> Hi Trond, >> >> Will the Linux NFS client try to us krb5i regardless of the MDS >> configuration? >> >> Is there any option to avoid it? >> >> I was under the impression the linux client has no way of choosing a >> different auth_gss security flavor for the DS than the MDS. Meaning >> >> That's a good point, I completely missed that this is specifically for the DS. >> >> that if mount command has say sec=krb5i then both MDS and DS >> connections have to do krb5i and if say the DS isn't configured for >> Kerberos, then IO would fallback to MDS. I no longer have a pnfs >> >> That's what I would expect, too. >> >> server to verify whether or not what I say is true but that is what my >> memory tells me is the case. >> >> >> Thanks, Marc. >> >> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >> stripe count 1 >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >> ds_num 1 >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >> auth handle (flavor 390004) >> >> ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-09 16:26 ` marc eshel @ 2024-08-10 21:20 ` Olga Kornievskaia 2024-08-11 19:41 ` Mkrtchyan, Tigran 0 siblings, 1 reply; 17+ messages in thread From: Olga Kornievskaia @ 2024-08-10 21:20 UTC (permalink / raw) To: marc eshel; +Cc: Anna Schumaker, linux-nfs@vger.kernel.org, Trond Myklebust On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote: > > > On 8/9/24 8:15 AM, Olga Kornievskaia wrote: > > On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: > >> Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS. > >> > >> Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i. > > For 4.1, the client will always try to do state operations with krb5i > > even when sec=sys when the client detects that it's configured to do > > Kerberos (ie., gssd is running). This context creation is triggered > > regardless of whether the rpc client is used for MDS or DS. > > > > My question to you: is the MDS configured with Kerberos but the DS > > isn't? And also, does this lead to a failure? > Both MDS DS are configured for sec=sys and it is leading to client > switching from DS to MDS so yes, it is pNFS failure. What I see on the > DS is the client creating a session and than imminently destroying it > before committing it. If the is something else that I can debug I will > be happy to. pnfs failure is unexpected. I'm pretty confident that a non-kerberos configured client can do normal pnfs with sec=sys. I can help debug, if you want to send me a network trace and tracepoint output. Btw what kernel/distro are you using? > > Marc. > > > > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > >> auth handle (flavor 390004) > >> > >> Marc. > >> > >> On 8/9/24 6:06 AM, Anna Schumaker wrote: > >> > >> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: > >> > >> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: > >> > >> Hi Trond, > >> > >> Will the Linux NFS client try to us krb5i regardless of the MDS > >> configuration? > >> > >> Is there any option to avoid it? > >> > >> I was under the impression the linux client has no way of choosing a > >> different auth_gss security flavor for the DS than the MDS. Meaning > >> > >> That's a good point, I completely missed that this is specifically for the DS. > >> > >> that if mount command has say sec=krb5i then both MDS and DS > >> connections have to do krb5i and if say the DS isn't configured for > >> Kerberos, then IO would fallback to MDS. I no longer have a pnfs > >> > >> That's what I would expect, too. > >> > >> server to verify whether or not what I say is true but that is what my > >> memory tells me is the case. > >> > >> > >> Thanks, Marc. > >> > >> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > >> stripe count 1 > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > >> ds_num 1 > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > >> auth handle (flavor 390004) > >> > >> > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-10 21:20 ` Olga Kornievskaia @ 2024-08-11 19:41 ` Mkrtchyan, Tigran 2024-08-11 19:50 ` marc eshel 2024-09-09 23:23 ` marc eshel 0 siblings, 2 replies; 17+ messages in thread From: Mkrtchyan, Tigran @ 2024-08-11 19:41 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: marc eshel, Anna Schumaker, linux-nfs, Trond Myklebust > > On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote: > > On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote: > > > > > > On 8/9/24 8:15 AM, Olga Kornievskaia wrote: > > > On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: > > >> Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS. > > >> > > >> Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i. > > > For 4.1, the client will always try to do state operations with krb5i > > > even when sec=sys when the client detects that it's configured to do > > > Kerberos (ie., gssd is running). This context creation is triggered > > > regardless of whether the rpc client is used for MDS or DS. > > > > > > My question to you: is the MDS configured with Kerberos but the DS > > > isn't? And also, does this lead to a failure? > > Both MDS DS are configured for sec=sys and it is leading to client > > switching from DS to MDS so yes, it is pNFS failure. What I see on the > > DS is the client creating a session and than imminently destroying it > > before committing it. If the is something else that I can debug I will > > be happy to. > > pnfs failure is unexpected. I'm pretty confident that a non-kerberos > configured client can do normal pnfs with sec=sys. I can help debug, Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus works as expected. We mount always with sec=sys, and despite the fact, that hosts configured to support kerberos due to AFS and GPFS, the access to DSes is never an issue and never tries to use kerberos. Tigran. > if you want to send me a network trace and tracepoint output. Btw what > kernel/distro are you using? > > > > > Marc. > > > > > > > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > > >> auth handle (flavor 390004) > > >> > > >> Marc. > > >> > > >> On 8/9/24 6:06 AM, Anna Schumaker wrote: > > >> > > >> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: > > >> > > >> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: > > >> > > >> Hi Trond, > > >> > > >> Will the Linux NFS client try to us krb5i regardless of the MDS > > >> configuration? > > >> > > >> Is there any option to avoid it? > > >> > > >> I was under the impression the linux client has no way of choosing a > > >> different auth_gss security flavor for the DS than the MDS. Meaning > > >> > > >> That's a good point, I completely missed that this is specifically for the DS. > > >> > > >> that if mount command has say sec=krb5i then both MDS and DS > > >> connections have to do krb5i and if say the DS isn't configured for > > >> Kerberos, then IO would fallback to MDS. I no longer have a pnfs > > >> > > >> That's what I would expect, too. > > >> > > >> server to verify whether or not what I say is true but that is what my > > >> memory tells me is the case. > > >> > > >> > > >> Thanks, Marc. > > >> > > >> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > > >> stripe count 1 > > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node > > >> ds_num 1 > > >> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create > > >> auth handle (flavor 390004) > > >> > > >> > > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-11 19:41 ` Mkrtchyan, Tigran @ 2024-08-11 19:50 ` marc eshel 2024-08-12 6:52 ` Mkrtchyan, Tigran 2024-09-09 23:23 ` marc eshel 1 sibling, 1 reply; 17+ messages in thread From: marc eshel @ 2024-08-11 19:50 UTC (permalink / raw) To: Mkrtchyan, Tigran, Olga Kornievskaia Cc: Anna Schumaker, linux-nfs, Trond Myklebust On 8/11/24 12:41 PM, Mkrtchyan, Tigran wrote: >> On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote: >> >> On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote: >>> >>> On 8/9/24 8:15 AM, Olga Kornievskaia wrote: >>>> On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: >>>>> Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS. >>>>> >>>>> Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i. >>>> For 4.1, the client will always try to do state operations with krb5i >>>> even when sec=sys when the client detects that it's configured to do >>>> Kerberos (ie., gssd is running). This context creation is triggered >>>> regardless of whether the rpc client is used for MDS or DS. >>>> >>>> My question to you: is the MDS configured with Kerberos but the DS >>>> isn't? And also, does this lead to a failure? >>> Both MDS DS are configured for sec=sys and it is leading to client >>> switching from DS to MDS so yes, it is pNFS failure. What I see on the >>> DS is the client creating a session and than imminently destroying it >>> before committing it. If the is something else that I can debug I will >>> be happy to. >> pnfs failure is unexpected. I'm pretty confident that a non-kerberos >> configured client can do normal pnfs with sec=sys. I can help debug, > Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus works as expected. We mount always with sec=sys, and despite the fact, that hosts configured to support kerberos due to AFS and GPFS, the access to DSes is never an issue and never tries to use kerberos. > > Tigran. Hi Tigran, It is possible that it is failing back to MDS for another reason and the error message that I see is not the reason for the failure. Are you using pNFS with Ganesha and GPFS? Marc. > >> if you want to send me a network trace and tracepoint output. Btw what >> kernel/distro are you using? >> >>> Marc. >>> >>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>>>> auth handle (flavor 390004) >>>>> >>>>> Marc. >>>>> >>>>> On 8/9/24 6:06 AM, Anna Schumaker wrote: >>>>> >>>>> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: >>>>> >>>>> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: >>>>> >>>>> Hi Trond, >>>>> >>>>> Will the Linux NFS client try to us krb5i regardless of the MDS >>>>> configuration? >>>>> >>>>> Is there any option to avoid it? >>>>> >>>>> I was under the impression the linux client has no way of choosing a >>>>> different auth_gss security flavor for the DS than the MDS. Meaning >>>>> >>>>> That's a good point, I completely missed that this is specifically for the DS. >>>>> >>>>> that if mount command has say sec=krb5i then both MDS and DS >>>>> connections have to do krb5i and if say the DS isn't configured for >>>>> Kerberos, then IO would fallback to MDS. I no longer have a pnfs >>>>> >>>>> That's what I would expect, too. >>>>> >>>>> server to verify whether or not what I say is true but that is what my >>>>> memory tells me is the case. >>>>> >>>>> >>>>> Thanks, Marc. >>>>> >>>>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>>>> stripe count 1 >>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>>>> ds_num 1 >>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>>>> auth handle (flavor 390004) >>>>> >>>>> ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-11 19:50 ` marc eshel @ 2024-08-12 6:52 ` Mkrtchyan, Tigran 2024-08-12 14:37 ` marc eshel 0 siblings, 1 reply; 17+ messages in thread From: Mkrtchyan, Tigran @ 2024-08-12 6:52 UTC (permalink / raw) To: marc eshel; +Cc: Olga Kornievskaia, schumaker anna, linux-nfs, Trond Myklebust [-- Attachment #1: Type: text/plain, Size: 4358 bytes --] ----- Original Message ----- > From: "marc eshel" <eshel.marc@gmail.com> > To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Olga Kornievskaia" <aglo@umich.edu> > Cc: "schumaker anna" <schumaker.anna@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Trond Myklebust" > <trondmy@hammerspace.com> > Sent: Sunday, 11 August, 2024 21:50:36 > Subject: Re: NFS client to pNFS DS > On 8/11/24 12:41 PM, Mkrtchyan, Tigran wrote: >>> On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote: >>> >>> On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote: >>>> >>>> On 8/9/24 8:15 AM, Olga Kornievskaia wrote: >>>>> On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: >>>>>> Thanks for the replies, I am a little rusty with debugging NFS but this what I >>>>>> see when the NFS client tried to create a session with the DS. >>>>>> >>>>>> Ganesha was configured for sec=sys and the client mount had the option sec=sys, >>>>>> I assume flavor 390004 means it was trying to use krb5i. >>>>> For 4.1, the client will always try to do state operations with krb5i >>>>> even when sec=sys when the client detects that it's configured to do >>>>> Kerberos (ie., gssd is running). This context creation is triggered >>>>> regardless of whether the rpc client is used for MDS or DS. >>>>> >>>>> My question to you: is the MDS configured with Kerberos but the DS >>>>> isn't? And also, does this lead to a failure? >>>> Both MDS DS are configured for sec=sys and it is leading to client >>>> switching from DS to MDS so yes, it is pNFS failure. What I see on the >>>> DS is the client creating a session and than imminently destroying it >>>> before committing it. If the is something else that I can debug I will >>>> be happy to. >>> pnfs failure is unexpected. I'm pretty confident that a non-kerberos >>> configured client can do normal pnfs with sec=sys. I can help debug, >> Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus >> works as expected. We mount always with sec=sys, and despite the fact, that >> hosts configured to support kerberos due to AFS and GPFS, the access to DSes is >> never an issue and never tries to use kerberos. >> >> Tigran. > > Hi Tigran, > > It is possible that it is failing back to MDS for another reason and the > error message that I see is not the reason for the failure. Are you > using pNFS with Ganesha and GPFS? > > Marc. Hi Marc, pNFS is used exclusively with dCache. Can you provide the packet capture of failing pNFS traffic, like mount + cp + umount? Best regards, Tigran. > >> >>> if you want to send me a network trace and tracepoint output. Btw what >>> kernel/distro are you using? >>> >>>> Marc. >>>> >>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>>>>> auth handle (flavor 390004) >>>>>> >>>>>> Marc. >>>>>> >>>>>> On 8/9/24 6:06 AM, Anna Schumaker wrote: >>>>>> >>>>>> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>> >>>>>> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: >>>>>> >>>>>> Hi Trond, >>>>>> >>>>>> Will the Linux NFS client try to us krb5i regardless of the MDS >>>>>> configuration? >>>>>> >>>>>> Is there any option to avoid it? >>>>>> >>>>>> I was under the impression the linux client has no way of choosing a >>>>>> different auth_gss security flavor for the DS than the MDS. Meaning >>>>>> >>>>>> That's a good point, I completely missed that this is specifically for the DS. >>>>>> >>>>>> that if mount command has say sec=krb5i then both MDS and DS >>>>>> connections have to do krb5i and if say the DS isn't configured for >>>>>> Kerberos, then IO would fallback to MDS. I no longer have a pnfs >>>>>> >>>>>> That's what I would expect, too. >>>>>> >>>>>> server to verify whether or not what I say is true but that is what my >>>>>> memory tells me is the case. >>>>>> >>>>>> >>>>>> Thanks, Marc. >>>>>> >>>>>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>>>>> stripe count 1 >>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>>>>> ds_num 1 >>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>>>>> auth handle (flavor 390004) >>>>>> [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 2826 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-12 6:52 ` Mkrtchyan, Tigran @ 2024-08-12 14:37 ` marc eshel 0 siblings, 0 replies; 17+ messages in thread From: marc eshel @ 2024-08-12 14:37 UTC (permalink / raw) To: Mkrtchyan, Tigran Cc: Olga Kornievskaia, schumaker anna, linux-nfs, Trond Myklebust I confirmed that it was Ganesha problem, it was configured to be DS but for some reason it responded to EXCHANGE_ID that it is not a DS so the client switched back to the MDS. Again wondering if Ganesha is used for pNFS by anyone. Thank you all for the help, Marc. On 8/11/24 11:52 PM, Mkrtchyan, Tigran wrote: > > ----- Original Message ----- >> From: "marc eshel" <eshel.marc@gmail.com> >> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Olga Kornievskaia" <aglo@umich.edu> >> Cc: "schumaker anna" <schumaker.anna@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Trond Myklebust" >> <trondmy@hammerspace.com> >> Sent: Sunday, 11 August, 2024 21:50:36 >> Subject: Re: NFS client to pNFS DS >> On 8/11/24 12:41 PM, Mkrtchyan, Tigran wrote: >>>> On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> >>>> On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote: >>>>> On 8/9/24 8:15 AM, Olga Kornievskaia wrote: >>>>>> On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote: >>>>>>> Thanks for the replies, I am a little rusty with debugging NFS but this what I >>>>>>> see when the NFS client tried to create a session with the DS. >>>>>>> >>>>>>> Ganesha was configured for sec=sys and the client mount had the option sec=sys, >>>>>>> I assume flavor 390004 means it was trying to use krb5i. >>>>>> For 4.1, the client will always try to do state operations with krb5i >>>>>> even when sec=sys when the client detects that it's configured to do >>>>>> Kerberos (ie., gssd is running). This context creation is triggered >>>>>> regardless of whether the rpc client is used for MDS or DS. >>>>>> >>>>>> My question to you: is the MDS configured with Kerberos but the DS >>>>>> isn't? And also, does this lead to a failure? >>>>> Both MDS DS are configured for sec=sys and it is leading to client >>>>> switching from DS to MDS so yes, it is pNFS failure. What I see on the >>>>> DS is the client creating a session and than imminently destroying it >>>>> before committing it. If the is something else that I can debug I will >>>>> be happy to. >>>> pnfs failure is unexpected. I'm pretty confident that a non-kerberos >>>> configured client can do normal pnfs with sec=sys. I can help debug, >>> Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus >>> works as expected. We mount always with sec=sys, and despite the fact, that >>> hosts configured to support kerberos due to AFS and GPFS, the access to DSes is >>> never an issue and never tries to use kerberos. >>> >>> Tigran. >> Hi Tigran, >> >> It is possible that it is failing back to MDS for another reason and the >> error message that I see is not the reason for the failure. Are you >> using pNFS with Ganesha and GPFS? >> >> Marc. > Hi Marc, > > pNFS is used exclusively with dCache. Can you provide the packet capture > of failing pNFS traffic, like mount + cp + umount? > > Best regards, > Tigran. > >>>> if you want to send me a network trace and tracepoint output. Btw what >>>> kernel/distro are you using? >>>> >>>>> Marc. >>>>> >>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>>>>>> auth handle (flavor 390004) >>>>>>> >>>>>>> Marc. >>>>>>> >>>>>>> On 8/9/24 6:06 AM, Anna Schumaker wrote: >>>>>>> >>>>>>> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>> >>>>>>> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote: >>>>>>> >>>>>>> Hi Trond, >>>>>>> >>>>>>> Will the Linux NFS client try to us krb5i regardless of the MDS >>>>>>> configuration? >>>>>>> >>>>>>> Is there any option to avoid it? >>>>>>> >>>>>>> I was under the impression the linux client has no way of choosing a >>>>>>> different auth_gss security flavor for the DS than the MDS. Meaning >>>>>>> >>>>>>> That's a good point, I completely missed that this is specifically for the DS. >>>>>>> >>>>>>> that if mount command has say sec=krb5i then both MDS and DS >>>>>>> connections have to do krb5i and if say the DS isn't configured for >>>>>>> Kerberos, then IO would fallback to MDS. I no longer have a pnfs >>>>>>> >>>>>>> That's what I would expect, too. >>>>>>> >>>>>>> server to verify whether or not what I say is true but that is what my >>>>>>> memory tells me is the case. >>>>>>> >>>>>>> >>>>>>> Thanks, Marc. >>>>>>> >>>>>>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>>>>>> stripe count 1 >>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node >>>>>>> ds_num 1 >>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create >>>>>>> auth handle (flavor 390004) >>>>>>> ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-08-11 19:41 ` Mkrtchyan, Tigran 2024-08-11 19:50 ` marc eshel @ 2024-09-09 23:23 ` marc eshel 2024-09-10 7:23 ` Mkrtchyan, Tigran 1 sibling, 1 reply; 17+ messages in thread From: marc eshel @ 2024-09-09 23:23 UTC (permalink / raw) To: Mkrtchyan, Tigran, Olga Kornievskaia Cc: Anna Schumaker, linux-nfs, Trond Myklebust Can someone explain why the NFS client is reading 1M followed by 2 reads of 1/2M and repeats this pattern. For pNFS or NFS4. The mount was for 4M rsize=4194304. Thanks, Marc. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-09-09 23:23 ` marc eshel @ 2024-09-10 7:23 ` Mkrtchyan, Tigran 2024-09-11 5:54 ` Marc Eshel 2024-09-13 16:46 ` pNFS client is not using the stripe value marc eshel 0 siblings, 2 replies; 17+ messages in thread From: Mkrtchyan, Tigran @ 2024-09-10 7:23 UTC (permalink / raw) To: marc eshel; +Cc: Olga Kornievskaia, schumaker anna, linux-nfs, Trond Myklebust [-- Attachment #1: Type: text/plain, Size: 732 bytes --] Hi Marc, AFAIK, 1M is the max IO size that linux nfs client will use. linux/nfs_xdr.h:#define NFS_MAX_FILE_IO_SIZE (1048576U) Best regards, Tigran. ----- Original Message ----- > From: "marc eshel" <eshel.marc@gmail.com> > To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Olga Kornievskaia" <aglo@umich.edu> > Cc: "schumaker anna" <schumaker.anna@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Trond Myklebust" > <trondmy@hammerspace.com> > Sent: Tuesday, 10 September, 2024 01:23:43 > Subject: Re: NFS client to pNFS DS > Can someone explain why the NFS client is reading 1M followed by 2 reads > of 1/2M and repeats this pattern. For pNFS or NFS4. > > The mount was for 4M rsize=4194304. > > Thanks, Marc. [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 2826 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: NFS client to pNFS DS 2024-09-10 7:23 ` Mkrtchyan, Tigran @ 2024-09-11 5:54 ` Marc Eshel 2024-09-13 16:46 ` pNFS client is not using the stripe value marc eshel 1 sibling, 0 replies; 17+ messages in thread From: Marc Eshel @ 2024-09-11 5:54 UTC (permalink / raw) To: Mkrtchyan, Tigran Cc: Olga Kornievskaia, schumaker anna, linux-nfs, Trond Myklebust Hi Tigran, Thank you for the information, but still why there are more 512K reads than 1M, I see now about 5 reads of size 512K for every 1M read. Marc. On 9/10/24 12:23 AM, Mkrtchyan, Tigran wrote: > Hi Marc, > > AFAIK, 1M is the max IO size that linux nfs client will use. > > linux/nfs_xdr.h:#define NFS_MAX_FILE_IO_SIZE (1048576U) > > Best regards, > Tigran. > > ----- Original Message ----- >> From: "marc eshel" <eshel.marc@gmail.com> >> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Olga Kornievskaia" <aglo@umich.edu> >> Cc: "schumaker anna" <schumaker.anna@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Trond Myklebust" >> <trondmy@hammerspace.com> >> Sent: Tuesday, 10 September, 2024 01:23:43 >> Subject: Re: NFS client to pNFS DS >> Can someone explain why the NFS client is reading 1M followed by 2 reads >> of 1/2M and repeats this pattern. For pNFS or NFS4. >> >> The mount was for 4M rsize=4194304. >> >> Thanks, Marc. ^ permalink raw reply [flat|nested] 17+ messages in thread
* pNFS client is not using the stripe value 2024-09-10 7:23 ` Mkrtchyan, Tigran 2024-09-11 5:54 ` Marc Eshel @ 2024-09-13 16:46 ` marc eshel 1 sibling, 0 replies; 17+ messages in thread From: marc eshel @ 2024-09-13 16:46 UTC (permalink / raw) To: Olga Kornievskaia Cc: Olga Kornievskaia, schumaker anna, linux-nfs, Trond Myklebust, Mkrtchyan, Tigran This trace on the client shows that it was requested to use *nfl_util 0x400000* filelayout_decode_layout: set_layout_map Begin Sep 13 09:20:10 svl-marcrh-node-1 kernel: nfs4_print_deviceid: device id= [3000035a0b38c07df0465] Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_decode_layout: *nfl_util 0x400000* num_fh 1 fsi 0 po 0 Sep 13 09:20:10 svl-marcrh-node-1 kernel: DEBUG: filelayout_decode_layout: fh len 61 Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_check_layout Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_check_layout returns 0 This trace on the DS show that it reads 0x48000 before skipping to 0x880000 So it is skipping *0x400000 *but way was the first chuck 0x48000 Also way some read are 1048576 and most are 524288 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0C47878B0 vinfoP 0xFFFF91F81352FA30 off 0 len 1048576 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0C47878B0 vinfoP 0xFFFF91F81352FA30 off 1048576 len 524288 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA01F8B0 vinfoP 0xFFFF91F81352FA30 off 1572864 len 1048576 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA5FB8B0 vinfoP 0xFFFF91F81352FA30 off 2621440 len 524288 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA01F8B0 vinfoP 0xFFFF91F81352FA30 off 3145728 len 524288 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0C47878B0 vinfoP 0xFFFF91F81352FA30 off 3670016 len 1048576 gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA01F8B0 vinfoP 0xFFFF91F81352FA30 off 8912896 len 524288 Thanks, Marc. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2024-09-13 16:46 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <c4f1d8bf-745b-4a98-9d38-2da4c355d691@gmail.com>
2024-07-29 21:56 ` NFS client failure marc eshel
2024-08-05 21:51 ` NFS client to pNFS DS marc eshel
2024-08-08 14:22 ` Anna Schumaker
2024-08-08 22:07 ` Olga Kornievskaia
2024-08-09 13:06 ` Anna Schumaker
2024-08-09 14:29 ` marc eshel
[not found] ` <8ab0fd49-0c90-42bd-a34e-9dcf63a99bd5@gmail.com>
2024-08-09 15:15 ` Olga Kornievskaia
2024-08-09 16:26 ` marc eshel
2024-08-10 21:20 ` Olga Kornievskaia
2024-08-11 19:41 ` Mkrtchyan, Tigran
2024-08-11 19:50 ` marc eshel
2024-08-12 6:52 ` Mkrtchyan, Tigran
2024-08-12 14:37 ` marc eshel
2024-09-09 23:23 ` marc eshel
2024-09-10 7:23 ` Mkrtchyan, Tigran
2024-09-11 5:54 ` Marc Eshel
2024-09-13 16:46 ` pNFS client is not using the stripe value marc eshel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.