Re: NFS client to pNFS DS

["Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>]

[-- Attachment #1: Type: text/plain, Size: 4358 bytes --]



----- Original Message -----
> From: "marc eshel" <eshel.marc@gmail.com>
> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Olga Kornievskaia" <aglo@umich.edu>
> Cc: "schumaker anna" <schumaker.anna@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Trond Myklebust"
> <trondmy@hammerspace.com>
> Sent: Sunday, 11 August, 2024 21:50:36
> Subject: Re: NFS client to pNFS DS

> On 8/11/24 12:41 PM, Mkrtchyan, Tigran wrote:
>>> On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>
>>> On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote:
>>>>
>>>> On 8/9/24 8:15 AM, Olga Kornievskaia wrote:
>>>>> On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote:
>>>>>> Thanks for the replies, I am a little rusty with debugging NFS but this what I
>>>>>> see when the NFS client tried to create a session with the DS.
>>>>>>
>>>>>> Ganesha was configured for sec=sys and the client mount had the option sec=sys,
>>>>>> I assume flavor 390004 means it was trying to use krb5i.
>>>>> For 4.1, the client will always try to do state operations with krb5i
>>>>> even when sec=sys when the client detects that it's configured to do
>>>>> Kerberos (ie., gssd is running). This context creation is triggered
>>>>> regardless of whether the rpc client is used for MDS or DS.
>>>>>
>>>>> My question to you: is the MDS configured with Kerberos but the DS
>>>>> isn't? And also, does this lead to a failure?
>>>> Both MDS DS are configured for sec=sys and it is leading to client
>>>> switching from DS to MDS so yes, it is pNFS failure. What I see on the
>>>> DS is the client creating a session and than imminently destroying it
>>>> before committing it. If the is something else that I can debug I will
>>>> be happy to.
>>> pnfs failure is unexpected. I'm pretty confident that a non-kerberos
>>> configured client can do normal pnfs with sec=sys. I can help debug,
>> Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus
>> works as expected. We mount always with sec=sys, and despite the fact, that
>> hosts configured to support kerberos due to AFS and GPFS, the access to DSes is
>> never an issue and never tries to use kerberos.
>>
>> Tigran.
> 
> Hi Tigran,
> 
> It is possible that it is failing back to MDS for another reason and the
> error message that I see is not the reason for the failure. Are you
> using pNFS with Ganesha and GPFS?
> 
> Marc.

Hi Marc,

pNFS is used exclusively with dCache. Can you provide the packet capture
of failing pNFS traffic, like mount + cp + umount?

Best regards,
   Tigran.

> 
>>
>>> if you want to send me a network trace and tracepoint output. Btw what
>>> kernel/distro are you using?
>>>
>>>> Marc.
>>>>
>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create
>>>>>> auth handle (flavor 390004)
>>>>>>
>>>>>> Marc.
>>>>>>
>>>>>> On 8/9/24 6:06 AM, Anna Schumaker wrote:
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>>>
>>>>>> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote:
>>>>>>
>>>>>> Hi Trond,
>>>>>>
>>>>>> Will the Linux NFS client try to us krb5i regardless of the MDS
>>>>>> configuration?
>>>>>>
>>>>>> Is there any option to avoid it?
>>>>>>
>>>>>> I was under the impression the linux client has no way of choosing a
>>>>>> different auth_gss security flavor for the DS than the MDS. Meaning
>>>>>>
>>>>>> That's a good point, I completely missed that this is specifically for the DS.
>>>>>>
>>>>>> that if mount command has say sec=krb5i then both MDS and DS
>>>>>> connections have to do krb5i and if say the DS isn't configured for
>>>>>> Kerberos, then IO would fallback to MDS. I no longer have a pnfs
>>>>>>
>>>>>> That's what I would expect, too.
>>>>>>
>>>>>> server to verify whether or not what I say is true but that is what my
>>>>>> memory tells me is the case.
>>>>>>
>>>>>>
>>>>>> Thanks, Marc.
>>>>>>
>>>>>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node
>>>>>> stripe count 1
>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node
>>>>>> ds_num 1
>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create
>>>>>> auth handle (flavor 390004)
>>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2826 bytes --]