Re: NFS client to pNFS DS

[marc eshel <eshel.marc@gmail.com>]

I confirmed that it was Ganesha problem, it was configured to be DS but 
for some reason it responded to EXCHANGE_ID that it is not a DS so the 
client switched back to the MDS. Again wondering if Ganesha is used for 
pNFS by anyone.

Thank you all for the help, Marc.

On 8/11/24 11:52 PM, Mkrtchyan, Tigran wrote:
>
> ----- Original Message -----
>> From: "marc eshel" <eshel.marc@gmail.com>
>> To: "Tigran Mkrtchyan" <tigran.mkrtchyan@desy.de>, "Olga Kornievskaia" <aglo@umich.edu>
>> Cc: "schumaker anna" <schumaker.anna@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Trond Myklebust"
>> <trondmy@hammerspace.com>
>> Sent: Sunday, 11 August, 2024 21:50:36
>> Subject: Re: NFS client to pNFS DS
>> On 8/11/24 12:41 PM, Mkrtchyan, Tigran wrote:
>>>> On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>
>>>> On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote:
>>>>> On 8/9/24 8:15 AM, Olga Kornievskaia wrote:
>>>>>> On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote:
>>>>>>> Thanks for the replies, I am a little rusty with debugging NFS but this what I
>>>>>>> see when the NFS client tried to create a session with the DS.
>>>>>>>
>>>>>>> Ganesha was configured for sec=sys and the client mount had the option sec=sys,
>>>>>>> I assume flavor 390004 means it was trying to use krb5i.
>>>>>> For 4.1, the client will always try to do state operations with krb5i
>>>>>> even when sec=sys when the client detects that it's configured to do
>>>>>> Kerberos (ie., gssd is running). This context creation is triggered
>>>>>> regardless of whether the rpc client is used for MDS or DS.
>>>>>>
>>>>>> My question to you: is the MDS configured with Kerberos but the DS
>>>>>> isn't? And also, does this lead to a failure?
>>>>> Both MDS DS are configured for sec=sys and it is leading to client
>>>>> switching from DS to MDS so yes, it is pNFS failure. What I see on the
>>>>> DS is the client creating a session and than imminently destroying it
>>>>> before committing it. If the is something else that I can debug I will
>>>>> be happy to.
>>>> pnfs failure is unexpected. I'm pretty confident that a non-kerberos
>>>> configured client can do normal pnfs with sec=sys. I can help debug,
>>> Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus
>>> works as expected. We mount always with sec=sys, and despite the fact, that
>>> hosts configured to support kerberos due to AFS and GPFS, the access to DSes is
>>> never an issue and never tries to use kerberos.
>>>
>>> Tigran.
>> Hi Tigran,
>>
>> It is possible that it is failing back to MDS for another reason and the
>> error message that I see is not the reason for the failure. Are you
>> using pNFS with Ganesha and GPFS?
>>
>> Marc.
> Hi Marc,
>
> pNFS is used exclusively with dCache. Can you provide the packet capture
> of failing pNFS traffic, like mount + cp + umount?
>
> Best regards,
>     Tigran.
>
>>>> if you want to send me a network trace and tracepoint output. Btw what
>>>> kernel/distro are you using?
>>>>
>>>>> Marc.
>>>>>
>>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create
>>>>>>> auth handle (flavor 390004)
>>>>>>>
>>>>>>> Marc.
>>>>>>>
>>>>>>> On 8/9/24 6:06 AM, Anna Schumaker wrote:
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>>>>
>>>>>>> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Trond,
>>>>>>>
>>>>>>> Will the Linux NFS client try to us krb5i regardless of the MDS
>>>>>>> configuration?
>>>>>>>
>>>>>>> Is there any option to avoid it?
>>>>>>>
>>>>>>> I was under the impression the linux client has no way of choosing a
>>>>>>> different auth_gss security flavor for the DS than the MDS. Meaning
>>>>>>>
>>>>>>> That's a good point, I completely missed that this is specifically for the DS.
>>>>>>>
>>>>>>> that if mount command has say sec=krb5i then both MDS and DS
>>>>>>> connections have to do krb5i and if say the DS isn't configured for
>>>>>>> Kerberos, then IO would fallback to MDS. I no longer have a pnfs
>>>>>>>
>>>>>>> That's what I would expect, too.
>>>>>>>
>>>>>>> server to verify whether or not what I say is true but that is what my
>>>>>>> memory tells me is the case.
>>>>>>>
>>>>>>>
>>>>>>> Thanks, Marc.
>>>>>>>
>>>>>>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node
>>>>>>> stripe count 1
>>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node
>>>>>>> ds_num 1
>>>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create
>>>>>>> auth handle (flavor 390004)
>>>>>>>