Re: NFS client to pNFS DS

[marc eshel <eshel.marc@gmail.com>]

On 8/11/24 12:41 PM, Mkrtchyan, Tigran wrote:
>> On 10. Aug 2024, at 23:20, Olga Kornievskaia <aglo@umich.edu> wrote:
>>
>> On Fri, Aug 9, 2024 at 12:27 PM marc eshel <eshel.marc@gmail.com> wrote:
>>>
>>> On 8/9/24 8:15 AM, Olga Kornievskaia wrote:
>>>> On Fri, Aug 9, 2024 at 10:09 AM marc eshel <eshel.marc@gmail.com> wrote:
>>>>> Thanks for the replies, I am a little rusty with debugging NFS but this what I see when the NFS client tried to create a session with the DS.
>>>>>
>>>>> Ganesha was configured for sec=sys and the client mount had the option sec=sys, I assume flavor 390004 means it was trying to use krb5i.
>>>> For 4.1, the client will always try to do state operations with krb5i
>>>> even when sec=sys when the client detects that it's configured to do
>>>> Kerberos (ie., gssd is running). This context creation is triggered
>>>> regardless of whether the rpc client is used for MDS or DS.
>>>>
>>>> My question to you: is the MDS configured with Kerberos but the DS
>>>> isn't? And also, does this lead to a failure?
>>> Both MDS DS are configured for sec=sys and it is leading to client
>>> switching from DS to MDS so yes, it is pNFS failure. What I see on the
>>> DS is the client creating a session and than imminently destroying it
>>> before committing it. If the is something else that I can debug I will
>>> be happy to.
>> pnfs failure is unexpected. I'm pretty confident that a non-kerberos
>> configured client can do normal pnfs with sec=sys. I can help debug,
> Yes, I can confirm that. All RHEL kernels and weekly -rc kernels from Linus works as expected. We mount always with sec=sys, and despite the fact, that hosts configured to support kerberos due to AFS and GPFS, the access to DSes is never an issue and never tries to use kerberos.
>
> Tigran.

Hi Tigran,

It is possible that it is failing back to MDS for another reason and the 
error message that I see is not the reason for the failure. Are you 
using pNFS with Ganesha and GPFS?

Marc.

>
>> if you want to send me a network trace and tracepoint output. Btw what
>> kernel/distro are you using?
>>
>>> Marc.
>>>
>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create
>>>>> auth handle (flavor 390004)
>>>>>
>>>>> Marc.
>>>>>
>>>>> On 8/9/24 6:06 AM, Anna Schumaker wrote:
>>>>>
>>>>> On Thu, Aug 8, 2024 at 6:07 PM Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>>
>>>>> On Mon, Aug 5, 2024 at 5:51 PM marc eshel <eshel.marc@gmail.com> wrote:
>>>>>
>>>>> Hi Trond,
>>>>>
>>>>> Will the Linux NFS client try to us krb5i regardless of the MDS
>>>>> configuration?
>>>>>
>>>>> Is there any option to avoid it?
>>>>>
>>>>> I was under the impression the linux client has no way of choosing a
>>>>> different auth_gss security flavor for the DS than the MDS. Meaning
>>>>>
>>>>> That's a good point, I completely missed that this is specifically for the DS.
>>>>>
>>>>> that if mount command has say sec=krb5i then both MDS and DS
>>>>> connections have to do krb5i and if say the DS isn't configured for
>>>>> Kerberos, then IO would fallback to MDS. I no longer have a pnfs
>>>>>
>>>>> That's what I would expect, too.
>>>>>
>>>>> server to verify whether or not what I say is true but that is what my
>>>>> memory tells me is the case.
>>>>>
>>>>>
>>>>> Thanks, Marc.
>>>>>
>>>>> ul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node
>>>>> stripe count 1
>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: nfs4_fl_alloc_deviceid_node
>>>>> ds_num 1
>>>>> Jul 30 11:10:58 svl-marcrh-node-1 kernel: RPC: Couldn't create
>>>>> auth handle (flavor 390004)
>>>>>
>>>>>