issues with Linux client pNFS

issues with Linux client pNFS

[marc eshel]

Hi,

Who is the current maintainer of the Linux NFS client?

I have an issue with the way pNFS client is distributing the reads among 
the DSs and I can provide the traces if anyone cares.

Thanks, Marc.

Re: issues with Linux client pNFS

[Anna Schumaker]

Hi Marc,

On 9/16/24 4:27 PM, marc eshel wrote:
> Hi,
> 
> Who is the current maintainer of the Linux NFS client?

That would be me and Trond.

> 
> I have an issue with the way pNFS client is distributing the reads among the DSs and I can provide the traces if anyone cares.

What is the problem? And sure, send along the traces! I'll try to take a look as soon as I can.

Anna

> 
> Thanks, Marc.
> 

Re: issues with Linux client pNFS

[marc eshel]

Hi Anna,

This trace on the client shows that it was requested to use *nfl_util 
0x400000*

filelayout_decode_layout: set_layout_map Begin

Sep 13 09:20:10 svl-marcrh-node-1 kernel: nfs4_print_deviceid: device 
id= [3000035a0b38c07df0465]

Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_decode_layout: 
*nfl_util 0x400000* num_fh 1 fsi 0 po 0

Sep 13 09:20:10 svl-marcrh-node-1 kernel: DEBUG: 
filelayout_decode_layout: fh len 61

Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_check_layout

Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_check_layout 
returns 0


This trace on the DS show that it reads 0x48000 before skipping to 0x880000
as you can see it reads 0x48000 from DS 10.11.56.194 before starting to 
read from 10.11.56.195
and I also show it on the server side that revived those reads.


Sep 13 09:20:10 svl-marcrh-node-1 kernel: 
pnfs_generic_layout_insert_lseg:Begin
Sep 13 09:20:10 svl-marcrh-node-1 kernel: 
pnfs_generic_layout_insert_lseg: inserted lseg 00000000867a0461 iomode 1 
offset 0 length 18446744073709551615 at tail
Sep 13 09:20:10 svl-marcrh-node-1 kernel: 
pnfs_generic_layout_insert_lseg:Return
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 1048576@0
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.194:2049,} cl_count 2
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 524288@1048576
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.194:2049,} cl_count 1
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 1048576@1572864
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.194:2049,} cl_count 2
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 524288@2621440
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.194:2049,} cl_count 1
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 524288@3145728
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.194:2049,} cl_count 2
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 1048576@3670016
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.194:2049,} cl_count 1
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_update_layout: inode 
0:51/88064 pNFS layout segment found for (read-only, offset: 0, length: 
18446744073709551615)
Sep 13 09:20:10 svl-marcrh-node-1 kernel: --> filelayout_read_pagelist 
ino 88064 pgbase 0 req 524288@4718592
Sep 13 09:20:10 svl-marcrh-node-1 kernel: filelayout_read_pagelist USE 
DS: {10.11.56.195:2049,} cl_count 2
Sep 13 09:20:10 svl-marcrh-node-1 kernel: pnfs_find_alloc_layout Begin 
ino=000000003c8009bb layout=00000000f5222588


So it is skipping *0x400000 *but why was the first chuck 0x48000

Also way some read are 1048576 and most are 524288

And this is on the server side.

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0C47878B0 
vinfoP 0xFFFF91F81352FA30 off 0 len 1048576

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0C47878B0 
vinfoP 0xFFFF91F81352FA30 off 1048576 len 524288

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA01F8B0 
vinfoP 0xFFFF91F81352FA30 off 1572864 len 1048576

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA5FB8B0 
vinfoP 0xFFFF91F81352FA30 off 2621440 len 524288

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA01F8B0 
vinfoP 0xFFFF91F81352FA30 off 3145728 len 524288

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0C47878B0 
vinfoP 0xFFFF91F81352FA30 off 3670016 len 1048576

gpfsRead enter: gnP 0xFFFF91F810942688 flags 0x1 uioP 0xFFFFAED0CA01F8B0 
vinfoP 0xFFFF91F81352FA30 off 8912896 len 524288


Thanks, Marc.


On 9/16/24 1:33 PM, Anna Schumaker wrote:
> Hi Marc,
>
> On 9/16/24 4:27 PM, marc eshel wrote:
>> Hi,
>>
>> Who is the current maintainer of the Linux NFS client?
> That would be me and Trond.
>
>> I have an issue with the way pNFS client is distributing the reads among the DSs and I can provide the traces if anyone cares.
> What is the problem? And sure, send along the traces! I'll try to take a look as soon as I can.
>
> Anna
>
>> Thanks, Marc.
>>

Re: issues with Linux client pNFS

[marc eshel]

Hi,

I was just trying pNFS recall any but it look like the NFS client 
received but ignored the call, is that expected ?

[287100.214040] NFS: RECALL_ANY callback request from 10.11.96.99
[287100.214073] nfs4_callback_recallany: exit with status = 0

Thanks, Marc.

On 9/16/24 1:33 PM, Anna Schumaker wrote:
> Hi Marc,
>
> On 9/16/24 4:27 PM, marc eshel wrote:
>> Hi,
>>
>> Who is the current maintainer of the Linux NFS client?
> That would be me and Trond.
>
>> I have an issue with the way pNFS client is distributing the reads among the DSs and I can provide the traces if anyone cares.
> What is the problem? And sure, send along the traces! I'll try to take a look as soon as I can.
>
> Anna
>
>> Thanks, Marc.
>>

Re: issues with Linux client pNFS

[Anna Schumaker]

Hi Marc,

On Wed, Sep 16, 2026, at 6:30 PM, marc eshel wrote:
> Hi,
>
> I was just trying pNFS recall any but it look like the NFS client 
> received but ignored the call, is that expected ?
>
> [287100.214040] NFS: RECALL_ANY callback request from 10.11.96.99
> [287100.214073] nfs4_callback_recallany: exit with status = 0

Trond just had a patch the other day to fix an issue with CB_RECALL_ANY.
Any chance you're able to test it to see if it fixes your issue?

https://lore.kernel.org/linux-nfs/115e4102b1bab159c25e8e903070b55cc4de77eb.1789148631.git.trond.myklebust@hammerspace.com/

Thanks,
Anna
>
> Thanks, Marc.
>
> On 9/16/24 1:33 PM, Anna Schumaker wrote:
>> Hi Marc,
>>
>> On 9/16/24 4:27 PM, marc eshel wrote:
>>> Hi,
>>>
>>> Who is the current maintainer of the Linux NFS client?
>> That would be me and Trond.
>>
>>> I have an issue with the way pNFS client is distributing the reads among the DSs and I can provide the traces if anyone cares.
>> What is the problem? And sure, send along the traces! I'll try to take a look as soon as I can.
>>
>> Anna
>>
>>> Thanks, Marc.
>>>

Re: issues with Linux client pNFS

[marc eshel]

Hi Anna,

This fix is related to DELEGATION_RETURN only, I was trying to recall 
all file layouts.

Marc.

On 9/17/26 6:29 AM, Anna Schumaker wrote:
> Hi Marc,
>
> On Wed, Sep 16, 2026, at 6:30 PM, marc eshel wrote:
>> Hi,
>>
>> I was just trying pNFS recall any but it look like the NFS client
>> received but ignored the call, is that expected ?
>>
>> [287100.214040] NFS: RECALL_ANY callback request from 10.11.96.99
>> [287100.214073] nfs4_callback_recallany: exit with status = 0
> Trond just had a patch the other day to fix an issue with CB_RECALL_ANY.
> Any chance you're able to test it to see if it fixes your issue?
>
> https://lore.kernel.org/linux-nfs/115e4102b1bab159c25e8e903070b55cc4de77eb.1789148631.git.trond.myklebust@hammerspace.com/
>
> Thanks,
> Anna
>> Thanks, Marc.
>>
>> On 9/16/24 1:33 PM, Anna Schumaker wrote:
>>> Hi Marc,
>>>
>>> On 9/16/24 4:27 PM, marc eshel wrote:
>>>> Hi,
>>>>
>>>> Who is the current maintainer of the Linux NFS client?
>>> That would be me and Trond.
>>>
>>>> I have an issue with the way pNFS client is distributing the reads among the DSs and I can provide the traces if anyone cares.
>>> What is the problem? And sure, send along the traces! I'll try to take a look as soon as I can.
>>>
>>> Anna
>>>
>>>> Thanks, Marc.
>>>>

Re: issues with Linux client pNFS

[marc eshel]

Hi Anna,

Reading the code around this patch it looks like some other flags are 
checked, so I tried to add RCA4_TYPE_MASK_OTHER_LAYOUT_MAX and with this 
flags file layout recall did work.

Thanks, Marc.

On 9/17/26 8:42 AM, marc eshel wrote:
> Hi Anna,
>
> This fix is related to DELEGATION_RETURN only, I was trying to recall 
> all file layouts.
>
> Marc.
>
> On 9/17/26 6:29 AM, Anna Schumaker wrote:
>> Hi Marc,
>>
>> On Wed, Sep 16, 2026, at 6:30 PM, marc eshel wrote:
>>> Hi,
>>>
>>> I was just trying pNFS recall any but it look like the NFS client
>>> received but ignored the call, is that expected ?
>>>
>>> [287100.214040] NFS: RECALL_ANY callback request from 10.11.96.99
>>> [287100.214073] nfs4_callback_recallany: exit with status = 0
>> Trond just had a patch the other day to fix an issue with CB_RECALL_ANY.
>> Any chance you're able to test it to see if it fixes your issue?
>>
>> https://lore.kernel.org/linux-nfs/115e4102b1bab159c25e8e903070b55cc4de77eb.1789148631.git.trond.myklebust@hammerspace.com/ 
>>
>>
>> Thanks,
>> Anna
>>> Thanks, Marc.
>>>
>>> On 9/16/24 1:33 PM, Anna Schumaker wrote:
>>>> Hi Marc,
>>>>
>>>> On 9/16/24 4:27 PM, marc eshel wrote:
>>>>> Hi,
>>>>>
>>>>> Who is the current maintainer of the Linux NFS client?
>>>> That would be me and Trond.
>>>>
>>>>> I have an issue with the way pNFS client is distributing the reads 
>>>>> among the DSs and I can provide the traces if anyone cares.
>>>> What is the problem? And sure, send along the traces! I'll try to 
>>>> take a look as soon as I can.
>>>>
>>>> Anna
>>>>
>>>>> Thanks, Marc.
>>>>>