Re: [PATCH 1/6] NFSD: add the ability to enable use of RWF_DONTCACHE for all IO

Chuck Lever
2025-06-30T14:50:58Z

On 6/19/25 4:19 PM, Mike Snitzer wrote:
> On Tue, Jun 17, 2025 at 01:31:23PM -0400, Chuck Lever wrote:
>> On 6/17/25 1:22 PM, Mike Snitzer wrote:
>>> On Mon, Jun 16, 2025 at 12:10:38PM -0400, Mike Snitzer wrote:
>>>> On Mon, Jun 16, 2025 at 09:32:16AM -0400, Chuck Lever wrote:
>>>>> On 6/12/25 12:00 PM, Mike Snitzer wrote:
>>>>>> On Thu, Jun 12, 2025 at 09:21:35AM -0400, Chuck Lever wrote:
>>>>>>> On 6/11/25 3:18 PM, Mike Snitzer wrote:
>>>>>>>> On Wed, Jun 11, 2025 at 10:31:20AM -0400, Chuck Lever wrote:
>>>>>>>>> On 6/10/25 4:57 PM, Mike Snitzer wrote:
>>>>>>>>>> Add 'enable-dontcache' to NFSD's debugfs interface so that: Any data
>>>>>>>>>> read or written by NFSD will either not be cached (thanks to O_DIRECT)
>>>>>>>>>> or will be removed from the page cache upon completion (DONTCACHE).
>>>>>>>>>
>>>>>>>>> I thought we were going to do two switches: One for reads and one for
>>>>>>>>> writes? I could be misremembering.
>>>>>>>>
>>>>>>>> We did discuss the possibility of doing that. Still can-do if that's
>>>>>>>> what you'd prefer.
>>>>>>>
>>>>>>> For our experimental interface, I think having read and write enablement
>>>>>>> as separate settings is wise, so please do that.
>>>>>>>
>>>>>>> One quibble, though: The name "enable_dontcache" might be directly
>>>>>>> meaningful to you, but I think others might find "enable_dont" to be
>>>>>>> oxymoronic. And, it ties the setting to a specific kernel technology:
>>>>>>> RWF_DONTCACHE.
>>>>>>>
>>>>>>> So: Can we call these settings "io_cache_read" and "io_cache_write" ?
>>>>>>>
>>>>>>> They could each carry multiple settings:
>>>>>>>
>>>>>>> 0: Use page cache
>>>>>>> 1: Use RWF_DONTCACHE
>>>>>>> 2: Use O_DIRECT
>>>>>>>
>>>>>>> You can choose to implement any or all of the above three mechanisms.
>>>>>>
>>>>>> I like it, will do for v2. But will have O_DIRECT=1 and RWF_DONTCACHE=2.
>>>>>
>>>>> For io_cache_read, either settings 1 and 2 need to set
>>>>> disable_splice_read, or the io_cache_read setting has to be considered
>>>>> by nfsd_read_splice_ok() when deciding to use nfsd_iter_read() or
>>>>> splice read.
>>>>
>>>> Yes, I understand.
>>>>
>>>>> However, it would be slightly nicer if we could decide whether splice
>>>>> read can be removed /before/ this series is merged. Can you get NFSD
>>>>> tested with IOR with disable_splice_read both enabled and disabled (no
>>>>> direct I/O)? Then we can compare the results to ensure that there is no
>>>>> negative performance impact for removing the splice read code.
>>>>
>>>> I can ask if we have a small window of opportunity to get this tested,
>>>> will let you know if so.
>>>>
>>>
>>> I was able to enlist the help of Keith (cc'd) to get some runs in to
>>> compare splice_read vs vectored read. A picture is worth 1000 words:
>>> https://original.art/NFSD_splice_vs_buffered_read_IOR_EASY.jpg
>>>
>>> Left side is with splice_read running IOR_EASY with 48, 64, 96 PPN
>>> (Processes Per Node on each client) respectively. Then the same
>>> IOR_EASY workload progression for buffered IO on the right side.
>>>
>>> 6x servers with 1TB memory and 48 cpus, each configured with 32 NFSD
>>> threads, with CPU pinning and 4M Read Ahead. 6x clients running IOR_EASY.
>>>
>>> This was Keith's take on splice_read's benefits:
>>> - Is overall faster than buffered at any PPN.
>>> - Is able to scale higher with PPN (whereas buffered is flat).
>>> - Safe to say splice_read allows NFSD to do more IO then standard
>>> buffered.
>>
>> I thank you and Keith for the data!
>
> You're welcome.
>
>>> (These results came _after_ I did the patch to remove all the
>>> splice_read related code from NFSD and SUNRPC.. while cathartic, alas
>>> it seems it isn't meant to be at this point. I'll let you do the
>>> honors in the future if/when you deem splice_read worthy of removal.)
>>
>> If we were to make all NFS READ operations use O_DIRECT, then of course
>> NFSD's splice read should be removed at that point.
>
> Yes, that makes sense. I still need to try Christoph's idea (hope to
> do so over next 24hrs):
> https://lore.kernel.org/linux-nfs/aEu3o9imaQQF9vyg@infradead.org/
>
> But for now, here is my latest NFSD O_DIRECT/DONTCACHE work, think of
> the top 6 commits as a preview of what'll be v2 of this series:
> https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=kernel-6.12.24/nfsd-testing

I was waiting for a series repost, but in the meantime...

The one thing that caught my eye was the relocation of fh_getattr().

- If fh_getattr() is to be moved to fs/nfsd/vfs.c, then it should be
renamed nfsd_getattr() (or similar) to match the API naming
convention in that file.

- If fh_getattr() is to keep its current name, then it should be
moved to where the other fh_yada() functions reside, in
fs/nfsd/nfsfh.c

In a private tree, I constructed a patch to do the latter. I can
post that for comment.

--
Chuck Lever