[LSF/MM/BPF TOPIC] Netfs support library

[LSF/MM/BPF TOPIC] Netfs support library

[David Howells]

I've been working on a library (in fs/netfs/) to provide network filesystem
support services, with help particularly from Jeff Layton.  The idea is to
move the common features of the VM interface, including request splitting,
operation retrying, local caching, content encryption, bounce buffering and
compression into one place so that various filesystems can share it.

This also intersects with the folios topic as one of the reasons for this now
is to hide as much of the existence of folios/pages from the filesystem,
instead giving it persistent iov iterators to describe the buffers available
to it.

It could be useful to get various network filesystem maintainers together to
discuss it and how to do parts of it and how to roll it out into more
filesystems if it suits them.  This might qualify more for a BoF session than
a full FS track session.

Further, discussion of designing a more effective cache backend could be
useful.  I'm thinking along the lines of something that can store its data on
a single file (or a raw blockdev) with indexing along the lines of what
filesystem drivers such as openafs do.

David

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Chuck Lever III]

> On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
> 
> I've been working on a library (in fs/netfs/) to provide network filesystem
> support services, with help particularly from Jeff Layton.  The idea is to
> move the common features of the VM interface, including request splitting,
> operation retrying, local caching, content encryption, bounce buffering and
> compression into one place so that various filesystems can share it.

IIUC this suite of functions is beneficial mainly to clients,
is that correct? I'd like to be clear about that, this is not
an objection to the topic.

I'm interested in discussing how folios might work for the
NFS _server_, perhaps as a separate or adjunct conversation.


> This also intersects with the folios topic as one of the reasons for this now
> is to hide as much of the existence of folios/pages from the filesystem,
> instead giving it persistent iov iterators to describe the buffers available
> to it.
> 
> It could be useful to get various network filesystem maintainers together to
> discuss it and how to do parts of it and how to roll it out into more
> filesystems if it suits them.  This might qualify more for a BoF session than
> a full FS track session.
> 
> Further, discussion of designing a more effective cache backend could be
> useful.  I'm thinking along the lines of something that can store its data on
> a single file (or a raw blockdev) with indexing along the lines of what
> filesystem drivers such as openafs do.
> 
> David
> 

--
Chuck Lever

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Matthew Wilcox]

On Tue, Feb 01, 2022 at 02:16:54AM +0000, Chuck Lever III wrote:
> 
> 
> > On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
> > 
> > I've been working on a library (in fs/netfs/) to provide network filesystem
> > support services, with help particularly from Jeff Layton.  The idea is to
> > move the common features of the VM interface, including request splitting,
> > operation retrying, local caching, content encryption, bounce buffering and
> > compression into one place so that various filesystems can share it.
> 
> IIUC this suite of functions is beneficial mainly to clients,
> is that correct? I'd like to be clear about that, this is not
> an objection to the topic.
> 
> I'm interested in discussing how folios might work for the
> NFS _server_, perhaps as a separate or adjunct conversation.

I'd be happy to have that discussion with you, possibly in advance of
LSFMM.  I have a fortnightly Zoom call (which I put up on Youtube),
and I'd be happy to have this as a topic one week if more people are
interested than just you & I.  If it's just you & me, then we can chat
any time ;-)

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Gao Xiang]

Hi,

On Mon, Jan 31, 2022 at 09:06:13PM +0000, David Howells wrote:
> I've been working on a library (in fs/netfs/) to provide network filesystem
> support services, with help particularly from Jeff Layton.  The idea is to
> move the common features of the VM interface, including request splitting,
> operation retrying, local caching, content encryption, bounce buffering and
> compression into one place so that various filesystems can share it.
> 
> This also intersects with the folios topic as one of the reasons for this now
> is to hide as much of the existence of folios/pages from the filesystem,
> instead giving it persistent iov iterators to describe the buffers available
> to it.
> 
> It could be useful to get various network filesystem maintainers together to
> discuss it and how to do parts of it and how to roll it out into more
> filesystems if it suits them.  This might qualify more for a BoF session than
> a full FS track session.

We are interested in fscache on-demand read use cases, assuming fscache
plans to be a generic filesystem caching framework besides just caching
network data [1] [2] [3].

So fscache is still preferred to us instead of re-inventing another
cache-managing wheel like this and the modification seems very limited.
We hope this feature could be upstreamed this year in order to benefit
the whole container image ecosystem. Jeffle will post v3 after lunar
new year holidays.

I also think more potential possibility, such as fscache over iomap
interface or similiar, since iomap is also another powerful I/O
framework. IMHO, at least having a unique interface in the long
term may be a good idea for all fses.

> 
> Further, discussion of designing a more effective cache backend could be
> useful.  I'm thinking along the lines of something that can store its data on
> a single file (or a raw blockdev) with indexing along the lines of what
> filesystem drivers such as openafs do.

With my own limited understanding, I'm not sure how one single huge
file works quite well (assuming appending write is needed comparing
with raw blockdev). Or you may need some fallocate() to allocate space
in bulk if there is not enough caching space, which is much like what
now databases do.

[1] https://lore.kernel.org/r/20220118131216.85338-1-jefflexu@linux.alibaba.com
[2] https://lore.kernel.org/r/Yeeye2AUZITDsdh8@B-P7TQMD6M-0146.local
[3] https://lore.kernel.org/r/8f73d28e-db30-f2e4-0143-9d75c4b13087@linux.alibaba.com

Thanks,
Gao Xiang

> 
> David
> 

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Christian Brauner]

On Mon, Jan 31, 2022 at 09:06:13PM +0000, David Howells wrote:
> I've been working on a library (in fs/netfs/) to provide network filesystem
> support services, with help particularly from Jeff Layton.  The idea is to
> move the common features of the VM interface, including request splitting,
> operation retrying, local caching, content encryption, bounce buffering and
> compression into one place so that various filesystems can share it.
> 
> This also intersects with the folios topic as one of the reasons for this now
> is to hide as much of the existence of folios/pages from the filesystem,
> instead giving it persistent iov iterators to describe the buffers available
> to it.
> 
> It could be useful to get various network filesystem maintainers together to
> discuss it and how to do parts of it and how to roll it out into more
> filesystems if it suits them.  This might qualify more for a BoF session than
> a full FS track session.
> 
> Further, discussion of designing a more effective cache backend could be
> useful.  I'm thinking along the lines of something that can store its data on
> a single file (or a raw blockdev) with indexing along the lines of what
> filesystem drivers such as openafs do.

I do have some interest in this area as well so would like to attend
that session.

Christian

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Chuck Lever III]

Hi Matthew-

> On Jan 31, 2022, at 9:32 PM, Matthew Wilcox <willy@infradead.org> wrote:
> 
> On Tue, Feb 01, 2022 at 02:16:54AM +0000, Chuck Lever III wrote:
>> 
>> 
>>> On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
>>> 
>>> I've been working on a library (in fs/netfs/) to provide network filesystem
>>> support services, with help particularly from Jeff Layton.  The idea is to
>>> move the common features of the VM interface, including request splitting,
>>> operation retrying, local caching, content encryption, bounce buffering and
>>> compression into one place so that various filesystems can share it.
>> 
>> IIUC this suite of functions is beneficial mainly to clients,
>> is that correct? I'd like to be clear about that, this is not
>> an objection to the topic.
>> 
>> I'm interested in discussing how folios might work for the
>> NFS _server_, perhaps as a separate or adjunct conversation.
> 
> I'd be happy to have that discussion with you, possibly in advance of
> LSFMM.  I have a fortnightly Zoom call (which I put up on Youtube),
> and I'd be happy to have this as a topic one week if more people are
> interested than just you & I.  If it's just you & me, then we can chat
> any time ;-)

I suspect the set of interested parties is small, but is not
limited to just two. A separate Zoom discussion for NFS server
developers might be helpful. Let's co-ordinate offline.


--
Chuck Lever

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Steve French]

This discussion of netfs, folios and loosely related fscache changes
is important for multiple filesystems, and I would be very interested
in participating.   I think a BoF on the network fs related parts of
it makes sense, but the folios topic is broad enough to for an FS
track session.

On Tue, Feb 1, 2022 at 2:50 PM David Howells <dhowells@redhat.com> wrote:
>
> I've been working on a library (in fs/netfs/) to provide network filesystem
> support services, with help particularly from Jeff Layton.  The idea is to
> move the common features of the VM interface, including request splitting,
> operation retrying, local caching, content encryption, bounce buffering and
> compression into one place so that various filesystems can share it.
>
> This also intersects with the folios topic as one of the reasons for this now
> is to hide as much of the existence of folios/pages from the filesystem,
> instead giving it persistent iov iterators to describe the buffers available
> to it.
>
> It could be useful to get various network filesystem maintainers together to
> discuss it and how to do parts of it and how to roll it out into more
> filesystems if it suits them.  This might qualify more for a BoF session than
> a full FS track session.
>
> Further, discussion of designing a more effective cache backend could be
> useful.  I'm thinking along the lines of something that can store its data on
> a single file (or a raw blockdev) with indexing along the lines of what
> filesystem drivers such as openafs do.
>
> David
>


-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] Netfs support library

[David Howells]

Chuck Lever III <chuck.lever@oracle.com> wrote:

> > I've been working on a library (in fs/netfs/) to provide network filesystem
> > support services, with help particularly from Jeff Layton.  The idea is to
> > move the common features of the VM interface, including request splitting,
> > operation retrying, local caching, content encryption, bounce buffering and
> > compression into one place so that various filesystems can share it.
> 
> IIUC this suite of functions is beneficial mainly to clients,
> is that correct? I'd like to be clear about that, this is not
> an objection to the topic.

Yes, this is intended as client-side support only.

David

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Steve French]

On Tue, Feb 1, 2022 at 10:51 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
> > On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
> >
> > I've been working on a library (in fs/netfs/) to provide network filesystem
> > support services, with help particularly from Jeff Layton.  The idea is to
> > move the common features of the VM interface, including request splitting,
> > operation retrying, local caching, content encryption, bounce buffering and
> > compression into one place so that various filesystems can share it.
>
> IIUC this suite of functions is beneficial mainly to clients,
> is that correct? I'd like to be clear about that, this is not
> an objection to the topic.
>
> I'm interested in discussing how folios might work for the
> NFS _server_, perhaps as a separate or adjunct conversation.

That is an interesting point.   Would like to also discuss whether it
could help ksmbd,
and would like to continue discussion of netfs improvements - especially am
interested in how we can improve throttling when network (or server)
is congested
(or as network adapters are added/removed and additional bandwidth is
available).


-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Chuck Lever III]

> On Mar 4, 2022, at 3:06 PM, Steve French <smfrench@gmail.com> wrote:
> 
> On Tue, Feb 1, 2022 at 10:51 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>> 
>>> On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
>>> 
>>> I've been working on a library (in fs/netfs/) to provide network filesystem
>>> support services, with help particularly from Jeff Layton.  The idea is to
>>> move the common features of the VM interface, including request splitting,
>>> operation retrying, local caching, content encryption, bounce buffering and
>>> compression into one place so that various filesystems can share it.
>> 
>> IIUC this suite of functions is beneficial mainly to clients,
>> is that correct? I'd like to be clear about that, this is not
>> an objection to the topic.
>> 
>> I'm interested in discussing how folios might work for the
>> NFS _server_, perhaps as a separate or adjunct conversation.
> 
> That is an interesting point.   Would like to also discuss whether it
> could help ksmbd,

Agreed that ksmbd might also benefit.

Of primary interest is how read and write operations move
their payloads from the network transports directly to the
page cache. For the most efficient operation, the transport
layer would need to set up a receive buffer using page
cache pages instead of what is currently done: receive
buffers are constructed from anonymous pages and then these
pages replace the file's page cache pages.

RDMA transports make this straightforward, at least from
the transport layer's perspective.


> and would like to continue discussion of netfs improvements - especially am
> interested in how we can improve throttling when network (or server)
> is congested
> (or as network adapters are added/removed and additional bandwidth is
> available).

As I understand it, the protocols themselves have flow control
built in. At least SMB Direct and NFS/RDMA do, and NFSv4
sessions provides a similar mechanism for transports that don't
have a native flow control mechanism. I'm not sufficiently
familiar with SMB to know how flow control works there.

However, each of these mechanisms all have their own specific
rules. I'm not sure if there's much that can be extracted into
a common library, but can you be more specific about what is
needed?


--
Chuck Lever

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Steve French]

On Wed, Mar 9, 2022 at 10:17 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
> > On Mar 4, 2022, at 3:06 PM, Steve French <smfrench@gmail.com> wrote:
> >
> > On Tue, Feb 1, 2022 at 10:51 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >>
> >>> On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
> >>>
> >>> I've been working on a library (in fs/netfs/) to provide network filesystem
> >>> support services, with help particularly from Jeff Layton.  The idea is to
> >>> move the common features of the VM interface, including request splitting,
> >>> operation retrying, local caching, content encryption, bounce buffering and
> >>> compression into one place so that various filesystems can share it.
> >>
> >> IIUC this suite of functions is beneficial mainly to clients,
> >> is that correct? I'd like to be clear about that, this is not
> >> an objection to the topic.
> >>
> >> I'm interested in discussing how folios might work for the
> >> NFS _server_, perhaps as a separate or adjunct conversation.
> >
> > That is an interesting point.   Would like to also discuss whether it
> > could help ksmbd,
>
> Agreed that ksmbd might also benefit.
>
> Of primary interest is how read and write operations move
> their payloads from the network transports directly to the
> page cache. For the most efficient operation, the transport
> layer would need to set up a receive buffer using page
> cache pages instead of what is currently done: receive
> buffers are constructed from anonymous pages and then these
> pages replace the file's page cache pages.
>
> RDMA transports make this straightforward, at least from
> the transport layer's perspective.
>
>
> > and would like to continue discussion of netfs improvements - especially am
> > interested in how we can improve throttling when network (or server)
> > is congested
> > (or as network adapters are added/removed and additional bandwidth is
> > available).
>
> As I understand it, the protocols themselves have flow control
> built in. At least SMB Direct and NFS/RDMA do, and NFSv4
> sessions provides a similar mechanism for transports that don't
> have a native flow control mechanism. I'm not sufficiently
> familiar with SMB to know how flow control works there.
>
> However, each of these mechanisms all have their own specific
> rules. I'm not sure if there's much that can be extracted into
> a common library, but can you be more specific about what is
> needed?

At a high level given network latencies, we want to make sure
a few requests are in process on each interface unless server
is throttling back (or client observers problems) so the
network or cluster fs probably needs to be able to set:
- optimal i/o size (larger is usually better, for most SMB3.1.1
cases it would typically be 1M or 4M, but server negotiates
this with client at mount time)
- typical number of I/Os in flight (e.g. for SMB3.1.1 this
might be 2 or 3 times the number of channels)

and then dynamically we need to be able to reduce
(and then increase back to the default) the number of i/o when two things
happen:
- low on credits (the server returns 'credits' available
on each response, so when server is busy this number
can go down to the point where we don't want to do
much readahead)
- network or server events that indicate temporarily stopping
readahead (e.g. reconnect, or errors indicating server
resource problems)
- server notification about resource movement (server or share move)
- server notification about adding or deleting a network interface
(this and the above are handled via a loosely related
protocol called the 'Witness protocol' which generates dynamic
server notification to the client about important events).
Samuel Cabrero might have additional ideas about new events
we could add that could help here.

There also may be low memory situations where we see a need to
reduce the amount of resources used by the client fs.
-- 
Thanks,

Steve