Re: [LSF/MM/BPF TOPIC] Lustre filesystem upstreaming

["Day, Timothy" <timday@amazon.com>]

> On 1/24/25, 4:09 PM, "Matthew Wilcox" <willy@infradead.org <mailto:willy@infradead.org>> wrote:
> > On Fri, Jan 24, 2025 at 08:50:02PM +0000, Day, Timothy wrote:
> > Lustre has already received a plethora of feedback in the past.
> > While much of that has been addressed since - the kernel is a
> > moving target. Several filesystems have been merged (or removed)
> > since Lustre left staging. We're aiming to avoid the mistakes of
> > the past and hope to address as many concerns as possible before
> > submitting for inclusion.
>
>
> I'm broadly in favour of adding Lustre, however I'd really like it to not
> increase my workload substantially. Ideally it would use iomap instead of
> buffer heads (although maybe that's not feasible).

The place Lustre uses buffer heads is osd-ldiskfs (the interface between
the Lustre server and ext4). And that's an artifact of ext4's usage of
buffer heads. I don’t see usage otherwise. The way the Lustre server
interfaces with ext4 is probably a bigger question.

> What's not negotiable for me is the use of folios; Lustre must be
> fully converted to the folio API. No use of any of the functions in
> mm/folio-compat.c. If you can grep for 'struct page' in Lustre and
> find nothing, that's a great place to be (not all filesystems in the
> kernel have reached that stage yet, and there are good reasons why some
> filesystems still use bare pages).
>
>
> Support for large folios would not be a requirement. It's just a really
> good idea if you care about performance ;-)

There's been some work towards folios, but nothing comprehensive i.e. we
still have a bunch of users of mm/folio-compat.c. I've seen some patches in-flight
for large folios, but nothing landed.

> I hope it doesn't still use ->writepage. We're almost rid of it in
> filesystems.

It's still there - I don't think anyone has seriously looked at how well
Lustre behaves without it.

> Ultimately, I think you'll want to describe the workflow you see Lustre
> adopting once it's upstream -- I've had too many filesystems say to me
> "Oh, you have to submit your patch against our git tree and then we'll
> apply it to the kernel later". That's not acceptable; the kernel is
> upstream, not your private git tree.

This is probably the biggest question. Staging didn't fare well with most
development happening out-of-tree. We have to rework the development
workflow to somehow generate patches against an actual kernel
tree versus our separate git tree. I have a high level idea of how we'd
get there - in terms of reorganizing and splitting the existing repo [1].

That's what I'd be most interested in discussing at LSF. If we change
the development model, how do we demonstrate the model is effective?

I guess another interesting question would be: has any other subsystem
or major driver undergone a transition like this before?

Tim Day

[1] https://wiki.lustre.org/Upstream_contributing