Are THPs the right model for the pagecache?

[Matthew Wilcox <willy@infradead.org>]

When I started working on using larger pages in the page cache, I was
thinking about calling them large pages or lpages.  As I worked my way
through the code, I switched to simply adopting the transparent huge
page terminology that is used by anonymous and shmem.  I just changed
the definition so that a thp is a page of arbitrary order.

But now I'm wondering if that expediency has brought me to the right
place.  To enable THP, you have to select CONFIG_TRANSPARENT_HUGEPAGE,
which is only available on architectures which support using larger TLB
entries to map PMD-sized pages.  Fair enough, since that was the original
definition, but the point of suppoting larger page sizes in the page
cache is to reduce software overhead.  Why shouldn't Alpha or m68k use
large pages in the page cache, even if they can't use them in their TLBs?

I'm also thinking about the number of asserts about
PageHead/PageTail/PageCompound and the repeated invocations of
compound_head().  If we had a different type for large pages, we could use
the compiler to assert these things instead of putting in runtime asserts.

IOWs, something like this:

struct lpage {
	struct page subpages[4];
};

static inline struct lpage *page_lpage(struct page *page)
{
	unsigned long head = READ_ONCE(page->compound_head);

	if (unlikely(head & 1))
		return (struct lpage *)(head - 1);
	return (struct lpage *)page;
}

We can then work our way through the code, distinguishing between
functions which really want to get an lpage (ie ones which currently
assert that they see only a PageHead) and functions which want to get
a particular subpage.

Some functions are going to need to be split.  eg pagecache_get_page()
currently takes an FGP_HEAD flag which determines whether it returns
a head page or the subpage for the index.  FGP_HEAD will have to
go away in favour of having separate pagecache_get_subpage() and
pagecache_get_lpage().  Or preferably, all callers of pagecache_get_page()
get converted to use lpages and they can call find_subpage() all by
themselves, if they need it.

Feels like a lot of work, but it can be done gradually.  My fear with
the current code is that filesystem writers who want to convert to
supporting THPs are not going to understand which interfaces expect a
THP and which expect a subpage.  For example, vmf->page (in the mkwrite
handler) is a subpage.  But the page passed to ->readpage is a THP.
I don't think we're going to be able to switch either of those any time
soon, so distinguishing them with a type seems only fair to fs authors.
See, for example, Darrick's reasonable question here:
https://lore.kernel.org/linux-fsdevel/20201014161216.GE9832@magnolia/

I'm not volunteering to do any of this in time for the next merge window!
I have lots of patches to get approved by various maintainers in the
next two weeks!