[PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Pankaj Raghav (Samsung)]

From: Pankaj Raghav <p.raghav@samsung.com>

THP support for non-tmpfs filesystem has been around for some time now.
Update the admin guide to reflect it.

While we are at it, move FilePmdMapped to previous paragraph for clarity,
and clarify ShmemPmdMapped & ShmemHugePage.

Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
Acked-by: David Hildenbrand <david@redhat.com>
---

Changes since v2:
- Address comment from Bagas Sanjaya
- Squash commits and Ack from David

 Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
index dff8d5985f0f..f8aae64e38d0 100644
--- a/Documentation/admin-guide/mm/transhuge.rst
+++ b/Documentation/admin-guide/mm/transhuge.rst
@@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
 that supports the automatic promotion and demotion of page sizes and
 without the shortcomings of hugetlbfs.
 
-Currently THP only works for anonymous memory mappings and tmpfs/shmem.
-But in the future it can expand to other filesystems.
+Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
+filesystems that support large folios.
 
 .. note::
    in the examples below we presume that the basic page size is 4K and
@@ -463,11 +463,19 @@ fields for each mapping. (Note that AnonHugePages only applies to traditional
 PMD-sized THP for historical reasons and should have been called
 AnonHugePmdMapped).
 
-The number of file transparent huge pages mapped to userspace is available
-by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``.
-To identify what applications are mapping file transparent huge pages, it
-is necessary to read ``/proc/PID/smaps`` and count the FilePmdMapped fields
-for each mapping.
+The number of PMD-sized transparent huge pages currently used by
+filesystem data (page cache) is available by reading the FileHugePages field
+in ``/proc/meminfo``. The number of these huge pages that are mapped to userspace
+is available by reading FilePmdMapped field in ``proc/meminfo``. To identify
+what applications are mapping these huge pages, it is necessary to read
+``/proc/PID/smaps`` and count the FilePmdMapped fields for each mapping.
+
+In similar fashion, the number of PMD-sized transparent huge pages currently
+used by tmpfs/shmem is available by reading the ShmemHugePages field
+in ``/proc/meminfo``. The number of these huge pages that are mapped to userspace
+is available by reading ShmemPmdMapped field in ``proc/meminfo``. To identify
+what applications are mapping these huge pages, it is necessary to read
+``/proc/PID/smaps`` and count the ShmemPmdMapped fields for each mapping.
 
 Note that reading the smaps file is expensive and reading it
 frequently will incur overhead.

base-commit: f0a16f5363325cc8d9382471cdc7b654c53254c9
-- 
2.47.2

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Luis Chamberlain]

On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> From: Pankaj Raghav <p.raghav@samsung.com>
> 
> THP support for non-tmpfs filesystem has been around for some time now.
> Update the admin guide to reflect it.
> 
> While we are at it, move FilePmdMapped to previous paragraph for clarity,
> and clarify ShmemPmdMapped & ShmemHugePage.
> 
> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> Acked-by: David Hildenbrand <david@redhat.com>
> ---
> 
> Changes since v2:
> - Address comment from Bagas Sanjaya
> - Squash commits and Ack from David
> 
>  Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
>  1 file changed, 15 insertions(+), 7 deletions(-)
> 
> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> index dff8d5985f0f..f8aae64e38d0 100644
> --- a/Documentation/admin-guide/mm/transhuge.rst
> +++ b/Documentation/admin-guide/mm/transhuge.rst
> @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
>  that supports the automatic promotion and demotion of page sizes and
>  without the shortcomings of hugetlbfs.
>  
> -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> -But in the future it can expand to other filesystems.
> +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> +filesystems that support large folios.

That seems to allude that THP can be supported on filesystems
that suppor large folios. I don't think we want to call that THP
and that can confuse folks. Leaving "currently" also seems to
indicate that there is more work to be done for THP for filesystems
but that's not true as well. So how about something like:

THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
filesystem to support it. The alternative to THP for other filesystems is to
support large folios and with it you can end up using huge pages
opportunistically.

  Luis

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[David Hildenbrand]

On 04.04.25 17:32, Luis Chamberlain wrote:
> On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
>> From: Pankaj Raghav <p.raghav@samsung.com>
>>
>> THP support for non-tmpfs filesystem has been around for some time now.
>> Update the admin guide to reflect it.
>>
>> While we are at it, move FilePmdMapped to previous paragraph for clarity,
>> and clarify ShmemPmdMapped & ShmemHugePage.
>>
>> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
>> Acked-by: David Hildenbrand <david@redhat.com>
>> ---
>>
>> Changes since v2:
>> - Address comment from Bagas Sanjaya
>> - Squash commits and Ack from David
>>
>>   Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
>>   1 file changed, 15 insertions(+), 7 deletions(-)
>>
>> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
>> index dff8d5985f0f..f8aae64e38d0 100644
>> --- a/Documentation/admin-guide/mm/transhuge.rst
>> +++ b/Documentation/admin-guide/mm/transhuge.rst
>> @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
>>   that supports the automatic promotion and demotion of page sizes and
>>   without the shortcomings of hugetlbfs.
>>   
>> -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
>> -But in the future it can expand to other filesystems.
>> +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
>> +filesystems that support large folios.
> 
> That seems to allude that THP can be supported on filesystems
> that suppor large folios. I don't think we want to call that THP
> and that can confuse folks. Leaving "currently" also seems to
> indicate that there is more work to be done for THP for filesystems
> but that's not true as well. So how about something like:
> 
> THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
> filesystem to support it. The alternative to THP for other filesystems is to
> support large folios and with it you can end up using huge pages

That makes things more complicated without a good reason.

See CONFIG_READ_ONLY_THP_FOR_FS as an early usage of the term "THP" for 
stuff we have in the pagecache. (with large folios we now properly 
implement this concept, and support more than only PMD size)

-- 
Cheers,

David / dhildenb

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Pankaj Raghav (Samsung)]

On Fri, Apr 04, 2025 at 08:32:15AM -0700, Luis Chamberlain wrote:
> On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> > From: Pankaj Raghav <p.raghav@samsung.com>
> > 
> > THP support for non-tmpfs filesystem has been around for some time now.
> > Update the admin guide to reflect it.
> > 
> > While we are at it, move FilePmdMapped to previous paragraph for clarity,
> > and clarify ShmemPmdMapped & ShmemHugePage.
> > 
> > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > Acked-by: David Hildenbrand <david@redhat.com>
> > ---
> > 
> > Changes since v2:
> > - Address comment from Bagas Sanjaya
> > - Squash commits and Ack from David
> > 
> >  Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
> >  1 file changed, 15 insertions(+), 7 deletions(-)
> > 
> > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> > index dff8d5985f0f..f8aae64e38d0 100644
> > --- a/Documentation/admin-guide/mm/transhuge.rst
> > +++ b/Documentation/admin-guide/mm/transhuge.rst
> > @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
> >  that supports the automatic promotion and demotion of page sizes and
> >  without the shortcomings of hugetlbfs.
> >  
> > -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> > -But in the future it can expand to other filesystems.
> > +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> > +filesystems that support large folios.
> 
> That seems to allude that THP can be supported on filesystems
> that suppor large folios. I don't think we want to call that THP

But we do allocate a THP in the page cache if we support large folios.

See [1] where THP was supported through page cache. From what I
understand, THP support was added first to the page cache and then large
folios (orders in between) support came later.

For example, you can mount XFS and do a simple dd as follows:

$ dd if=/dev/random of=/mnt/hello bs=1G count=2

And if you monitor `FileHugePages` after dd is done, it would have filled
up if THP is enabled.

> and that can confuse folks. Leaving "currently" also seems to
> indicate that there is more work to be done for THP for filesystems
> but that's not true as well. So how about something like:

There is more work to be done in the filesystems right? Adding large
folio support is the prerequisite here to get the THP support.

> 
> THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
> filesystem to support it. The alternative to THP for other filesystems is to
> support large folios and with it you can end up using huge pages
> opportunistically.
> 
>   Luis

[1] https://lore.kernel.org/linux-mm/20201029193405.29125-1-willy@infradead.org/

--
Pankaj

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Luis Chamberlain]

On Fri, Apr 04, 2025 at 06:18:12PM +0200, David Hildenbrand wrote:
> On 04.04.25 17:32, Luis Chamberlain wrote:
> > On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> > > From: Pankaj Raghav <p.raghav@samsung.com>
> > > 
> > > THP support for non-tmpfs filesystem has been around for some time now.
> > > Update the admin guide to reflect it.
> > > 
> > > While we are at it, move FilePmdMapped to previous paragraph for clarity,
> > > and clarify ShmemPmdMapped & ShmemHugePage.
> > > 
> > > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > > Acked-by: David Hildenbrand <david@redhat.com>
> > > ---
> > > 
> > > Changes since v2:
> > > - Address comment from Bagas Sanjaya
> > > - Squash commits and Ack from David
> > > 
> > >   Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
> > >   1 file changed, 15 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> > > index dff8d5985f0f..f8aae64e38d0 100644
> > > --- a/Documentation/admin-guide/mm/transhuge.rst
> > > +++ b/Documentation/admin-guide/mm/transhuge.rst
> > > @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
> > >   that supports the automatic promotion and demotion of page sizes and
> > >   without the shortcomings of hugetlbfs.
> > > -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> > > -But in the future it can expand to other filesystems.
> > > +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> > > +filesystems that support large folios.
> > 
> > That seems to allude that THP can be supported on filesystems
> > that suppor large folios. I don't think we want to call that THP
> > and that can confuse folks. Leaving "currently" also seems to
> > indicate that there is more work to be done for THP for filesystems
> > but that's not true as well. So how about something like:
> > 
> > THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
> > filesystem to support it. The alternative to THP for other filesystems is to
> > support large folios and with it you can end up using huge pages
> 
> That makes things more complicated without a good reason.
> 
> See CONFIG_READ_ONLY_THP_FOR_FS as an early usage of the term "THP" for
> stuff we have in the pagecache. 

OK.

> (with large folios we now properly implement
> this concept, and support more than only PMD size)

Do we really want to call large folio support on filesystems THP?

  Luis

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Luis Chamberlain]

On Fri, Apr 04, 2025 at 06:31:16PM +0200, Pankaj Raghav (Samsung) wrote:
> On Fri, Apr 04, 2025 at 08:32:15AM -0700, Luis Chamberlain wrote:
> > On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> > > From: Pankaj Raghav <p.raghav@samsung.com>
> > > 
> > > THP support for non-tmpfs filesystem has been around for some time now.
> > > Update the admin guide to reflect it.
> > > 
> > > While we are at it, move FilePmdMapped to previous paragraph for clarity,
> > > and clarify ShmemPmdMapped & ShmemHugePage.
> > > 
> > > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > > Acked-by: David Hildenbrand <david@redhat.com>
> > > ---
> > > 
> > > Changes since v2:
> > > - Address comment from Bagas Sanjaya
> > > - Squash commits and Ack from David
> > > 
> > >  Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
> > >  1 file changed, 15 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> > > index dff8d5985f0f..f8aae64e38d0 100644
> > > --- a/Documentation/admin-guide/mm/transhuge.rst
> > > +++ b/Documentation/admin-guide/mm/transhuge.rst
> > > @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
> > >  that supports the automatic promotion and demotion of page sizes and
> > >  without the shortcomings of hugetlbfs.
> > >  
> > > -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> > > -But in the future it can expand to other filesystems.
> > > +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> > > +filesystems that support large folios.
> > 
> > That seems to allude that THP can be supported on filesystems
> > that suppor large folios. I don't think we want to call that THP
> 
> But we do allocate a THP in the page cache if we support large folios.
> 
> See [1] where THP was supported through page cache. From what I
> understand, THP support was added first to the page cache and then large
> folios (orders in between) support came later.

I see, yes Do we want to clarify this further?

> For example, you can mount XFS and do a simple dd as follows:
> 
> $ dd if=/dev/random of=/mnt/hello bs=1G count=2
> 
> And if you monitor `FileHugePages` after dd is done, it would have filled
> up if THP is enabled.
> 
> > and that can confuse folks. Leaving "currently" also seems to
> > indicate that there is more work to be done for THP for filesystems
> > but that's not true as well. So how about something like:
> 
> There is more work to be done in the filesystems right? Adding large
> folio support is the prerequisite here to get the THP support.

This would be good to clarify as well.

  Luis

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[David Hildenbrand]

On 04.04.25 19:58, Luis Chamberlain wrote:
> On Fri, Apr 04, 2025 at 06:18:12PM +0200, David Hildenbrand wrote:
>> On 04.04.25 17:32, Luis Chamberlain wrote:
>>> On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
>>>> From: Pankaj Raghav <p.raghav@samsung.com>
>>>>
>>>> THP support for non-tmpfs filesystem has been around for some time now.
>>>> Update the admin guide to reflect it.
>>>>
>>>> While we are at it, move FilePmdMapped to previous paragraph for clarity,
>>>> and clarify ShmemPmdMapped & ShmemHugePage.
>>>>
>>>> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
>>>> Acked-by: David Hildenbrand <david@redhat.com>
>>>> ---
>>>>
>>>> Changes since v2:
>>>> - Address comment from Bagas Sanjaya
>>>> - Squash commits and Ack from David
>>>>
>>>>    Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
>>>>    1 file changed, 15 insertions(+), 7 deletions(-)
>>>>
>>>> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
>>>> index dff8d5985f0f..f8aae64e38d0 100644
>>>> --- a/Documentation/admin-guide/mm/transhuge.rst
>>>> +++ b/Documentation/admin-guide/mm/transhuge.rst
>>>> @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
>>>>    that supports the automatic promotion and demotion of page sizes and
>>>>    without the shortcomings of hugetlbfs.
>>>> -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
>>>> -But in the future it can expand to other filesystems.
>>>> +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
>>>> +filesystems that support large folios.
>>>
>>> That seems to allude that THP can be supported on filesystems
>>> that suppor large folios. I don't think we want to call that THP
>>> and that can confuse folks. Leaving "currently" also seems to
>>> indicate that there is more work to be done for THP for filesystems
>>> but that's not true as well. So how about something like:
>>>
>>> THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
>>> filesystem to support it. The alternative to THP for other filesystems is to
>>> support large folios and with it you can end up using huge pages
>>
>> That makes things more complicated without a good reason.
>>
>> See CONFIG_READ_ONLY_THP_FOR_FS as an early usage of the term "THP" for
>> stuff we have in the pagecache.
> 
> OK.
> 
>> (with large folios we now properly implement
>> this concept, and support more than only PMD size)
> 
> Do we really want to call large folio support on filesystems THP?

Good question.

"folio" is just the metadata we currently use to manage a chunk of 
memory, and a "large folio" is one that spans more than a single page -- 
huge page, large page, super page, ... in the past the metadata for that 
used to be a complicated piece of "compound page". In the future, we 
might call it differently (struct file_mem ?), who knows.

So "large folio" support in a fs allows for the usage of these larger 
chunks of memory (huge pages).


The filesystem tries to transparently make use of huge pages (of various 
sizes); in contrast to hugetlb, that is "transparent" in a way that 
these can be split, partially mapped, partially truncated -- just like 
for anon or shmem.

(Well, the "transparent" part was never really 100% true, but that's a 
different story)

I see more similarity with THP than with hugetlb in that sense.

In the end I don't think the terminology matters that much -- note that 
we are talking about an admin guide that resides in transhuge.rst -- as 
long as people don't confuse it with "ordinary" huge pages as in 
hugetlb. "file huge page" is rather misleading.

I'd be curious in which sense these file thingies are different than 
what we have for anon/shmem/... except that some toggles don't apply to 
them and the pagecache mostly "more transparently" makes use of them. 
(which makes them even more transparent? ;) )

-- 
Cheers,

David / dhildenb

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[David Hildenbrand]

On 04.04.25 16:06, Pankaj Raghav (Samsung) wrote:
> From: Pankaj Raghav <p.raghav@samsung.com>
> 
> THP support for non-tmpfs filesystem has been around for some time now.
> Update the admin guide to reflect it.
> 
> While we are at it, move FilePmdMapped to previous paragraph for clarity,
> and clarify ShmemPmdMapped & ShmemHugePage.
> 
> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> Acked-by: David Hildenbrand <david@redhat.com>
> ---

BTW, we should go over the whole document and bring it up to date. 
Interestingly, it starts with

"Performance critical computing applications dealing with large memory
working sets are already running on top of libhugetlbfs and in turn
hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of
using huge pages for the backing of virtual memory with huge pages
that supports the automatic promotion and demotion of page sizes and
without the shortcomings of hugetlbfs.

Currently THP only works for anonymous memory mappings and tmpfs/shmem.
But in the future it can expand to other filesystems."

-- 
Cheers,

David / dhildenb

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Daniel Gomez]

On Fri, Apr 04, 2025 at 09:07:23PM +0100, David Hildenbrand wrote:
> On 04.04.25 19:58, Luis Chamberlain wrote:
> > On Fri, Apr 04, 2025 at 06:18:12PM +0200, David Hildenbrand wrote:
> > > On 04.04.25 17:32, Luis Chamberlain wrote:
> > > > On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> > > > > From: Pankaj Raghav <p.raghav@samsung.com>
> > > > > 
> > > > > THP support for non-tmpfs filesystem has been around for some time now.
> > > > > Update the admin guide to reflect it.
> > > > > 
> > > > > While we are at it, move FilePmdMapped to previous paragraph for clarity,
> > > > > and clarify ShmemPmdMapped & ShmemHugePage.
> > > > > 
> > > > > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > > > > Acked-by: David Hildenbrand <david@redhat.com>
> > > > > ---
> > > > > 
> > > > > Changes since v2:
> > > > > - Address comment from Bagas Sanjaya
> > > > > - Squash commits and Ack from David
> > > > > 
> > > > >    Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
> > > > >    1 file changed, 15 insertions(+), 7 deletions(-)
> > > > > 
> > > > > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> > > > > index dff8d5985f0f..f8aae64e38d0 100644
> > > > > --- a/Documentation/admin-guide/mm/transhuge.rst
> > > > > +++ b/Documentation/admin-guide/mm/transhuge.rst
> > > > > @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
> > > > >    that supports the automatic promotion and demotion of page sizes and
> > > > >    without the shortcomings of hugetlbfs.
> > > > > -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> > > > > -But in the future it can expand to other filesystems.
> > > > > +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> > > > > +filesystems that support large folios.
> > > > 
> > > > That seems to allude that THP can be supported on filesystems
> > > > that suppor large folios. I don't think we want to call that THP
> > > > and that can confuse folks. Leaving "currently" also seems to
> > > > indicate that there is more work to be done for THP for filesystems
> > > > but that's not true as well. So how about something like:
> > > > 
> > > > THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
> > > > filesystem to support it. The alternative to THP for other filesystems is to
> > > > support large folios and with it you can end up using huge pages
> > > 
> > > That makes things more complicated without a good reason.
> > > 
> > > See CONFIG_READ_ONLY_THP_FOR_FS as an early usage of the term "THP" for
> > > stuff we have in the pagecache.
> > 
> > OK.
> > 
> > > (with large folios we now properly implement
> > > this concept, and support more than only PMD size)
> > 
> > Do we really want to call large folio support on filesystems THP?
> 
> Good question.
> 
> "folio" is just the metadata we currently use to manage a chunk of memory,
> and a "large folio" is one that spans more than a single page -- huge page,
> large page, super page, ... in the past the metadata for that used to be a
> complicated piece of "compound page". In the future, we might call it
> differently (struct file_mem ?), who knows.
> 
> So "large folio" support in a fs allows for the usage of these larger chunks
> of memory (huge pages).

I'm a bit confused here. I thought the term 'huge pages' referred to specific
page table level like PMD, PUD, or PGD (and not PTE). And "large folio" term can
span up to anything (currently up to PMD). Could you clarify if I'm
misunderstanding something?

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[David Hildenbrand]

On 04.04.25 21:44, Daniel Gomez wrote:
> On Fri, Apr 04, 2025 at 09:07:23PM +0100, David Hildenbrand wrote:
>> On 04.04.25 19:58, Luis Chamberlain wrote:
>>> On Fri, Apr 04, 2025 at 06:18:12PM +0200, David Hildenbrand wrote:
>>>> On 04.04.25 17:32, Luis Chamberlain wrote:
>>>>> On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
>>>>>> From: Pankaj Raghav <p.raghav@samsung.com>
>>>>>>
>>>>>> THP support for non-tmpfs filesystem has been around for some time now.
>>>>>> Update the admin guide to reflect it.
>>>>>>
>>>>>> While we are at it, move FilePmdMapped to previous paragraph for clarity,
>>>>>> and clarify ShmemPmdMapped & ShmemHugePage.
>>>>>>
>>>>>> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
>>>>>> Acked-by: David Hildenbrand <david@redhat.com>
>>>>>> ---
>>>>>>
>>>>>> Changes since v2:
>>>>>> - Address comment from Bagas Sanjaya
>>>>>> - Squash commits and Ack from David
>>>>>>
>>>>>>     Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
>>>>>>     1 file changed, 15 insertions(+), 7 deletions(-)
>>>>>>
>>>>>> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
>>>>>> index dff8d5985f0f..f8aae64e38d0 100644
>>>>>> --- a/Documentation/admin-guide/mm/transhuge.rst
>>>>>> +++ b/Documentation/admin-guide/mm/transhuge.rst
>>>>>> @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
>>>>>>     that supports the automatic promotion and demotion of page sizes and
>>>>>>     without the shortcomings of hugetlbfs.
>>>>>> -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
>>>>>> -But in the future it can expand to other filesystems.
>>>>>> +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
>>>>>> +filesystems that support large folios.
>>>>>
>>>>> That seems to allude that THP can be supported on filesystems
>>>>> that suppor large folios. I don't think we want to call that THP
>>>>> and that can confuse folks. Leaving "currently" also seems to
>>>>> indicate that there is more work to be done for THP for filesystems
>>>>> but that's not true as well. So how about something like:
>>>>>
>>>>> THP only works for anonymous memory mappings, and the tmpfs/shmem is the only
>>>>> filesystem to support it. The alternative to THP for other filesystems is to
>>>>> support large folios and with it you can end up using huge pages
>>>>
>>>> That makes things more complicated without a good reason.
>>>>
>>>> See CONFIG_READ_ONLY_THP_FOR_FS as an early usage of the term "THP" for
>>>> stuff we have in the pagecache.
>>>
>>> OK.
>>>
>>>> (with large folios we now properly implement
>>>> this concept, and support more than only PMD size)
>>>
>>> Do we really want to call large folio support on filesystems THP?
>>
>> Good question.
>>
>> "folio" is just the metadata we currently use to manage a chunk of memory,
>> and a "large folio" is one that spans more than a single page -- huge page,
>> large page, super page, ... in the past the metadata for that used to be a
>> complicated piece of "compound page". In the future, we might call it
>> differently (struct file_mem ?), who knows.
>>
>> So "large folio" support in a fs allows for the usage of these larger chunks
>> of memory (huge pages).
> 
> I'm a bit confused here. I thought the term 'huge pages' referred to specific
> page table level like PMD, PUD, or PGD (and not PTE). And "large folio" term can
> span up to anything (currently up to PMD). Could you clarify if I'm
> misunderstanding something?

The transhuge.rst is still a bit inconsistent and confusing, but we 
started generalizing the concept to special-case "Traditional THPs" to 
be PMD-sized THPs. See the description about mTHP in the document.

If a small folio represents a single page, then a large folio represents 
  multiple pages (large/huge/super page). But again, "folio" is  the 
metadata we use to manage this chunk of memory. Just like we didn't call 
a "huge page" a "compound page" back in the days; the "compound page" 
was a way to manage the metadata assigned to a "larger chunk of memory 
covering multiple base pages" as one unit.

I once had a longer writeup about all that (including how hugetlb 
supports different huge page sizes, how freebsd calls these thing super 
pages etc).

This topic seems to pop every 6 months, likely we really should do a 
better job at documenting all that.

-- 
Cheers,

David / dhildenb

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 2619 bytes --]

On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> index dff8d5985f0f..f8aae64e38d0 100644
> --- a/Documentation/admin-guide/mm/transhuge.rst
> +++ b/Documentation/admin-guide/mm/transhuge.rst
> @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
>  that supports the automatic promotion and demotion of page sizes and
>  without the shortcomings of hugetlbfs.
>  
> -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> -But in the future it can expand to other filesystems.
> +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> +filesystems that support large folios.
>  
>  .. note::
>     in the examples below we presume that the basic page size is 4K and
> @@ -463,11 +463,19 @@ fields for each mapping. (Note that AnonHugePages only applies to traditional
>  PMD-sized THP for historical reasons and should have been called
>  AnonHugePmdMapped).
>  
> -The number of file transparent huge pages mapped to userspace is available
> -by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``.
> -To identify what applications are mapping file transparent huge pages, it
> -is necessary to read ``/proc/PID/smaps`` and count the FilePmdMapped fields
> -for each mapping.
> +The number of PMD-sized transparent huge pages currently used by
> +filesystem data (page cache) is available by reading the FileHugePages field
> +in ``/proc/meminfo``. The number of these huge pages that are mapped to userspace
> +is available by reading FilePmdMapped field in ``proc/meminfo``. To identify
> +what applications are mapping these huge pages, it is necessary to read
> +``/proc/PID/smaps`` and count the FilePmdMapped fields for each mapping.
> +
> +In similar fashion, the number of PMD-sized transparent huge pages currently
> +used by tmpfs/shmem is available by reading the ShmemHugePages field
> +in ``/proc/meminfo``. The number of these huge pages that are mapped to userspace
> +is available by reading ShmemPmdMapped field in ``proc/meminfo``. To identify
> +what applications are mapping these huge pages, it is necessary to read
> +``/proc/PID/smaps`` and count the ShmemPmdMapped fields for each mapping.
>  
>  Note that reading the smaps file is expensive and reading it
>  frequently will incur overhead.
> 

Looks good, thanks!

Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Pankaj Raghav (Samsung)]

On Fri, Apr 04, 2025 at 09:16:10PM +0200, David Hildenbrand wrote:
> On 04.04.25 16:06, Pankaj Raghav (Samsung) wrote:
> > From: Pankaj Raghav <p.raghav@samsung.com>
> > 
> > THP support for non-tmpfs filesystem has been around for some time now.
> > Update the admin guide to reflect it.
> > 
> > While we are at it, move FilePmdMapped to previous paragraph for clarity,
> > and clarify ShmemPmdMapped & ShmemHugePage.
> > 
> > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > Acked-by: David Hildenbrand <david@redhat.com>
> > ---
> 
> BTW, we should go over the whole document and bring it up to date.
> Interestingly, it starts with
> 
> "Performance critical computing applications dealing with large memory
> working sets are already running on top of libhugetlbfs and in turn
> hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of
> using huge pages for the backing of virtual memory with huge pages
> that supports the automatic promotion and demotion of page sizes and
> without the shortcomings of hugetlbfs.
> 
> Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> But in the future it can expand to other filesystems."

As a part of this patch, I do change 2nd paragraph:

-Currently THP only works for anonymous memory mappings and tmpfs/shmem.
-But in the future it can expand to other filesystems.
+Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
+filesystems that support large folios.

Do we need to add more changes to the first paragraph as well?

--
Pankaj

Re: [PATCH v3] docs: update THP admin guide about non-tmpfs filesystem support

[Daniel Gomez]

On Fri, Apr 04, 2025 at 11:02:46AM +0100, Luis Chamberlain wrote:
> On Fri, Apr 04, 2025 at 06:31:16PM +0200, Pankaj Raghav (Samsung) wrote:
> > On Fri, Apr 04, 2025 at 08:32:15AM -0700, Luis Chamberlain wrote:
> > > On Fri, Apr 04, 2025 at 04:06:57PM +0200, Pankaj Raghav (Samsung) wrote:
> > > > From: Pankaj Raghav <p.raghav@samsung.com>
> > > > 
> > > > THP support for non-tmpfs filesystem has been around for some time now.
> > > > Update the admin guide to reflect it.
> > > > 
> > > > While we are at it, move FilePmdMapped to previous paragraph for clarity,
> > > > and clarify ShmemPmdMapped & ShmemHugePage.
> > > > 
> > > > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > > > Acked-by: David Hildenbrand <david@redhat.com>
> > > > ---
> > > > 
> > > > Changes since v2:
> > > > - Address comment from Bagas Sanjaya
> > > > - Squash commits and Ack from David
> > > > 
> > > >  Documentation/admin-guide/mm/transhuge.rst | 22 +++++++++++++++-------
> > > >  1 file changed, 15 insertions(+), 7 deletions(-)
> > > > 
> > > > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> > > > index dff8d5985f0f..f8aae64e38d0 100644
> > > > --- a/Documentation/admin-guide/mm/transhuge.rst
> > > > +++ b/Documentation/admin-guide/mm/transhuge.rst
> > > > @@ -12,8 +12,8 @@ using huge pages for the backing of virtual memory with huge pages
> > > >  that supports the automatic promotion and demotion of page sizes and
> > > >  without the shortcomings of hugetlbfs.
> > > >  
> > > > -Currently THP only works for anonymous memory mappings and tmpfs/shmem.
> > > > -But in the future it can expand to other filesystems.
> > > > +Currently, THP only works for anonymous memory mappings, tmpfs/shmem and
> > > > +filesystems that support large folios.
> > > 
> > > That seems to allude that THP can be supported on filesystems
> > > that suppor large folios. I don't think we want to call that THP
> > 
> > But we do allocate a THP in the page cache if we support large folios.
> > 
> > See [1] where THP was supported through page cache. From what I
> > understand, THP support was added first to the page cache and then large
> > folios (orders in between) support came later.
> 
> I see, yes Do we want to clarify this further?

According to this thread [1], large folios currently depends on
CONFIG_TRANSPARENT_HUGEPAGE, but that will be removed eventually.

https://lore.kernel.org/all/ZPINmXyTgy2wqLqr@casper.infradead.org/

I agree this needs to be clarified. THP for anonymous memory and tmpfs/shmem is
not the same for filesystems that support large folios.