[LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Matthew Wilcox]

Our documentation of the current page flags is ... not great.  I think
I can improve it for the page cache side of things; I understand the
meanings of locked, writeback, uptodate, dirty, head, waiters, slab,
mlocked, mappedtodisk, error, hwpoison, readahead, anon_exclusive,
has_hwpoisoned, hugetlb and large_remappable.

Where I'm a lot more shaky is the meaning of the more "real MM" flags,
like active, referenced, lru, workingset, reserved, reclaim, swapbacked,
unevictable, young, idle, swapcache, isolated, and reported.

Perhaps we could have an MM session where we try to explain slowly and
carefully to each other what all these flags actually mean, talk about
what combinations of them make sense, how we might eliminate some of
them to make more space in the flags word, and what all this looks like
in a memdesc world.

And maybe we can get some documentation written about it!  Not trying
to nerd snipe Jon into attending this session, but if he did ...

[thanks to Amir for reminding me that I meant to propose this topic]

[LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Matthew Wilcox]

LFrom willy@infradead.org Fri Feb  2 16:28:25 2024
Date: Fri, 2 Feb 2024 16:28:25 +0000
From: Matthew Wilcox <willy@infradead.org>
To: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Cc: linux-kernel@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [resend, PATCH v1 1/1] logic_pio: Use RESOURCE_SIZE_MAX
 definition
References: <20231016132611.1201402-1-andriy.shevchenko@linux.intel.com>
 <Zb0LzpBkE71wWyqO@smile.fi.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Zb0LzpBkE71wWyqO@smile.fi.intel.com>
X-Mutt-References: <Zb0LzpBkE71wWyqO@smile.fi.intel.com>
X-Mutt-Fcc: ~/sent
Status: RO
Content-Length: 237
Lines: 7

On Fri, Feb 02, 2024 at 05:35:42PM +0200, Andy Shevchenko wrote:
> On Mon, Oct 16, 2023 at 04:26:11PM +0300, Andy Shevchenko wrote:
> > Use a predefined limit instead of hardcoding it.
> 
> Can we apply this one?

Why are you asking me?

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Mike Rapoport]

On Mon, Jan 29, 2024 at 04:32:03AM +0000, Matthew Wilcox wrote:
> Our documentation of the current page flags is ... not great.  I think
> I can improve it for the page cache side of things; I understand the
> meanings of locked, writeback, uptodate, dirty, head, waiters, slab,
> mlocked, mappedtodisk, error, hwpoison, readahead, anon_exclusive,
> has_hwpoisoned, hugetlb and large_remappable.
> 
> Where I'm a lot more shaky is the meaning of the more "real MM" flags,
> like active, referenced, lru, workingset, reserved, reclaim, swapbacked,
> unevictable, young, idle, swapcache, isolated, and reported.
> 
> Perhaps we could have an MM session where we try to explain slowly and
> carefully to each other what all these flags actually mean, talk about
> what combinations of them make sense, how we might eliminate some of
> them to make more space in the flags word, and what all this looks like
> in a memdesc world.
> 
> And maybe we can get some documentation written about it!  Not trying
> to nerd snipe Jon into attending this session, but if he did ...

I suspect Jon will be there anyway, but not sure he'd be willing to do the
writing :)

I was going to propose the "mm docs" session again, but this one seems more
useful than talking yet again about how hard it is to get MM documentation
done.

And I can take on myself putting the explanations from this session into
writing.
 
> [thanks to Amir for reminding me that I meant to propose this topic]
> 

-- 
Sincerely yours,
Mike.

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Matthew Wilcox]

On Sun, Feb 04, 2024 at 11:39:33AM +0100, Mike Rapoport wrote:
> On Mon, Jan 29, 2024 at 04:32:03AM +0000, Matthew Wilcox wrote:
> > Our documentation of the current page flags is ... not great.  I think
> > I can improve it for the page cache side of things; I understand the
> > meanings of locked, writeback, uptodate, dirty, head, waiters, slab,
> > mlocked, mappedtodisk, error, hwpoison, readahead, anon_exclusive,
> > has_hwpoisoned, hugetlb and large_remappable.
> > 
> > Where I'm a lot more shaky is the meaning of the more "real MM" flags,
> > like active, referenced, lru, workingset, reserved, reclaim, swapbacked,
> > unevictable, young, idle, swapcache, isolated, and reported.
> > 
> > Perhaps we could have an MM session where we try to explain slowly and
> > carefully to each other what all these flags actually mean, talk about
> > what combinations of them make sense, how we might eliminate some of
> > them to make more space in the flags word, and what all this looks like
> > in a memdesc world.
> > 
> > And maybe we can get some documentation written about it!  Not trying
> > to nerd snipe Jon into attending this session, but if he did ...
> 
> I suspect Jon will be there anyway, but not sure he'd be willing to do the
> writing :)
> 
> I was going to propose the "mm docs" session again, but this one seems more
> useful than talking yet again about how hard it is to get MM documentation
> done.

I'm doing my best to write documentation as I go.  I think we're a bit
better off than we were last year.  Do we have scripts to tell us which
public functions (ie EXPORT_SYMBOL and static inline functions in header
files) have kernel-doc?  And could we run them against kernels from, say,
April 2023, 2022, 2021, 2020, 2019 (and in two months against April 2024)
and see how we're doing in terms of percentage undocumented functions?

There's also the problem of getting long-form documentation done.
But I think that's a different problem from getting kernel-doc written.
Looking at the 55 commits in the last year to Documentation/mm, we seems
to be doing a pretty good job of keeping the documentation we have up
to date.  Just not a great job of adding new documentation.

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Mike Rapoport]

On Sun, Feb 04, 2024 at 09:34:01PM +0000, Matthew Wilcox wrote:
> On Sun, Feb 04, 2024 at 11:39:33AM +0100, Mike Rapoport wrote:
> > On Mon, Jan 29, 2024 at 04:32:03AM +0000, Matthew Wilcox wrote:
> > > Our documentation of the current page flags is ... not great.  I think
> > > I can improve it for the page cache side of things; I understand the
> > > meanings of locked, writeback, uptodate, dirty, head, waiters, slab,
> > > mlocked, mappedtodisk, error, hwpoison, readahead, anon_exclusive,
> > > has_hwpoisoned, hugetlb and large_remappable.
> > > 
> > > Where I'm a lot more shaky is the meaning of the more "real MM" flags,
> > > like active, referenced, lru, workingset, reserved, reclaim, swapbacked,
> > > unevictable, young, idle, swapcache, isolated, and reported.
> > > 
> > > Perhaps we could have an MM session where we try to explain slowly and
> > > carefully to each other what all these flags actually mean, talk about
> > > what combinations of them make sense, how we might eliminate some of
> > > them to make more space in the flags word, and what all this looks like
> > > in a memdesc world.
> > > 
> > > And maybe we can get some documentation written about it!  Not trying
> > > to nerd snipe Jon into attending this session, but if he did ...
> > 
> > I suspect Jon will be there anyway, but not sure he'd be willing to do the
> > writing :)
> > 
> > I was going to propose the "mm docs" session again, but this one seems more
> > useful than talking yet again about how hard it is to get MM documentation
> > done.
> 
> I'm doing my best to write documentation as I go.  I think we're a bit
> better off than we were last year.  Do we have scripts to tell us which
> public functions (ie EXPORT_SYMBOL and static inline functions in header
> files) have kernel-doc?  And could we run them against kernels from, say,
> April 2023, 2022, 2021, 2020, 2019 (and in two months against April 2024)
> and see how we're doing in terms of percentage undocumented functions?

We didn't have such script, but it was easy to compare "grep
EXPORT_SYMBOL\|static inline" with ".. c:function" in kernel-doc.
We do improve slowly, but we are still below 50% with kernel-doc for
EXPORT_SYMBOL functions and slightly above 10% for static inlines.

Although with static inlines it's quite possible that the percentage of
actual public API documentation is higher because some of the functions in
inlcude/linux/ are only used inside mm.

There are also APIs that are not EXPORT_SYMBOL, but I didn't find an easy
way to check how well there are documented.

EXPORT_SYMBOL
version     	funcs	docs	percent
v5.0        	514	177	34
v5.6        	538	208	38
v5.12       	550	209	38
v5.17       	580	228	39
v6.3        	580	235	40
v6.8-rc1    	565	238	42

static inline
version     	funcs	docs	percent
v5.0        	581	33	5
v5.6        	596	41	6
v5.12       	629	42	6
v5.17       	746	74	9
v6.3        	867	95	10
v6.8-rc1    	944	116	12

 
> There's also the problem of getting long-form documentation done.
> But I think that's a different problem from getting kernel-doc written.
> Looking at the 55 commits in the last year to Documentation/mm, we seems
> to be doing a pretty good job of keeping the documentation we have up
> to date.  Just not a great job of adding new documentation.

I agree that long-form documentation is a different problem from getting
kernel-doc written and we are not doing a great job in writing new
documentation.

-- 
Sincerely yours,
Mike.

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Matthew Wilcox]

On Wed, Feb 07, 2024 at 05:51:44PM +0200, Mike Rapoport wrote:
> On Sun, Feb 04, 2024 at 09:34:01PM +0000, Matthew Wilcox wrote:
> > I'm doing my best to write documentation as I go.  I think we're a bit
> > better off than we were last year.  Do we have scripts to tell us which
> > public functions (ie EXPORT_SYMBOL and static inline functions in header
> > files) have kernel-doc?  And could we run them against kernels from, say,
> > April 2023, 2022, 2021, 2020, 2019 (and in two months against April 2024)
> > and see how we're doing in terms of percentage undocumented functions?
> 
> We didn't have such script, but it was easy to compare "grep
> EXPORT_SYMBOL\|static inline" with ".. c:function" in kernel-doc.
> We do improve slowly, but we are still below 50% with kernel-doc for
> EXPORT_SYMBOL functions and slightly above 10% for static inlines.

Thanks for doing this!  Data is good ;-)

I just came across an interesting example of a function which I believe
should NOT have kernel-doc.  But it should have documentation for why it
doesn't have kernel-doc!  Any thoughts about how we might accomplish that?

The example is filemap_range_has_writeback().  It's EXPORT_SYMBOL_GPL()
and it's a helper function for filemap_range_needs_writeback().
filemap_range_needs_writeback() has kernel-doc, but nobody should be
calling filemap_range_has_writeback() directly, so it shouldn't even
exist in the htmldocs.  But we should have a comment on it saying
"Use filemap_range_needs_writeback(), don't use this", in case anyone
discovers it.  And the existance of that comment should be enough to
tell our tools to not flag this as a function that needs kernel-doc.

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[NeilBrown]

On Tue, 20 Feb 2024, Matthew Wilcox wrote:
> On Wed, Feb 07, 2024 at 05:51:44PM +0200, Mike Rapoport wrote:
> > On Sun, Feb 04, 2024 at 09:34:01PM +0000, Matthew Wilcox wrote:
> > > I'm doing my best to write documentation as I go.  I think we're a bit
> > > better off than we were last year.  Do we have scripts to tell us which
> > > public functions (ie EXPORT_SYMBOL and static inline functions in header
> > > files) have kernel-doc?  And could we run them against kernels from, say,
> > > April 2023, 2022, 2021, 2020, 2019 (and in two months against April 2024)
> > > and see how we're doing in terms of percentage undocumented functions?
> > 
> > We didn't have such script, but it was easy to compare "grep
> > EXPORT_SYMBOL\|static inline" with ".. c:function" in kernel-doc.
> > We do improve slowly, but we are still below 50% with kernel-doc for
> > EXPORT_SYMBOL functions and slightly above 10% for static inlines.
> 
> Thanks for doing this!  Data is good ;-)
> 
> I just came across an interesting example of a function which I believe
> should NOT have kernel-doc.  But it should have documentation for why it
> doesn't have kernel-doc!  Any thoughts about how we might accomplish that?
> 
> The example is filemap_range_has_writeback().  It's EXPORT_SYMBOL_GPL()
> and it's a helper function for filemap_range_needs_writeback().
> filemap_range_needs_writeback() has kernel-doc, but nobody should be
> calling filemap_range_has_writeback() directly, so it shouldn't even
> exist in the htmldocs.  But we should have a comment on it saying
> "Use filemap_range_needs_writeback(), don't use this", in case anyone
> discovers it.  And the existance of that comment should be enough to
> tell our tools to not flag this as a function that needs kernel-doc.
> 

Don't we use a __prefix for internal stuff that shouldn't be used?

NeilBrown

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Matthew Wilcox]

On Tue, Feb 20, 2024 at 09:45:36AM +1100, NeilBrown wrote:
> On Tue, 20 Feb 2024, Matthew Wilcox wrote:
> > The example is filemap_range_has_writeback().  It's EXPORT_SYMBOL_GPL()
> > and it's a helper function for filemap_range_needs_writeback().
> > filemap_range_needs_writeback() has kernel-doc, but nobody should be
> > calling filemap_range_has_writeback() directly, so it shouldn't even
> > exist in the htmldocs.  But we should have a comment on it saying
> > "Use filemap_range_needs_writeback(), don't use this", in case anyone
> > discovers it.  And the existance of that comment should be enough to
> > tell our tools to not flag this as a function that needs kernel-doc.
> > 
> 
> Don't we use a __prefix for internal stuff that shouldn't be used?

No?  Or if we do, we are inconsistent with that convention.  Let's
consider some examples.

__SetPageReferenced -- non-atomic version of SetPageReferenced.
Akin to __set_bit.

__filemap_fdatawrite_range() -- like filemap_fdatawrite_range but
allows the specification of sync_mode

__page_cache_alloc() -- like page_cache_alloc() but takes the gfp mask
directly instead of inferring it from mapping_gfp_mask()

__folio_lock() -- This does fit the "don't call this pattern"!

__set_page_dirty() -- Like set_page_dirty() but allows warn to be
specified.

__filemap_remove_folio() -- Like filemap_remove_folio() but allows it
to be replaced with a shadow entry.

__readahead_folio() -- Another internal one

I mostly confined myself to pagemap.h for this survey, but if you've
conducted a different survey that shows your assertion is generally true
and I've hit on the exceptions to the rule ... ?

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[NeilBrown]

On Tue, 20 Feb 2024, Matthew Wilcox wrote:
> On Tue, Feb 20, 2024 at 09:45:36AM +1100, NeilBrown wrote:
> > On Tue, 20 Feb 2024, Matthew Wilcox wrote:
> > > The example is filemap_range_has_writeback().  It's EXPORT_SYMBOL_GPL()
> > > and it's a helper function for filemap_range_needs_writeback().
> > > filemap_range_needs_writeback() has kernel-doc, but nobody should be
> > > calling filemap_range_has_writeback() directly, so it shouldn't even
> > > exist in the htmldocs.  But we should have a comment on it saying
> > > "Use filemap_range_needs_writeback(), don't use this", in case anyone
> > > discovers it.  And the existance of that comment should be enough to
> > > tell our tools to not flag this as a function that needs kernel-doc.
> > > 
> > 
> > Don't we use a __prefix for internal stuff that shouldn't be used?
> 
> No?  Or if we do, we are inconsistent with that convention.  Let's
> consider some examples.
> 
> __SetPageReferenced -- non-atomic version of SetPageReferenced.
> Akin to __set_bit.
> 
> __filemap_fdatawrite_range() -- like filemap_fdatawrite_range but
> allows the specification of sync_mode
> 
> __page_cache_alloc() -- like page_cache_alloc() but takes the gfp mask
> directly instead of inferring it from mapping_gfp_mask()
> 
> __folio_lock() -- This does fit the "don't call this pattern"!
> 
> __set_page_dirty() -- Like set_page_dirty() but allows warn to be
> specified.
> 
> __filemap_remove_folio() -- Like filemap_remove_folio() but allows it
> to be replaced with a shadow entry.
> 
> __readahead_folio() -- Another internal one
> 
> I mostly confined myself to pagemap.h for this survey, but if you've
> conducted a different survey that shows your assertion is generally true
> and I've hit on the exceptions to the rule ... ?
> 

Yes, __ is used for other things too.
It would be nice to have some consistency with naming, but probably
impossible.

And with 1074 functions named __foo having kernel doc already, it is too
late to close that gate.
:-(

Thanks,
NeilBrown

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Hannes Reinecke]

On 2/19/24 21:13, Matthew Wilcox wrote:
> On Wed, Feb 07, 2024 at 05:51:44PM +0200, Mike Rapoport wrote:
>> On Sun, Feb 04, 2024 at 09:34:01PM +0000, Matthew Wilcox wrote:
>>> I'm doing my best to write documentation as I go.  I think we're a bit
>>> better off than we were last year.  Do we have scripts to tell us which
>>> public functions (ie EXPORT_SYMBOL and static inline functions in header
>>> files) have kernel-doc?  And could we run them against kernels from, say,
>>> April 2023, 2022, 2021, 2020, 2019 (and in two months against April 2024)
>>> and see how we're doing in terms of percentage undocumented functions?
>>
>> We didn't have such script, but it was easy to compare "grep
>> EXPORT_SYMBOL\|static inline" with ".. c:function" in kernel-doc.
>> We do improve slowly, but we are still below 50% with kernel-doc for
>> EXPORT_SYMBOL functions and slightly above 10% for static inlines.
> 
> Thanks for doing this!  Data is good ;-)
> 
> I just came across an interesting example of a function which I believe
> should NOT have kernel-doc.  But it should have documentation for why it
> doesn't have kernel-doc!  Any thoughts about how we might accomplish that?
> 
> The example is filemap_range_has_writeback().  It's EXPORT_SYMBOL_GPL()
> and it's a helper function for filemap_range_needs_writeback().
> filemap_range_needs_writeback() has kernel-doc, but nobody should be
> calling filemap_range_has_writeback() directly, so it shouldn't even
> exist in the htmldocs.  But we should have a comment on it saying
> "Use filemap_range_needs_writeback(), don't use this", in case anyone
> discovers it.  And the existance of that comment should be enough to
> tell our tools to not flag this as a function that needs kernel-doc.
> 
> 
Or, indeed, coming up with a method of signalling "this is an internal
function for a specific need, don't use otherwise".

EXPORT_SYMBOL_INTERNAL?

I would love to have it; it would solve _so_ many problems we're having
wrt kABI...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Muhammad Usama Anjum]

On Mon, 2024-01-29 at 04:32 +0000, Matthew Wilcox wrote:
> Our documentation of the current page flags is ... not great.  I think
> I can improve it for the page cache side of things; I understand the
> meanings of locked, writeback, uptodate, dirty, head, waiters, slab,
> mlocked, mappedtodisk, error, hwpoison, readahead, anon_exclusive,
> has_hwpoisoned, hugetlb and large_remappable.
> 
> Where I'm a lot more shaky is the meaning of the more "real MM" flags,
> like active, referenced, lru, workingset, reserved, reclaim, swapbacked,
> unevictable, young, idle, swapcache, isolated, and reported.
> 
> Perhaps we could have an MM session where we try to explain slowly and
> carefully to each other what all these flags actually mean, talk about
> what combinations of them make sense, how we might eliminate some of
> them to make more space in the flags word, and what all this looks like
> in a memdesc world.
> 
> And maybe we can get some documentation written about it!  Not trying
> to nerd snipe Jon into attending this session, but if he did ...
This is great idea. Instead of having a session to write
documentation, we can have a session which would be documentation
itself even if nobody translates it to text.

> 
> [thanks to Amir for reminding me that I meant to propose this topic]
> 

Re: [LSF/MM/BPF TOPIC] Reclaiming & documenting page flags

[Navid]

On Mon, 2024-01-29 at 04:32 +0000, Matthew Wilcox wrote:
> Our documentation of the current page flags is ... not great.  I think
> I can improve it for the page cache side of things; I understand the
> meanings of locked, writeback, uptodate, dirty, head, waiters, slab,
> mlocked, mappedtodisk, error, hwpoison, readahead, anon_exclusive,
> has_hwpoisoned, hugetlb and large_remappable.
> 
> Where I'm a lot more shaky is the meaning of the more "real MM" flags,
> like active, referenced, lru, workingset, reserved, reclaim, swapbacked,
> unevictable, young, idle, swapcache, isolated, and reported.
> 
> Perhaps we could have an MM session where we try to explain slowly and
> carefully to each other what all these flags actually mean, talk about
> what combinations of them make sense, how we might eliminate some of
> them to make more space in the flags word, and what all this looks like
> in a memdesc world.
> 
> And maybe we can get some documentation written about it!  Not trying
> to nerd snipe Jon into attending this session, but if he did ...
> 
> [thanks to Amir for reminding me that I meant to propose this topic]
> 

On the "Reclaiming" part of this thread, we might consider this:

Optimizing Page Flags: Reclaiming Bits in page->flags via folio->lru

Limited bit space in the Linux kernel's page->flags field, especially on 32-bit
architectures, is a source of challenge [1]. This proposal aims to free up bits
by relocating flags like PG_active and PG_unevictable to the lower bits of
folio->lru as they are always unset. It helps with encoding zone, numa node,
and sparsemem section [2].

Proposed Process:

Candidate Evaluation: Assess flags for relocation suitability based on usage,
dependencies, and functional impact.
Impact Assessment: Evaluate the impact on kernel code to ensure correct behavior
and compatibility.
Relocation Implementation: Modify code to read/write flags from folio->lru and
adjust related macros/functions.
Thoroughly test changes.

[1] https://lwn.net/Articles/335768/
[2] https://blogs.oracle.com/linux/post/struct-page-the-linux-physical-page-frame-data-structure