Re: [PATCH RFC 04/37] mm: Add MIGRATE_METADATA allocation policy

[Alexandru Elisei <alexandru.elisei@arm.com>]

Hello,

On Thu, Oct 12, 2023 at 10:28:24AM +0900, Hyesoo Yu wrote:
> On Wed, Aug 23, 2023 at 02:13:17PM +0100, Alexandru Elisei wrote:
> > Some architectures implement hardware memory coloring to catch incorrect
> > usage of memory allocation. One such architecture is arm64, which calls its
> > hardware implementation Memory Tagging Extension.
> > 
> > So far, the memory which stores the metadata has been configured by
> > firmware and hidden from Linux. For arm64, it is impossible to to have the
> > entire system RAM allocated with metadata because executable memory cannot
> > be tagged. Furthermore, in practice, only a chunk of all the memory that
> > can have tags is actually used as tagged. which leaves a portion of
> > metadata memory unused. As such, it would be beneficial to use this memory,
> > which so far has been unaccessible to Linux, to service allocation
> > requests. To prepare for exposing this metadata memory a new migratetype is
> > being added to the page allocator, called MIGRATE_METADATA.
> > 
> > One important aspect is that for arm64 the memory that stores metadata
> > cannot have metadata associated with it, it can only be used to store
> > metadata for other pages. This means that the page allocator will *not*
> > allocate from this migratetype if at least one of the following is true:
> > 
> > - The allocation also needs metadata to be allocated.
> > - The allocation isn't movable. A metadata page storing data must be
> >   able to be migrated at any given time so it can be repurposed to store
> >   metadata.
> > 
> > Both cases are specific to arm64's implementation of memory metadata.
> > 
> > For now, metadata storage pages management is disabled, and it will be
> > enabled once the architecture-specific handling is added.
> > 
> > Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
> > ---
> > [..]
> > @@ -2144,6 +2156,15 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype,
> >  		if (alloc_flags & ALLOC_CMA)
> >  			page = __rmqueue_cma_fallback(zone, order);
> >  
> > +		/*
> > +		 * Allocate data pages from MIGRATE_METADATA only if the regular
> > +		 * allocation path fails to increase the chance that the
> > +		 * metadata page is available when the associated data page
> > +		 * needs it.
> > +		 */
> > +		if (!page && (alloc_flags & ALLOC_FROM_METADATA))
> > +			page = __rmqueue_metadata_fallback(zone, order);
> > +
> 
> Hi!
> 
> I guess it would cause non-movable page starving issue as CMA.

I don't understand what you mean by "non-movable page starving issue as
CMA". Would you care to elaborate?

> The metadata pages cannot be used for non-movable allocations.
> Metadata pages are utilized poorly, non-movable allocations may end up
> getting starved if all regular movable pages are allocated and the only
> pages left are metadata. If the system has a lot of CMA pages, then
> this problem would become more bad. I think it would be better to make
> use of it in places where performance is not critical, including some
> GFP_METADATA ?

GFP_METADATA pages must be used only for movable allocations. The kernel
must be able to migrate GFP_METADATA pages (if they have been allocated)
when they are reserved to serve as tag storage for a newly allocated tagged
page.

If you are referring to the fact that GFP_METADATA pages are allocated only
when there are no more free pages in the zone, then yes, I can understand
that that might be an issue. However, it's worth keeping in mind that if a
GFP_METADATA page is in use when it needs to be repurposed to serve as tag
storage, its contents must be migrated first, and this is obviously slow.

To put it another way, the more eager the page allocator is to allocate
from GFP_METADATA, the slower it will be to allocate tagged pages because
reserving the corresponding tag storage will be slow due to migration.

Before making a decision, I think it would be very helpful to run
performance tests with different allocation policies for GFP_METADATA. But I
would say that it's a bit premature for that, and I think it would be best
to wait until the series stabilizes.

And thank you for the feedback!

Alex

> 
> Thanks,
> Hyesoo Yu.