Re: [PATCH RFC 00/37] Add support for arm64 MTE dynamic tag storage reuse

[David Hildenbrand <david@redhat.com>]

On 11.09.23 13:52, Catalin Marinas wrote:
> On Wed, Sep 06, 2023 at 12:23:21PM +0100, Alexandru Elisei wrote:
>> On Thu, Aug 24, 2023 at 04:24:30PM +0100, Catalin Marinas wrote:
>>> On Thu, Aug 24, 2023 at 01:25:41PM +0200, David Hildenbrand wrote:
>>>> On 24.08.23 13:06, David Hildenbrand wrote:
>>>>> Regarding one complication: "The kernel needs to know where to allocate
>>>>> a PROT_MTE page from or migrate a current page if it becomes PROT_MTE
>>>>> (mprotect()) and the range it is in does not support tagging.",
>>>>> simplified handling would be if it's in a MIGRATE_CMA pageblock, it
>>>>> doesn't support tagging. You have to migrate to a !CMA page (for
>>>>> example, not specifying GFP_MOVABLE as a quick way to achieve that).
>>>>
>>>> Okay, I now realize that this patch set effectively duplicates some CMA
>>>> behavior using a new migrate-type.
> [...]
>> I considered mixing the tag storage memory memory with normal memory and
>> adding it to MIGRATE_CMA. But since tag storage memory cannot be tagged,
>> this means that it's not enough anymore to have a __GFP_MOVABLE allocation
>> request to use MIGRATE_CMA.
>>
>> I considered two solutions to this problem:
>>
>> 1. Only allocate from MIGRATE_CMA is the requested memory is not tagged =>
>> this effectively means transforming all memory from MIGRATE_CMA into the
>> MIGRATE_METADATA migratetype that the series introduces. Not very
>> appealing, because that means treating normal memory that is also on the
>> MIGRATE_CMA lists as tagged memory.
> 
> That's indeed not ideal. We could try this if it makes the patches
> significantly simpler, though I'm not so sure.
> 
> Allocating metadata is the easier part as we know the correspondence
> from the tagged pages (32 PROT_MTE page) to the metadata page (1 tag
> storage page), so alloc_contig_range() does this for us. Just adding it
> to the CMA range is sufficient.
> 
> However, making sure that we don't allocate PROT_MTE pages from the
> metadata range is what led us to another migrate type. I guess we could
> achieve something similar with a new zone or a CPU-less NUMA node,

Ideally, no significant core-mm changes to optimize for an architecture 
oddity. That implies, no new zones and no new migratetypes -- unless it 
is unavoidable and you are confident that you can convince core-MM 
people that the use case (giving back 3% of system RAM at max in some 
setups) is worth the trouble.

I also had CPU-less NUMA nodes in mind when thinking about that, but not 
sure how easy it would be to integrate it. If the tag memory has 
actually different performance characteristics as well, a NUMA node 
would be the right choice.

If we could find some way to easily support this either via CMA or 
CPU-less NUMA nodes, that would be much preferable; even if we cannot 
cover each and every future use case right now. I expect some issues 
with CXL+MTE either way , but are happy to be taught otherwise :)


Another thought I had was adding something like CMA memory 
characteristics. Like, asking if a given CMA area/page supports tagging 
(i.e., flag for the CMA area set?)?

When you need memory that supports tagging and have a page that does not 
support tagging (CMA && taggable), simply migrate to !MOVABLE memory 
(eventually we could also try adding !CMA).

Was that discussed and what would be the challenges with that? Page 
migration due to compaction comes to mind, but it might also be easy to 
handle if we can just avoid CMA memory for that.

> though the latter is not guaranteed not to allocate memory from the
> range, only make it less likely. Both these options are less flexible in
> terms of size/alignment/placement.
> 
> Maybe as a quick hack - only allow PROT_MTE from ZONE_NORMAL and
> configure the metadata range in ZONE_MOVABLE but at some point I'd
> expect some CXL-attached memory to support MTE with additional carveout
> reserved.

I have no idea how we could possibly cleanly support memory hotplug in 
virtual environments (virtual DIMMs, virtio-mem) with MTE. In contrast 
to s390x storage keys, the approach that arm64 with MTE took here 
(exposing tag memory to the VM) makes it rather hard and complicated.

-- 
Cheers,

David / dhildenb