From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA92EEE57DF for ; Mon, 11 Sep 2023 11:52:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41ABE6B027A; Mon, 11 Sep 2023 07:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A4CD6B027E; Mon, 11 Sep 2023 07:52:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21C686B027C; Mon, 11 Sep 2023 07:52:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 028566B0279 for ; Mon, 11 Sep 2023 07:52:42 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CC406B3AE8 for ; Mon, 11 Sep 2023 11:52:41 +0000 (UTC) X-FDA: 81224154522.21.F20C7F8 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 276B61C0013 for ; Mon, 11 Sep 2023 11:52:38 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694433160; a=rsa-sha256; cv=none; b=SyQ6Aw7DpDGN3uR3z13caCEDFu5Tc3DRcKBfDNlSiBgwygRpgL9b67K/sCfnZExDjtbWwV /zXwy29YqgK9cC3kGVuF5LBKVNwgdDkq1Nc98PEk72/+TTRDWfK3Slo3+WEYS4j9d72+B3 LN+ZTQqbuVY15oJpcFdFzxO85enPgQY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of cmarinas@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694433160; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KUVQI21Km4DpQT/a+VRpz70/BPsjp+d4yBNqNI+7o5w=; b=LbWnyxg8y9mne+Pn9Gg0J3WmJiF54KF8NZHaTR7AzcAUOdGBFSxSkwMyT6athkUsk/UUc6 m6/todGE1EkYfAhYzcSKZc3I9iYq1U2RUwI1edcW5g69WWYc0dMzoiVlacMj0aMBZG1ZD6 4RLl7MTAlJe+LQM1GhZ43Anz4uMdMOM= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 199FB61121; Mon, 11 Sep 2023 11:52:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C558C433C7; Mon, 11 Sep 2023 11:52:30 +0000 (UTC) Date: Mon, 11 Sep 2023 12:52:27 +0100 From: Catalin Marinas To: Alexandru Elisei Cc: David Hildenbrand , will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com, pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: Re: [PATCH RFC 00/37] Add support for arm64 MTE dynamic tag storage reuse Message-ID: References: <20230823131350.114942-1-alexandru.elisei@arm.com> <33def4fe-fdb8-6388-1151-fabd2adc8220@redhat.com> <0b9c122a-c05a-b3df-c69f-85f520294adc@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 276B61C0013 X-Stat-Signature: 5scgubbm386nnp6xngoui5jtzsaoasgd X-HE-Tag: 1694433158-941098 X-HE-Meta: U2FsdGVkX19TxbDoVpBnfV5M/2p367UPRCWHWKngfcHFy1K7LcD1nITioT1hTQNy27q9glV0nya5z/PzEfdllJ1f71hgjc9yNLifMDW2LBJAaidaO3PMhw/5I+/WgcwjpcePL2lEQ5WUAppu9/ncJWEov5j5osaXWv2tZxSqXJ9bfWAFS9kse3yNEky+coIdKjKjiN8mtysDNy2gm79Trj57to8+j9mqr4Gjh8ZgxuMf6s8DSMt0Cft4ROyWyzqEc2zWbOJ4MjKsa31U52daxuBwtfz99Kbx7sNC+lD/PR+fcjUaaFTACEaAsNhocZTH5n8apRVbo+lWtiYisV6ja60eoVPKf+mnamfCk7ZEbxZKdA7OyPf+B9fw4wtb13HyDVkcC65Q8ZODnIlYiMrzkQP9ROGVoAEYuwjz+VN97T+UjMnzFVYf+0aQiba+uotIi8V7N7QntalskcdlOb6ta3MGZ9Ucc2bfcI/mI17soF9bwGtohrfEbZ88LB8/Cw/0L7Zp0Z+rno4DDegcujVP8C/c4gG2aVIp1hZeUqB428lfeGCUpI4uPNDNTE3wk9HYQNnWuwVjtUcQrOtV/Av1mwg7Hg1FhkGC971wkT273ord8IubFGTq+Wt0CgLlxgt/vlDbiRGkO3LO8RibMfEFZ1+s6Bg/Pe5qH4sUUldItNFGBGofjKr+jondmUrgJ2mz5ZZ2OboCkFNgu1ptmkQo1RmMqUhhrkE4D1yqPaDx4FLL4/VVu69E20ismjE8B/zVGEIEwDsWTQAtX2bzkfOhGKwjSORiKJKGp5yNtRFuqgOBh7zFxr287vvfh5SKfaMgVUzBT/9SbOsa5F9uzQxXGmfn8JuWSUVPyrNpzZ6U5NRA3NBVP6aDXOl4oL2jN47nGNpns57NuSZUJVByGWeewGWTjV1vGpzKyfI0gUUwdmxlhbWgngrP9bc1BONwslZxAxYsVC2kWbWWFTGIUGD vT1yoeV7 8V+Z+5JV5y909yPmJHRRnITsumyR4jJMqel3d79HZteHewnf4eTcxS1n/Ig3ALIe+Wj/zAYNEalLZ2APNZiQMM0AKc8s2FIyS6WoOkKudo5jCFUq36JJEXaq3hPV7ja4TqOm6hSnsmYdfXzp+aeWJ6DFKRR/nOMpgeA22xNIC4yqBk1MS1PuoaQPWaJiLwsgZ8sWM6rfOm2lt6Eoorgjyvt8TYfAWPr0IxR7MkK5iJ8b+jw5EEYINDWXkI0G+GJiDrqZGcb1VfEWJbgf837VcJLsErXw7Py9OW9jXeYxpd0K/Xocn7RbKJQSVtKl4nCHymlq3RH5eMXMOZNTTDEADdtZW0rpQYqqvRn85DrZxEotok7vjtJQLJDuuL+FGwzmgmBXDeo+nl4nuZEnWa4cCbnlzR94YQnWt2OIKVJdcQ91DMVQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Sep 06, 2023 at 12:23:21PM +0100, Alexandru Elisei wrote: > On Thu, Aug 24, 2023 at 04:24:30PM +0100, Catalin Marinas wrote: > > On Thu, Aug 24, 2023 at 01:25:41PM +0200, David Hildenbrand wrote: > > > On 24.08.23 13:06, David Hildenbrand wrote: > > > > Regarding one complication: "The kernel needs to know where to allocate > > > > a PROT_MTE page from or migrate a current page if it becomes PROT_MTE > > > > (mprotect()) and the range it is in does not support tagging.", > > > > simplified handling would be if it's in a MIGRATE_CMA pageblock, it > > > > doesn't support tagging. You have to migrate to a !CMA page (for > > > > example, not specifying GFP_MOVABLE as a quick way to achieve that). > > > > > > Okay, I now realize that this patch set effectively duplicates some CMA > > > behavior using a new migrate-type. [...] > I considered mixing the tag storage memory memory with normal memory and > adding it to MIGRATE_CMA. But since tag storage memory cannot be tagged, > this means that it's not enough anymore to have a __GFP_MOVABLE allocation > request to use MIGRATE_CMA. > > I considered two solutions to this problem: > > 1. Only allocate from MIGRATE_CMA is the requested memory is not tagged => > this effectively means transforming all memory from MIGRATE_CMA into the > MIGRATE_METADATA migratetype that the series introduces. Not very > appealing, because that means treating normal memory that is also on the > MIGRATE_CMA lists as tagged memory. That's indeed not ideal. We could try this if it makes the patches significantly simpler, though I'm not so sure. Allocating metadata is the easier part as we know the correspondence from the tagged pages (32 PROT_MTE page) to the metadata page (1 tag storage page), so alloc_contig_range() does this for us. Just adding it to the CMA range is sufficient. However, making sure that we don't allocate PROT_MTE pages from the metadata range is what led us to another migrate type. I guess we could achieve something similar with a new zone or a CPU-less NUMA node, though the latter is not guaranteed not to allocate memory from the range, only make it less likely. Both these options are less flexible in terms of size/alignment/placement. Maybe as a quick hack - only allow PROT_MTE from ZONE_NORMAL and configure the metadata range in ZONE_MOVABLE but at some point I'd expect some CXL-attached memory to support MTE with additional carveout reserved. To recap, in this series, a PROT_MTE page allocation starts with a typical allocation from anywhere other than MIGRATE_METADATA followed by the hooks to reserve the corresponding metadata range at (pfn * 128 + offset) for a 4K page. The whole metadata page is reserved, so the adjacent 31 pages around the original allocation can also be mapped as PROT_MTE. (Peter and Evgenii @ Google had a slightly different approach in their prototype: separate free_area[] array for PROT_MTE pages; while it has some advantages, I found it more intrusive since the same page can be on a free_area/free_list or another) > 2. Keep track of which pages are tag storage at page granularity (either by > a page flag, or by checking that the pfn falls in one of the tag storage > region, or by some other mechanism). When the page allocator takes free > pages from the MIGRATE_METADATA list to satisfy an allocation, compare the > gfp mask with the page type, and if the allocation is tagged and the page > is a tag storage page, put it back at the tail of the free list and choose > the next page. Repeat until the page allocator finds a normal memory page > that can be tagged (some refinements obviously needed to need to avoid > infinite loops). With large enough CMA areas, there's a real risk of latency spikes, RCU stalls etc. Not really keen on such heuristics. -- Catalin