From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63E5BEE57DF for ; Mon, 11 Sep 2023 12:29:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:Subject:From:References:Cc:To: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3ux+rrVlU2peTWlFk4BlAMsIdFAUQWe5FJI+wBfRJH0=; b=tSHuaCShBPW+S3 1Fb5ZEIyRq9666RzgO9zgeqUxZJlRHkFGwFNRO01OQhEoyMtdStS9cxwE3BoddT0zA8KSfc1Q83Kg Lj2WMbyRPiPjrNk07NKhZgW7zDHj0GkZa7VXG5xEODZLwEP2bMA9NSjNzk0X1yrnADBeLovOg8zR5 a2gnDR2DZgv9aoLX/1fbG99idBBbj+E2hYssssxQPKy9+bVPlVLtFSbFSxmtNl7Q6Z9uUWNIM3nUd Cp9/5sx3IJQt86owHZFJh9h32CIJhQ4SEnAJoVjHP0Ku7GN2L58SVsRIbPBbUzCPW4XEEzCXBIYQN 5wukXbN1SuV04sILtEHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qfg2W-000UI0-1W; Mon, 11 Sep 2023 12:29:16 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qfg2R-000UGv-1c for linux-arm-kernel@lists.infradead.org; Mon, 11 Sep 2023 12:29:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694435350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DHwDjcrul9YxmPEcQYx4HYVTPitVbFTJsMO8Jxfoen4=; b=jFGuC02YllIwPdmgvqsOeDSBrvooJVIAdw3hwqOCFsJGZXX20yhVdCCtMrpcIkbVzDfwa8 8M5gKn6dyO5kEM4Gxmnd/hccL0CtOzhiHmHCTXA2824AEzKFHJ5mvOZIXuJ8E+mfRnktgw UkIdQO3CAyTOebjOKi76FfyoPu38zRs= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-486-0CaOp7vcNLKQ4R0k6hEfRw-1; Mon, 11 Sep 2023 08:29:07 -0400 X-MC-Unique: 0CaOp7vcNLKQ4R0k6hEfRw-1 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-3f5df65f9f4so32764325e9.2 for ; Mon, 11 Sep 2023 05:29:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694435346; x=1695040146; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=DHwDjcrul9YxmPEcQYx4HYVTPitVbFTJsMO8Jxfoen4=; b=YmQCgt3i64XHKLH6Y4ls7w5n9C8UDO5BTYTfILsiAD0hLd0kTSTVgl3KdkBwjiiFYM aBcLrtYno8z8WHy0rlcHFwmJLMNl44NAWFQpYeoBa6b7QNMqHvNGs8qRbMvGnuH9/wh6 Ufd2v8A+uvgUsRLBRijOleCvksOwCUZsQg3o13jt66HvLQgFmPAXxAKoXI1naswo9OmT BKlgBI1Y9CclNgmOB1kWpMtzJhuTRc0kJ7QHKNtYO4ApaOYyRPhaVfvJwGgxHN+rNX2v fCRlYolSMaEAVD6R3Ne0nQ1DmZxEZqiVh7t2EaficnVIpYKzZLhf2KPdzY1EoKeVqEJC CysA== X-Gm-Message-State: AOJu0YwIPz+FaFWWD3f6fkplusvgCoiymgsQ6vG7IzIz7VU4YhNVpD7K rfruVpuD/VqvfVlWF+vel1mSAXRMmw4CzBcGLjPSOIl7Z3prI/ZxZTvAKPjN1+waOhZG7JSpv0x XNGwCh3SN1wu+OJwAIBNyOq+qxxFVk1GM7Gw= X-Received: by 2002:a05:600c:2208:b0:3fe:f667:4e4c with SMTP id z8-20020a05600c220800b003fef6674e4cmr8229327wml.12.1694435346279; Mon, 11 Sep 2023 05:29:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF0LpLhNizaUMU3zKfHh49aqw4McELTtqbEdG0mSCPhvh4bG6htIZyW8h2QYhMj39he9TwHvA== X-Received: by 2002:a05:600c:2208:b0:3fe:f667:4e4c with SMTP id z8-20020a05600c220800b003fef6674e4cmr8229290wml.12.1694435345880; Mon, 11 Sep 2023 05:29:05 -0700 (PDT) Received: from ?IPV6:2003:cb:c743:5500:a9bd:94ab:74e9:782f? (p200300cbc7435500a9bd94ab74e9782f.dip0.t-ipconnect.de. [2003:cb:c743:5500:a9bd:94ab:74e9:782f]) by smtp.gmail.com with ESMTPSA id z18-20020a1c4c12000000b003fedcd02e2asm9929615wmf.35.2023.09.11.05.29.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Sep 2023 05:29:05 -0700 (PDT) Message-ID: <0cc8a118-2522-f666-5bcc-af06263fd352@redhat.com> Date: Mon, 11 Sep 2023 14:29:03 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 To: Catalin Marinas , Alexandru Elisei Cc: will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com, pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org References: <20230823131350.114942-1-alexandru.elisei@arm.com> <33def4fe-fdb8-6388-1151-fabd2adc8220@redhat.com> <0b9c122a-c05a-b3df-c69f-85f520294adc@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH RFC 00/37] Add support for arm64 MTE dynamic tag storage reuse In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230911_052911_635254_DFD4E655 X-CRM114-Status: GOOD ( 35.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 11.09.23 13:52, Catalin Marinas wrote: > On Wed, Sep 06, 2023 at 12:23:21PM +0100, Alexandru Elisei wrote: >> On Thu, Aug 24, 2023 at 04:24:30PM +0100, Catalin Marinas wrote: >>> On Thu, Aug 24, 2023 at 01:25:41PM +0200, David Hildenbrand wrote: >>>> On 24.08.23 13:06, David Hildenbrand wrote: >>>>> Regarding one complication: "The kernel needs to know where to allocate >>>>> a PROT_MTE page from or migrate a current page if it becomes PROT_MTE >>>>> (mprotect()) and the range it is in does not support tagging.", >>>>> simplified handling would be if it's in a MIGRATE_CMA pageblock, it >>>>> doesn't support tagging. You have to migrate to a !CMA page (for >>>>> example, not specifying GFP_MOVABLE as a quick way to achieve that). >>>> >>>> Okay, I now realize that this patch set effectively duplicates some CMA >>>> behavior using a new migrate-type. > [...] >> I considered mixing the tag storage memory memory with normal memory and >> adding it to MIGRATE_CMA. But since tag storage memory cannot be tagged, >> this means that it's not enough anymore to have a __GFP_MOVABLE allocation >> request to use MIGRATE_CMA. >> >> I considered two solutions to this problem: >> >> 1. Only allocate from MIGRATE_CMA is the requested memory is not tagged => >> this effectively means transforming all memory from MIGRATE_CMA into the >> MIGRATE_METADATA migratetype that the series introduces. Not very >> appealing, because that means treating normal memory that is also on the >> MIGRATE_CMA lists as tagged memory. > > That's indeed not ideal. We could try this if it makes the patches > significantly simpler, though I'm not so sure. > > Allocating metadata is the easier part as we know the correspondence > from the tagged pages (32 PROT_MTE page) to the metadata page (1 tag > storage page), so alloc_contig_range() does this for us. Just adding it > to the CMA range is sufficient. > > However, making sure that we don't allocate PROT_MTE pages from the > metadata range is what led us to another migrate type. I guess we could > achieve something similar with a new zone or a CPU-less NUMA node, Ideally, no significant core-mm changes to optimize for an architecture oddity. That implies, no new zones and no new migratetypes -- unless it is unavoidable and you are confident that you can convince core-MM people that the use case (giving back 3% of system RAM at max in some setups) is worth the trouble. I also had CPU-less NUMA nodes in mind when thinking about that, but not sure how easy it would be to integrate it. If the tag memory has actually different performance characteristics as well, a NUMA node would be the right choice. If we could find some way to easily support this either via CMA or CPU-less NUMA nodes, that would be much preferable; even if we cannot cover each and every future use case right now. I expect some issues with CXL+MTE either way , but are happy to be taught otherwise :) Another thought I had was adding something like CMA memory characteristics. Like, asking if a given CMA area/page supports tagging (i.e., flag for the CMA area set?)? When you need memory that supports tagging and have a page that does not support tagging (CMA && taggable), simply migrate to !MOVABLE memory (eventually we could also try adding !CMA). Was that discussed and what would be the challenges with that? Page migration due to compaction comes to mind, but it might also be easy to handle if we can just avoid CMA memory for that. > though the latter is not guaranteed not to allocate memory from the > range, only make it less likely. Both these options are less flexible in > terms of size/alignment/placement. > > Maybe as a quick hack - only allow PROT_MTE from ZONE_NORMAL and > configure the metadata range in ZONE_MOVABLE but at some point I'd > expect some CXL-attached memory to support MTE with additional carveout > reserved. I have no idea how we could possibly cleanly support memory hotplug in virtual environments (virtual DIMMs, virtio-mem) with MTE. In contrast to s390x storage keys, the approach that arm64 with MTE took here (exposing tag memory to the VM) makes it rather hard and complicated. -- Cheers, David / dhildenb _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel