From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97341C3DA7A for ; Thu, 5 Jan 2023 13:29:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 026608E0002; Thu, 5 Jan 2023 08:29:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F18CD8E0001; Thu, 5 Jan 2023 08:29:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D92AD8E0002; Thu, 5 Jan 2023 08:29:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C73688E0001 for ; Thu, 5 Jan 2023 08:29:21 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9820414027D for ; Thu, 5 Jan 2023 13:29:21 +0000 (UTC) X-FDA: 80320826922.25.859ECC8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 4AFC3A001C for ; Thu, 5 Jan 2023 13:29:19 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SlE79pTc; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672925359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O6R7HSCOxRXuYv+zoqZxPQeYz/IwjoGgHnYsJS6+SSo=; b=U0mUSWtJoyZkSQCNzCntTyDDgEUw/ks1m5ZoyL4jELrEzLCJ1IClPQ5mlFo8JY23880Gdd uW+Nz8sG2j7iRqQPfJU36doJ+LYCQidA1ykfiA9PZCbWBa4jSOLKbYsAVn5bDtMeOxWr7S FFporpArXBy25qcocp+7/Ui/Brm3O9A= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SlE79pTc; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672925359; a=rsa-sha256; cv=none; b=Ai9+4XjmE767oBn2syW6vMRq8jaKHB5Mx6r2FhtydSnwTlCWPqIkH/1uaB5oqA+R+TIR32 9MNKIAiSbnZC5ikFKOblo0je/LJjXF0BOsIGt40ysMljkfKUE0Xs4CYTnZfdzv6fAxryFv zNnBWfkegXb1eeTjUtUyRdNQlwV1KnE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672925358; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O6R7HSCOxRXuYv+zoqZxPQeYz/IwjoGgHnYsJS6+SSo=; b=SlE79pTchv8VgWk9pSWc2229E4AMkFlr3q2mkICSp+MHodbPUFvY3jk/xiKAd2Pg469gp2 uzlB4NHx79+wFKEuHgN2U6yLZ5U60S8tKV0oRq7L5ox0aqkzeQbE6eJH1hw6yamlA4RCKI l2YbEZlJlZYAXJtnG9XFsVbyhnalHHw= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-136-o3Wp_cT4NTCBix9SS8w3YA-1; Thu, 05 Jan 2023 08:29:17 -0500 X-MC-Unique: o3Wp_cT4NTCBix9SS8w3YA-1 Received: by mail-wr1-f72.google.com with SMTP id e3-20020adfa443000000b00296b280432dso2149875wra.19 for ; Thu, 05 Jan 2023 05:29:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=O6R7HSCOxRXuYv+zoqZxPQeYz/IwjoGgHnYsJS6+SSo=; b=mm37WcPFVBtKFZvORSGATHQ6ts+x9aRcWs7QaA1dBe+9nbX8YbXzeUuOPfLhlpp4p6 Wx8zSQrEud3cqd5R9HzSeC0W//DQ82jybkDguKx+5IPD6pLmXiZIDEZuTBkMDkKHUVqv jy9M3+Q8YBqzVv5cT+2P1Y28kJBjYQQ+pS/yzdq0EMvH8gPHSrSet9Aw4Nd3zRZmgydL FuVBNrqE7lQFz0kag+j7OxwWXbWcAnhZB/wRxHw09ig191Xz3EZjNjdF8yBjeDbeMze9 CYz2ZDrUf11qZTsTuX0g57yHmVAh9P/v3cj2qL6RdN+xZScqV7yNGN9zWeRjzrkKdcx6 GAPA== X-Gm-Message-State: AFqh2kp3qPtutX9OkskIEa4/iXeN8d7lObIdVTrNKZpmWuMgAcMw1KwV sAfO2Qh/UNyefGEHxWbFX+pNvhch2nWyHsGNK0XlI/Ls3a+6IjSg9HgCcWPwmVODD6A0aNKAIFp j1y3E4sdPjXE= X-Received: by 2002:a1c:6a16:0:b0:3c6:f732:bf6f with SMTP id f22-20020a1c6a16000000b003c6f732bf6fmr35815228wmc.13.1672925356185; Thu, 05 Jan 2023 05:29:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXsIb4mlwr8N+FqFf4QYtSkskqWJZSxuZsms0l9suPP1E3kmInAUswjm9Kp2ieEsBXtnPXnr7Q== X-Received: by 2002:a1c:6a16:0:b0:3c6:f732:bf6f with SMTP id f22-20020a1c6a16000000b003c6f732bf6fmr35815203wmc.13.1672925355881; Thu, 05 Jan 2023 05:29:15 -0800 (PST) Received: from ?IPV6:2003:cb:c707:6e00:ff02:ec7a:ded5:ec1e? (p200300cbc7076e00ff02ec7aded5ec1e.dip0.t-ipconnect.de. [2003:cb:c707:6e00:ff02:ec7a:ded5:ec1e]) by smtp.gmail.com with ESMTPSA id bi10-20020a05600c3d8a00b003d9bd56e9c1sm2393180wmb.11.2023.01.05.05.29.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 05 Jan 2023 05:29:15 -0800 (PST) Message-ID: Date: Thu, 5 Jan 2023 14:29:14 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 To: Florian Fainelli , Doug Berger , Andrew Morton , Mel Gorman Cc: Jonathan Corbet , Mike Rapoport , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Muchun Song , Vlastimil Babka , Johannes Weiner , Michal Hocko , KOSAKI Motohiro , Mike Kravetz , Oscar Salvador , Joonsoo Kim , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20221020215318.4193269-1-opendmb@gmail.com> <0c4e35de-f790-5399-c812-ff90a4ab7531@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v3 0/9] mm: introduce Designated Movable Blocks In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4AFC3A001C X-Stat-Signature: w54ob1rg4fwpnxsfu6b8u581s4hirs8d X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1672925359-263916 X-HE-Meta: U2FsdGVkX1+EOEsb/Wg6AI4ge2IBMeea0PqCM+LZwHr2QSPAo6kWBYdfvBikLfRI4aE/Gdu+1tiYl58WwV+P0MUCZCC+dZjBfgBY2Afqxql5nB4vyglYQ6wYnu3z8C3AiH4YCv1trJ5qRjVVKFQB/lEpA2gsNvvPhs32GnFy6mSP4Q5KdfblKaIGyOdLw+P8YIuntl44pUtu/IeTJJJIXlXPk4NQhXvvE1+USsiTYuv864tptrqYgKph2zA3IhQ0dKlSx28Xp+ebnl9Pt9yOB273EecS2AKXvOr30NIarz4JT8IM33rkRO6V5JLv2UhECh3p9aMBA9fUiTKsB1Ciw+K8Tj2byC4mtblkftdDfV66t6pfZqgqpov6ZJX10UPMqMh7FwUGBSTTtLacjqmKM38tk9oDpCyGvayC00RWCZmUMgx0U6END91QjCCWBznYHWx4mX6+HvJpZCkhkBxDnLcAuMfbHgbf4u/LP7FrP6o4a79wO+59A3hbtEAVzbAphEEAhjnI6svryB0Y4n0jJIViJisZqe18HUYivJYmnq5Zw/ZmX4iirQUlJ3l4bEpkPGs77ZvBX410w937L7LlWmLFdnDbWZV2mTIgEIGMLiiCWB+DfAVox+zGqjuz+30bUg3ISZo8PyFB2RbXjZE/+gXPy+7kR4Skkner9iTNhfBTPwnQEsmhOL312TCdhpbaaL100bU4FTfv9yTuNZCarpaAcbFQ8a/Jc8zKzeZQ+YLA+OI8iJBRI6qH7fLpmPNRxOksmmlpkitefOoi3pFkVRE3pncE6NFoYC6Lw2Cb1lw8ceh06dR/+w+RPSViO3+geN58ifoYswnBf9fP+3HT6HukxQ/NB/yMBqbXOAwkwHJnBQZT1ldLx92D8Q07cIqV98aiGYPNyb2tVDuVdTEpT40PGJrdwjxfMGNMYErfFbpM8wGEddT4jQIVYMf9H8CjgAP4DpES0JLw80Lv5Ci PjopL20t IWYDAgJXcX87PqBC/juePCQSJGbvX/EA07r6r3jAwP4SKrevww7DbD2wef3Gs579BHE71TymuDIdhP945WZPZ4l3pzIlYgYfPJJjWkjBCFC2ZcSkV9ZxxW4xntymBYrUYApAEySa/wdaJoBEoxu3fIebrop/AFaCP/FaHtX9fQjgw4656Oax4ARG1Pn3bofvFxUu5qPr+6lmQ48bO16/2nHRYYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 04.01.23 20:00, Florian Fainelli wrote: > On 1/4/23 07:56, David Hildenbrand wrote: >> On 04.01.23 00:43, Florian Fainelli wrote: >>> On 10/20/22 14:53, Doug Berger wrote: >>>> MOTIVATION: >>>> Some Broadcom devices (e.g. 7445, 7278) contain multiple memory >>>> controllers with each mapped in a different address range within >>>> a Uniform Memory Architecture. Some users of these systems have >>>> expressed the desire to locate ZONE_MOVABLE memory on each >>>> memory controller to allow user space intensive processing to >>>> make better use of the additional memory bandwidth. >>>> Unfortunately, the historical monotonic layout of zones would >>>> mean that if the lowest addressed memory controller contains >>>> ZONE_MOVABLE memory then all of the memory available from >>>> memory controllers at higher addresses must also be in the >>>> ZONE_MOVABLE zone. This would force all kernel memory accesses >>>> onto the lowest addressed memory controller and significantly >>>> reduce the amount of memory available for non-movable >>>> allocations. >>>> >>>> The main objective of this patch set is therefore to allow a >>>> block of memory to be designated as part of the ZONE_MOVABLE >>>> zone where it will always only be used by the kernel page >>>> allocator to satisfy requests for movable pages. The term >>>> Designated Movable Block is introduced here to represent such a >>>> block. The favored implementation allows extension of the >>>> 'movablecore' kernel parameter to allow specification of a base >>>> address and support for multiple blocks. The existing >>>> 'movablecore' mechanisms are retained. >>>> >>>> BACKGROUND: >>>> NUMA architectures support distributing movablecore memory >>>> across each node, but it is undesirable to introduce the >>>> overhead and complexities of NUMA on systems that don't have a >>>> Non-Uniform Memory Architecture. >>>> >>>> Commit 342332e6a925 ("mm/page_alloc.c: introduce kernelcore=mirror >>>> option") >>>> also depends on zone overlap to support sytems with multiple >>>> mirrored ranges. >>>> >>>> Commit c6f03e2903c9 ("mm, memory_hotplug: remove zone restrictions") >>>> embraced overlapped zones for memory hotplug. >>>> >>>> This commit set follows their lead to allow the ZONE_MOVABLE >>>> zone to overlap other zones. Designated Movable Blocks are made >>>> absent from overlapping zones and present within the >>>> ZONE_MOVABLE zone. >>>> >>>> I initially investigated an implementation using a Designated >>>> Movable migrate type in line with comments[1] made by Mel Gorman >>>> regarding a "sticky" MIGRATE_MOVABLE type to avoid using >>>> ZONE_MOVABLE. However, this approach was riskier since it was >>>> much more instrusive on the allocation paths. Ultimately, the >>>> progress made by the memory hotplug folks to expand the >>>> ZONE_MOVABLE functionality convinced me to follow this approach. >>>> >>> >>> Mel, David, does the sub-thread discussion with Doug help ensuring that >>> all of the context is gathered before getting into a more detailed patch >>> review on a patch-by-patch basis? >>> >>> Eventually we may need a fairly firm answer as to whether the proposed >>> approach has any chance of landing upstream in order to either commit to >>> in subsequent iterations of this patch set, or find an alternative. >> >> >> As raised, I'd appreciate if less intrusive alternatives could be >> evaluated (e.g., fake NUMA nodes and being ablee to just use mbind(), >> moving such memory to ZONE_MOVABLE after boot via something like daxctl). > > This is not an option with the environment we have to ultimately fit in > which is Android TV utilizing the GKI kernel which does not enable NUMA > and probably never will, and for similar reasons bringing a whole swath > of user-space tools like daxctl may not be practical either, from both a > logistical perspective (simply getting the tools built with bionic, > accepted etc.) as well as system configuration perspective. Adding feature A because people don't want to (! whoever the "people" are) enable feature B? I hope I don't have to tell you what I think about statements like this :) If feature B is a problem, try stripping it down such that it can be enabled. If it's to hard to configure for your use case, maybe we can extend configuration options to avoid tools like daxctl for some special cases. But of course, only if feature B actually solves the problem. One issue I have with DMB is actual use cases / users / requirements. Maybe requirements are defined somewhere cleanly and I missed them. If we have clear requirements, we can talk about possible solutions. If we have a specific solution, it's harder to talk about requirements. > >> >> I'm not convinced that these intrusive changes are worth it at this >> point. Further, some of the assumptions (ZONE_MOVABLE == user space) are >> not really future proof as I raised. > > I find this patch set reasonably small in contrast to a lot of other mm/ > changes, what did you find intrusive specifically? > > AFAICT, there only assumption that is being made is that ZONE_MOVABLE > contains memory that can be moved, but even if it did not in the future, > there should hopefully be enough opportunities, given a large enough DMB > region to service the allocation requests of its users. I will go back > and read your comment to make sure I don't misunderstand it. Let me clarify what ZONE_MOVABLE can and cannot do: * We cannot assume that specific user space allocations are served from it, neither can we really modify behavior. * We cannot assume that user space allocations won't be migrated off that zone to another zone. * We cannot assume that no other (kernel) allocations will end up on it. * We cannot make specific processes preferably consume memory from it. Designing a feature that relies on any of these assumptions is IMHO wrong. If you want an application to consume memory from a specific address range, there are some possible ways I can see: (1) Model the special memory areas using fake NUMA nodes. e.g., mbind() the applications to these nodes. Use ZONE_MOVABLE to make sure we don't get unmovable allocations. The buddy will take care of it. (2) Use some driver that manages that memory and provides that memory to an application by mmap()'ing it. The buddy won't manage it (no swap, migration ...). DEVDAX is one possible such driver. (3) Use hugetlb and reserve them from the selected memory ranges. Make the application consume these hugetlb pages. For a single node, without a special driver, it gets more complicated: We'd need new way to tell the buddy that these memory ranges are "special". I don't want to use the word "new zone" but that's most likely what it would have to be. Further, one would need a way to specify that only specific allocations should end up on these ranges. Maybe I'm overthinking this. Having clear requirements such that we can try discussing solutions and exploring alternatives would be great. -- Thanks, David / dhildenb