From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DFDBC001DC for ; Wed, 19 Jul 2023 23:05:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93218280099; Wed, 19 Jul 2023 19:05:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E21D28004C; Wed, 19 Jul 2023 19:05:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 75BE0280099; Wed, 19 Jul 2023 19:05:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 645E428004C for ; Wed, 19 Jul 2023 19:05:22 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 20A8A1404CA for ; Wed, 19 Jul 2023 23:05:22 +0000 (UTC) X-FDA: 81029894484.06.5022CFF Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) by imf10.hostedemail.com (Postfix) with ESMTP id 52B8EC0008 for ; Wed, 19 Jul 2023 23:05:20 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=UKTkTk7Y; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of zwisler@google.com designates 209.85.166.50 as permitted sender) smtp.mailfrom=zwisler@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689807920; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NtoVTIzSaEDDzG0bR4jOtPcvMd189PXBa9TamlZo308=; b=t5JZa1bluHiJfHTmpzsTcx6bsUPMdRHTnb+fzdbo2Hl3ffqQh1ID6/EXgK+JFxGzPswW1q w5DB+FOPGTzYgSIcCKhhXnDRRtOiRXSJLnpKXVV2JWaK6G+uzJdcsHoE/0hmMAATsH3VS5 v157JIz8++Jlz4NUUyxRz3jTwZPD6cI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=UKTkTk7Y; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of zwisler@google.com designates 209.85.166.50 as permitted sender) smtp.mailfrom=zwisler@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689807920; a=rsa-sha256; cv=none; b=TU6NJL+cRgS7FzZBlTtaiKqpigLF99tjPNych0GLx/pWf0WXae878o0XwQxljzA40BBFX7 6lJ7ebp9zerHRu44vfEYE3mT9/Wr280dEqqPbozAMTo68I5ZiWk2pRTnqOSIbz3ZwKxS0/ Pt/qYIC6YgjG6nOpUcc8ubS2b7bTZl0= Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-78654448524so6895539f.2 for ; Wed, 19 Jul 2023 16:05:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689807919; x=1692399919; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=NtoVTIzSaEDDzG0bR4jOtPcvMd189PXBa9TamlZo308=; b=UKTkTk7YGb5sJ7JSucdkUItIsjvrFJMq+hXa/yO2GuqHnTAu7Cl6ve+b8+wbptuh69 VLH+HU3danMVPr/I+PdFWVrA78ovyfPAJoqzx83Nr0d3etmfqB7KlNjnU+53bFnqBm5h bJym6ezLXxvMwjpO5vyvR3FrM+VYAoRirtYYiLriz6Qi82hVxAb2a/VSWQcxcfbITdE6 REBx7cKruKtMUVABv5K/aYV7ues8suooxt0O3/j4FIk6358bgNVWlpbZjW0AUAybaJS+ /qtOVpoczOWQfazbmqVYQGQu4ZVMkhQHG1AJsiWOK5rMHxtd6SsCRo5Fb4KQu1jgpbi8 PujA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689807919; x=1692399919; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=NtoVTIzSaEDDzG0bR4jOtPcvMd189PXBa9TamlZo308=; b=WJCgpBW0GUwKzGaUEocrjz0jLs3r62Wku8W3jzPHkuw4I89fmKDYdfQaYGqsqAMHUw mSOYdl2M78auahomy6+LzuZmz72S+Vh++axPucEXWEGXgDpRHhg2wiIfG/6pkuw+n2lW xnsPt2sNiOEQU+pFEDTOUIX45B9qlxXj1VyUPx1WAIVLe4/e+lip0IzEnzaP4ID2/t+4 SVw+CbVlCOvS/bjZXYU+CA0xs3giJ+IP7LuJl3EXYvKZv8GyknKIQBkgaY1P1Tus3/gJ axG5HnhFg9W6WgBU+lPil/3N8EZ+tCXJuyPGpl3RBzvNChjbjUKrOLmCzfoLAHO4tg5a DvYg== X-Gm-Message-State: ABy/qLbN+Hq6TcEzhdKwMeZahQZMhJ2yep+k2JTcA0dhJZqvJBb2wfHz 6CUF9Ai+6fHh96IbUCuhDNnyZQ== X-Google-Smtp-Source: APBJJlGK517Ks2qQjDvuU28RE/+1S6J6gj01Hw8Ew77Bdf7DnmdFmIYgQT9VnrMGj1JSVi2zkl+vUA== X-Received: by 2002:a6b:610d:0:b0:783:5209:c01 with SMTP id v13-20020a6b610d000000b0078352090c01mr7971152iob.17.1689807919206; Wed, 19 Jul 2023 16:05:19 -0700 (PDT) Received: from google.com ([2620:15c:183:200:84dc:2218:a741:ad50]) by smtp.gmail.com with ESMTPSA id e9-20020a056638020900b0042bae96eba7sm1624193jaq.7.2023.07.19.16.05.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 16:05:18 -0700 (PDT) Date: Wed, 19 Jul 2023 17:05:15 -0600 From: Ross Zwisler To: David Hildenbrand Cc: Michal Hocko , Mike Rapoport , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Matthew Wilcox , Mel Gorman , Vlastimil Babka Subject: Re: collision between ZONE_MOVABLE and memblock allocations Message-ID: <20230719230515.GA3654720@google.com> References: <20230718220106.GA3117638@google.com> <20230719075952.GH1901145@kernel.org> <9770454d-f840-c7cf-314e-ce81839393e3@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9770454d-f840-c7cf-314e-ce81839393e3@redhat.com> X-Rspamd-Queue-Id: 52B8EC0008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: k31ehreig6zdccja5qdwy1361rrmdgty X-HE-Tag: 1689807920-921261 X-HE-Meta: U2FsdGVkX18s+UqlBHKY0VgaDqislH+5i0IBkaQUks58RJR90UG4XIcK19Qw6KUqURVAUkvHmVLEXO+BTEtMPfy5HiHUQFtGuO05W9jTkHshUdqIJMlI+q6dtU1BWz481KNVeQ7OITACQmxri66xCMcLg+q7s41zDExsZn/IHs2BpUA4TbW1TDy5/qcEZxiNWcZO6Z/NwOroghkFm2BVtbMZx/QGqI3HYnnRpG620Gw8nCDr/q7OhUrAHfFoKALvh9TSqOd+VgZV3puD6k+hEthTeNL4xWu3i/H+4A6s3BKAYwMgfVK0xmu0JECRshphwY49nvJuRYXDiKnNcI148NaFg5xGczt/lxf7TPo6Gu3dpvqkGn90LSoOTBWt7UdoboFQ4XwTtjsg2Q4flN8qfSf7IviprdBBH2IE8IoioWS3hzKepOjC1pAJzCZmvR6GxSKPD7IGo2lOsuLET0e7XB6Iqsy8DhQibYr2M0+udLDgrUhazqRD7bngmD1ITbV/XNB2+U2iqzIufsRxQdxJYOZefmpPbiRIv7E99cBFsouAWRF9fvIpQUWJDGTJlZjS2U9pYXOXi/oYFUgxDyjyl7b3smG0u9ga67oXnACu9aA69zyjSvY6PFo0u0/HwCMoLdFB0rVMW9QjIgSzB1LV74eLn/M7fa9BYXyEb2LqTrFWp/m3JY2kTt5NcvZshq0KgMnQgFz21nfR+kmT2eqnHnTLDy2nl0d62Z193qfgHPcK4CduY+L3wK8PebHKPIQgSkmFCdUFOhIbzF49+1JFLHXACKRoVrpmNwwKr1IGTrJgZgjR5P3BhmWHg9UgdkdPflKqAyHB0cX+85+51L1wVotKxwLBLkSA/CDbPL/XlC+gWAxnZnVwY2V6cz0Xf7tfvWtpiSyUj6dMRkz8P1f82CGIxzFpN6ft0iry/TUgX4ACq0xaISgJJ8f1iznHsRKL47SIupOSKMxYAgmnXJE hdWpYjvG bv2RtTnFQr47TWciNRHpqpnsvZGktEyOsOu3TtNQeK6MxidSVXdvT61MRBH5oGB61Fi4xw5eAl9L0k9iQ6nSMvshTnnswT9BrZxR/1e9nrb/K/GnSiXGjWUi1Dqjw34QOCxS3x4v6lHDlz8MHL47v3Q068bRdFhIDTzl5e4h+B/tn3V1clpvuv3a68v3cokt8gi4ytZrGi4q5byvSMdr2NO51l49fQDJ2EIkME6p3kv+I1TE8Zco+HNDzvuSqij+zy37jh6VJOOR1R/GT7t+9Fk/0YJmARTWmEUVwWPg+RhkM4ehP0wPv+U8Qj2Wl4xGOGjkQL2ZkfyF57RIC1i0JG5spTdlUAPHZQwY/AKKMsPfgXRSmM9TimqvWFaiMW6/O1UG+ZpGrRtzxSQl++sZ3jXIGDc/JrU5QsIud6nCulsa5sfTpYdtrceDYPTotd0i5hhTe/Co2aRUhxCuQu87rq2WBpvBzkXhOr2pB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 19, 2023 at 10:14:59AM +0200, David Hildenbrand wrote: > On 19.07.23 10:06, Michal Hocko wrote: > > On Wed 19-07-23 10:59:52, Mike Rapoport wrote: > > > On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote: > > > > On Tue 18-07-23 16:01:06, Ross Zwisler wrote: > > > > [...] > > > > > I do think that we need to fix this collision between ZONE_MOVABLE and memmap > > > > > allocations, because this issue essentially makes the movablecore= kernel > > > > > command line parameter useless in many cases, as the ZONE_MOVABLE region it > > > > > creates will often actually be unmovable. > > > > > > > > movablecore is kinda hack and I would be more inclined to get rid of it > > > > rather than build more into it. Could you be more specific about your > > > > use case? > > > > > > > > > Here are the options I currently see for resolution: > > > > > > > > > > 1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from > > > > > the beginning of the NUMA node instead of the end. This should fix my use case, > > > > > but again is prone to breakage in other configurations (# of NUMA nodes, other > > > > > architectures) where ZONE_MOVABLE and memblock allocations might overlap. I > > > > > think that this should be relatively straightforward and low risk, though. > > > > > > > > > > 2. Make the code which processes the movablecore= command line option aware of > > > > > the memblock allocations, and have it choose a region for ZONE_MOVABLE which > > > > > does not have these allocations. This might be done by checking for > > > > > PageReserved() as we do with offlining memory, though that will take some boot > > > > > time reordering, or we'll have to figure out the overlap in another way. This > > > > > may also result in us having two ZONE_NORMAL zones for a given NUMA node, with > > > > > a ZONE_MOVABLE section in between them. I'm not sure if this is allowed? > > > > > > > > Yes, this is no problem. Zones are allowed to be sparse. > > > > > > The current initialization order is roughly > > > > > > * very early initialization with some memblock allocations > > > * determine zone locations and sizes > > > * initialize memory map > > > - memblock_alloc(lots of memory) > > > * lots of unrelated initializations that may allocate memory > > > * release free pages from memblock to the buddy allocator > > > > > > With 2) we can make sure the memory map and early allocations won't be in > > > the ZONE_MOVABLE, but we'll still may have reserved pages there. > > > > Yes this will always be fragile. If the spefic placement of the movable > > memory is not important and the only thing that matters is the size and > > numa locality then an easier to maintain solution would be to simply > > offline enough memory blocks very early in the userspace bring up and > > online it back as movable. If offlining fails just try another > > memblock. This doesn't require any kernel code change. > > As an alternative, we might use the "memmap=nn[KMG]!ss[KMG]" [1] parameter > to mark some memory as protected. > > That memory can then be configured as devdax device and online to > ZONE_MOVABLE (dev/dax). > > [1] https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap I've previously been reconfiguring devdax memory like this: ndctl create-namespace --reconfig=namespace0.0 -m devdax -f daxctl reconfigure-device --mode=system-ram dax0.0 Is this how you've been doing it, or is there something else I should consider? I just sent mail to Michal outlining my use case, hopefully it makes sense. I had thought about using 'memmap=' in the first kernel and the worry was that I'd have to support many different machines with different memory configurations, and have to hard-code memory offsets and lengths for the various memmap= kernel command line parameters. If I can make ZONE_MOVABLE work that's preferable because the kernel will choose the correct usermem-only region for me, and then I can just use that region for the crash kernel and 3rd kernel boots.