From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F4017C001B0 for ; Wed, 19 Jul 2023 22:48:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40908280096; Wed, 19 Jul 2023 18:48:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B7AB28004C; Wed, 19 Jul 2023 18:48:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27FCB280096; Wed, 19 Jul 2023 18:48:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 17CD528004C for ; Wed, 19 Jul 2023 18:48:29 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C8ED01404C2 for ; Wed, 19 Jul 2023 22:48:28 +0000 (UTC) X-FDA: 81029851896.14.A79B459 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) by imf21.hostedemail.com (Postfix) with ESMTP id EB4B11C0024 for ; Wed, 19 Jul 2023 22:48:26 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=6tp966Gq; spf=pass (imf21.hostedemail.com: domain of zwisler@google.com designates 209.85.166.42 as permitted sender) smtp.mailfrom=zwisler@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689806907; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MEInahlEsHjcSww3xlsFFaRjkkq9xhhzX3ommlPB214=; b=W5H/QuTZL6hm4/OUv30ms5OdfXuXlj3vx8ArIa725C9lX3+MJuvYaaHGqjhCtVCi7FVEyZ kY72UkEUVk4wRjSPAgDKV6cy496qz4fNfg1vm4+ssRcRBSWaXGnyUSN6t7zMXwC/OshILO EV6cVWJneHQEi/7wI700fMqTDW2MXXc= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=6tp966Gq; spf=pass (imf21.hostedemail.com: domain of zwisler@google.com designates 209.85.166.42 as permitted sender) smtp.mailfrom=zwisler@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689806907; a=rsa-sha256; cv=none; b=f5dRNc7oSh8xYSJ8KvQ5rWtFi+dPsmSQxLUzJaySYbnfR3SjTFs0BklJD1ASoBrq4bIIzk xSx7s2b6dPhusjtG4ZjmyuE/2uLBz65On3GDZkTQqWcRAl/nxr0Ab6Vf0zuG4wTmlsWi4F 5UM0KguHcKkmnPZ7FVqjMXduusgeZRA= Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7836164a08aso6447539f.1 for ; Wed, 19 Jul 2023 15:48:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689806906; x=1692398906; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MEInahlEsHjcSww3xlsFFaRjkkq9xhhzX3ommlPB214=; b=6tp966Gqs/ZMxnKjLWUUJqUqBl4BzuZGXqP9Mmb4vHPejYI9MnffUrgLxx4/IF14o+ ikpT/1FbqXrEukdQDXj/VnjMHO9btaa/85Y/1rQ7clmzjsu32vyQ3bM02Ac3rp96p3eM Dqv+LaezvNgSUJ5N4P2wxPVF8ZHK5TiHd2EdSd+2S5ONj9OYc8DZXwnBm7wiWPQ4/ljz YCOL8Giz6Hocq+XMWB9ejRU+ilEnLD8oyyCgpQH5yXvHCIy0dZIugE+fahUmyXX0VTWf w8csHyIWL4HpUnI4UruW2T9bMLDjg3kXttaCHWa9OQmAH6v5h5YAcrrB88ckNsjvEGul h+uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689806906; x=1692398906; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MEInahlEsHjcSww3xlsFFaRjkkq9xhhzX3ommlPB214=; b=GF7uui9/0NQ4VPnEEjHFmTthScJVa6gvglmGnm+lXlGEIjTdgwyHi6LGHXFFhrKoo3 MMmPU9+HPCk8E9SrRS1WMggBWCGBm78k5POBna2Zeo0AVCxfVsfZgtvn26icdys7hf5p a8/UFEShyHUrkqfAhGhgKfBHzJsUArgpzs+kN+71nyXSqe/1ojBTtL/FUpE50aHdcLBe a7us4qIJf2duuHmSUxBEOI9gDdsS5dggNxkb6j7HHQP/qxoCxiQVNxBF6UJKEhQuGt+j duXx7RvlzzYNDL/Lr3sGPeO7C6WH2f+sIQudXUOrEcPLxLXjapssq6Jo+E315694Wcf3 M0RQ== X-Gm-Message-State: ABy/qLY2c3qyJKoP3Youbf+gM4Jbl0jGJnCVAAwt2YCSnQLPH8yJvSXm lx4fbQakb2VvPbK9eRFuWlYzsA== X-Google-Smtp-Source: APBJJlHG2p5kJkjtlR54T+n8e3Fhz/nng4jNTrNeUYgDF4OxcLYDA4PP4lkpT+l8Zh2rIK0GXczqjQ== X-Received: by 2002:a6b:fb0f:0:b0:784:e618:e6cd with SMTP id h15-20020a6bfb0f000000b00784e618e6cdmr695079iog.20.1689806905915; Wed, 19 Jul 2023 15:48:25 -0700 (PDT) Received: from google.com ([2620:15c:183:200:84dc:2218:a741:ad50]) by smtp.gmail.com with ESMTPSA id y20-20020a5ec814000000b00786fd8e764bsm1659096iol.0.2023.07.19.15.48.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 15:48:25 -0700 (PDT) Date: Wed, 19 Jul 2023 16:48:21 -0600 From: Ross Zwisler To: Michal Hocko Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Andrew Morton , Matthew Wilcox , Mel Gorman , Vlastimil Babka , David Hildenbrand Subject: Re: collision between ZONE_MOVABLE and memblock allocations Message-ID: <20230719224821.GC3528218@google.com> References: <20230718220106.GA3117638@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: EB4B11C0024 X-Rspam-User: X-Stat-Signature: xmca1ixrdpjxmwz35xq41rcjpwxigp7h X-Rspamd-Server: rspam01 X-HE-Tag: 1689806906-295713 X-HE-Meta: U2FsdGVkX19mnryojkoxU0j6iqgatFihrYezwnlb5rE8ij0oqTGLkx5Ur3cupqvZv6NaOQbLGh4Hiqm9rEjHV3AaxPW65F+rf0/DoQ9gSXqbyHxvLVuZAz57Tnpr3Xbi/uhqMgKqKn4D0Z3cG5xfwRLZwsP2qI3o929jvXXTx3TV97Veq87oyHg8RvuafW9jFLptkDakI9YXZ+yAxHI+4yG6nPdNOdBnmpf3ELRqsOJyhATMZgZTORJMN0tQZFSFt8yJfSDbvl6a9Yib+PryX1fQIuMyKMCvqwXUmW5HAK7VnkELaVciiHvNxmQVdjl3VxDgsdU7Pz3yj6aRQiyKPdFwA4BuDhSlpsz7u2ThgrhCdFJ0797y3IDO7oNrPuphbTGvOCKlQOkTUqHnN11b3hMt484Y6iEZxAyMVe754Mt24leG925/uvu3T0CyKyi+vTv0eLEZnGP7anxB0Byc+3/llqDKwKrwgpbZS+EJke8bzV8iymHvsMIqZ3aQKH294PYEldQnpGdDr6yiYisEy80Hbx4lstK0iOsAf9lkIj5OGauCZ28HgimZjkmEEPAo8dYWvxP+lxuEjKCkWsiyNwq3240tAdFpWHnAdqZWTVkdfTpZPF+6Ubgfj4zjXt3iqlwQudzYXS+jkcT7aCW8ZAfbg6+nLzfx0vQkBzDjT07ReREJ/FhLXO1Fw0wLdFQOHyKXKO9pcmef0svhEOAPfMTx+TlWhT6cktHnWbuSsXM0WLQTJJXUvjjAdsDPWQmu5JmFCgEpRH6DcvPBy0kTyxadBroINfV9C3NuQVBgaBigP3h/7HYatE8eu/KV32QhVdjkdYFZ7XaCYPKbN4iIjvD+GgwszYlbX4dfKnG0JI0ZD44T2ukykiInGZERWoQ8y4V9Nfdba7ksxV6MDgpxEcjv6FlVgpM06ajIdxYif6EBF79yCUYU1/Ssk/ZhCKr4uk2Zc3irSs5xYC3EGpg 9D9mN13F BRPmRo44EU/SUIGmV43/QRWd5qiLvqA6bKemkuJlbLqwsFi2rikvto5zBGr3iN48r0T5WLmfLazMsXlb27ktwgZ8wGUSDsG3L85YUPLP95xX1eClSx/SXj7ceOdxrdGYgdpGUweYsOx+BtE7UtU++rfxXNilbuzpSQDoRkzreJQc6BjN1ZynR6Sv1DE/fp84aS41zxYULzc418HDFoawipxzbgaFnWT0N7H4P3O0D/bdGMvCiUVYp9bHkbbSUDg9gbgSqZguWCkRlSkxBS3HhDOy6yZMgRxF+kfhrK5xOWfTMyZfHwJ8ENWrRFessD+Hjmchmf+NaXNLcU/O2AFmYag/nINI4Es0jeAp6qSTq0GjR9TrAoxmsmTsuqOg9i3pWa+qbju6q/AdcxZv0wdvLNbZb2G2w4W3q0CsV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote: > On Tue 18-07-23 16:01:06, Ross Zwisler wrote: > [...] > > I do think that we need to fix this collision between ZONE_MOVABLE and memmap > > allocations, because this issue essentially makes the movablecore= kernel > > command line parameter useless in many cases, as the ZONE_MOVABLE region it > > creates will often actually be unmovable. > > movablecore is kinda hack and I would be more inclined to get rid of it > rather than build more into it. Could you be more specific about your > use case? The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc. So, the rough plan right now is: 1) During boot set aside some memory that won't contain kernel allocations. I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways. We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel. So, in my sample 4G VM system, I see: # lsmem --split ZONES --output-all RANGE SIZE STATE REMOVABLE BLOCK NODE ZONES 0x0000000000000000-0x0000000007ffffff 128M online yes 0 0 None 0x0000000008000000-0x00000000bfffffff 2.9G online yes 1-23 0 DMA32 0x0000000100000000-0x000000012fffffff 768M online yes 32-37 0 Normal 0x0000000130000000-0x000000013fffffff 256M online yes 38-39 0 Movable Memory block size: 128M Total online memory: 4G Total offline memory: 0B so I'll pass "memmap=256M!0x130000000" to the crash kernel. 2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway. We make a filesystem in there, and create a kernel crash dump using 'makedumpfile': mkfs.ext4 /dev/pmem0 mount /dev/pmem0 /mnt makedumpfile -c -d 31 /proc/vmcore /mnt/kdump We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel. 3) The next full kernel will be a normal boot with a full networking stack, SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out the kdump and either store it somewhere persistent or upload it somewhere. We can then unmount the PMEM and reconfigure it back to system ram so that the live system isn't missing memory. ndctl create-namespace --reconfig=namespace0.0 -m devdax -f daxctl reconfigure-device --mode=system-ram dax0.0 This is the flow I'm trying to support, and have mostly working in a VM, except up until now makedumpfile would crash because all the memblock structures it needed were in the PMEM area that I had just wiped out by making a new filesystem. :) Do you see any blockers that would make this infeasible? For the non-kernel memory, is the ZONE_MOVABLE path that I'm currently pursuing the best option, or would we be better off with your suggestion elsewhere in this thread: > If the spefic placement of the movable memory is not important and the only > thing that matters is the size and numa locality then an easier to maintain > solution would be to simply offline enough memory blocks very early in the > userspace bring up and online it back as movable. If offlining fails just > try another memblock. This doesn't require any kernel code change. If this 2nd way is preferred, can you point me to how I can offline the memory blocks & then get them back later in boot? Thanks for the help! - Ross