From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DE38C04E69 for ; Mon, 24 Jul 2023 16:56:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A85E6B0074; Mon, 24 Jul 2023 12:56:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 658796B0075; Mon, 24 Jul 2023 12:56:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 520AA8E0001; Mon, 24 Jul 2023 12:56:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 459716B0074 for ; Mon, 24 Jul 2023 12:56:42 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1015F160B61 for ; Mon, 24 Jul 2023 16:56:42 +0000 (UTC) X-FDA: 81047109444.28.6A6FE8B Received: from mail-il1-f169.google.com (mail-il1-f169.google.com [209.85.166.169]) by imf03.hostedemail.com (Postfix) with ESMTP id 3BF7820005 for ; Mon, 24 Jul 2023 16:56:40 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="0Di7l/A2"; spf=pass (imf03.hostedemail.com: domain of zwisler@google.com designates 209.85.166.169 as permitted sender) smtp.mailfrom=zwisler@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690217800; a=rsa-sha256; cv=none; b=WgMM2yQyTiuufj3YaB+2YmymkSTdpaQo/b8JcMHHU/Kjj5ULgI5PKGlkQACKOlrm1W+J+4 m3DYgfYOZ0bJzTmrDHyrR1XBAX6YFAR2VI1Ls+YgpHiBjePNkSAEGapjM5gTitkP3DWjFu Ff8i4ZccucU/ECtRYq0PA1DFrRa6b5M= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="0Di7l/A2"; spf=pass (imf03.hostedemail.com: domain of zwisler@google.com designates 209.85.166.169 as permitted sender) smtp.mailfrom=zwisler@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690217800; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h1BawSFeKkQp75BQDYvDzIq+nGzVfTgNWo74YLqiWag=; b=AmZwawRgJtvuxpUCprC/OT2y9lbQiuW5/Pq4cbZ/U7m6jGyuBn83DUPEc+4naR2jRh3Q9q sMwAOTomS7Gb0G++MTqkJ38jn5SP5391e705Mbf067SiL4z4903Fr2cEW2rQGR1rpUrOFn b7Ne54c807jU+jt5gq3/SYcZbQ/Mr38= Received: by mail-il1-f169.google.com with SMTP id e9e14a558f8ab-3464c774f23so21942405ab.1 for ; Mon, 24 Jul 2023 09:56:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690217799; x=1690822599; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=h1BawSFeKkQp75BQDYvDzIq+nGzVfTgNWo74YLqiWag=; b=0Di7l/A21DoRGVwSScxYjAwwf2wr0vBXtlcLiqdjBt47aG9QdoJF39hVuMLPdVP5Rq O28YcrSwMCcTFviUZxbru+ThXscv8Nl/53zM863TvQOVsvnlL3trmSfDmpwh0rqiEC0x TbkXAIxRgYAwIBvln2Gl5a+29o+3DXyuE+nnFY8A6wvyo12JnOFtZqlrVGbjQe9V3zph 5e7vJuy4Gc2s6WjsIEzm2v4XA+09cB+RlR2r+OvlMsFFsAaEuZRQfxluqeD16KPu7mph LIoW+MP8llkdt8QxYgSywKn/UN0pDCs/hnYX0FIyCmooMJfFkEs8+X1GPpx6HTb6NP6k KUBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690217799; x=1690822599; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=h1BawSFeKkQp75BQDYvDzIq+nGzVfTgNWo74YLqiWag=; b=O3a7wBm+VHN+LSLehCE+JMLuCTHzFoMUP89O7GBVF571lfjAK/wjYcQBWsxki9898Y 9BVxbQKna9aX1KVR8K4blajsl/OmcGO4iXuOY869Rt6y4xbXsoAvEE8MvbpUYgnVCCOC CTb7N4UTdhWnSyxksQVAjoOhucc1tpAk7+7svo7K5M8wxj8jmDvUuVetq3epP7X9yh1h oZ/fUV9bQUyOS9mtB8w75vmwPqXyLbd09Sv6DOzmC1qPzBrvYeyjv29Vxm6kpziu2Mma Iu533p3anzigQaJ+h4OiSGolMRL9k++KrRVzdH+J36FKIh7rpM8o+olMmSWmGFxIrM2z nWwA== X-Gm-Message-State: ABy/qLaCniVGmN9luPVgq02v6zahKUGjrE3klQs45i+U3iSEqTHjlBVk U5A+dBXdtUwoiDZLb4Vgp7YW0Q== X-Google-Smtp-Source: APBJJlFGNB7cLBhkP74vkaL08AgDWHiguohc1Ze12ev5+LItdOXqFFru745rS9RyPQHvKtQhT8HCOA== X-Received: by 2002:a6b:916:0:b0:780:ce89:ca73 with SMTP id t22-20020a6b0916000000b00780ce89ca73mr477988ioi.14.1690217799172; Mon, 24 Jul 2023 09:56:39 -0700 (PDT) Received: from google.com ([2620:15c:183:200:5e44:ee40:d817:ec4a]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3071026jao.64.2023.07.24.09.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jul 2023 09:56:38 -0700 (PDT) Date: Mon, 24 Jul 2023 10:56:35 -0600 From: Ross Zwisler To: Michal Hocko Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Andrew Morton , Matthew Wilcox , Mel Gorman , Vlastimil Babka , David Hildenbrand Subject: Re: collision between ZONE_MOVABLE and memblock allocations Message-ID: <20230724165635.GA20994@google.com> References: <20230718220106.GA3117638@google.com> <20230719224821.GC3528218@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3BF7820005 X-Stat-Signature: ag9ng1xzaemdsw7hfyetgsu5pd5apqsb X-Rspam-User: X-HE-Tag: 1690217799-219832 X-HE-Meta: U2FsdGVkX1/tzh3Ft6Ijus33oivVMPe+TSaOQ7cLlIw/kgc2641uuIEcSI6GHGj1Ws4SUh0kiJd2Cg+y30Efbml9nBSQxuc4Qy+TBsY2KuSYPCHOzuKdZsEm3SLcC6fLh6R1gyZJreraTl/gT5gZyz5O3lSnV1A9dwwZxm/Y1FMJ5PkIJgIUtOKUbbGPZpdmQfuRa4OKFJqhwE9uEoEGEq8nDJtawf8bk3s3wCS2Yufb++YtlNyhJN1Rj3bdsoiZ0sRVqMGHwaeMUd1Mgq8zRPSknOBCn7ZMLKQE6T6eoTGC8Pzt2WroIsx4pqo3Q46U1gK1IOs5OVYXKiIqQDTyXMu/lvt7LbXnJwrBZ6NRkD/kQ5bA1QCkLC1YtAlnn4fcMf/ifj/cYowbuwkXutF5a4lZlrGe9Er9joH20nIJoCh20F8CXoLMdXEFKDHRWVafte1FLzf41LsczOeicrd++I/SzX3OMjO6FG9pM7luWFDtHSSxfzAtF44mDlqTf0I3twYNT4sO/kJhU7IlpwBHUWoYPjRq1Qf9YAwi3mGpScpdHpX42/qRStmJbXoX2Id/Q3Nksg2N4kbVST5h+sqErIFu4PPAsSwzT4+5B/Cda9J5Ef6EWJXDoiNYB3fY3a4RAy3IWJdUBjomB9iW/0Rot/Me0Ng1sW9dVdE86AKE46QTgwMQmgBi+1YD/cDOJhlHH3ArLc0C82A7i8IHvfqPjv74hYrYe08+A2ipWiEOytkEqIsdLLcO+UIZxVnDbe/DSaQwzg1mpZ3C3MyEazAXF1WGIuOkZKqbwZi4Y6mHbxh724BqjCqIBzuWio63z+TRIiuMxwJRgidYgnFKImybDGDcYo41BhdADc+WCcwmO7hKRsYgX6IM/5bp6OXcmqvsJqIFlteIhzbq1yt+CrmYkJ2daJYf6LuquV+d+2jI27QvQtk+Vhlahxv23QchHbBzI9tmlOdfRISwySqCYay VJGPy4Mx Vf73/PUawcYcmVLhe0SBT4O+6AKzNHWWNa+ZwvqHGjBjqnm7iU1JPk1tz0Auuy3XWUv1xV40GlaP3IAtGYo1SWFdx8blOwEY78Hwxuh4Rs8rDuqxg8tIwsXS555bTTVA5EUw0Sk0PDTxxnaxZqKwave55nLFoPwrzLolm1j7vbBklem1uKQxjXIhCSFFe/ravRwFSIgyRnuq60vGLf+fOdrLX8tjtS9DhkznXKr2iHQOyi60jTr0+FZlY/os8vD9qjzTpHNj2TzVMZJXAf6myV7oYjsi8y6vAbX9tY1a1P8aJnNjOwrtRz8Y+TItXwHdIFDeuismNqCNevcxKE+ZvuSpZ+LbQgEM3DlJdhQNmeIp9ybWCM4zAO5m2anu+hsvdahf2BuLHWnuXlBXAXSF1K+6ZtLbs42vfYAWk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jul 20, 2023 at 02:13:25PM +0200, Michal Hocko wrote: > On Wed 19-07-23 16:48:21, Ross Zwisler wrote: > > On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote: > > > On Tue 18-07-23 16:01:06, Ross Zwisler wrote: > > > [...] > > > > I do think that we need to fix this collision between ZONE_MOVABLE and memmap > > > > allocations, because this issue essentially makes the movablecore= kernel > > > > command line parameter useless in many cases, as the ZONE_MOVABLE region it > > > > creates will often actually be unmovable. > > > > > > movablecore is kinda hack and I would be more inclined to get rid of it > > > rather than build more into it. Could you be more specific about your > > > use case? > > > > The problem that I'm trying to solve is that I'd like to be able to get kernel > > core dumps off machines (chromebooks) so that we can debug crashes. Because > > the memory used by the crash kernel ("crashkernel=" kernel command line > > option) is consumed the entire time the machine is booted, there is a strong > > motivation to keep the crash kernel as small and as simple as possible. To > > this end I'm trying to get away without SSD drivers, not having to worry about > > encryption on the SSDs, etc. > > > > So, the rough plan right now is: > > > > 1) During boot set aside some memory that won't contain kernel allocations. > > I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways. > > > > We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE > > region (or whatever non-kernel region) will be set aside as PMEM in the crash > > kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line > > parameter passed to the crash kernel. > > > > So, in my sample 4G VM system, I see: > > > > # lsmem --split ZONES --output-all > > RANGE SIZE STATE REMOVABLE BLOCK NODE ZONES > > 0x0000000000000000-0x0000000007ffffff 128M online yes 0 0 None > > 0x0000000008000000-0x00000000bfffffff 2.9G online yes 1-23 0 DMA32 > > 0x0000000100000000-0x000000012fffffff 768M online yes 32-37 0 Normal > > 0x0000000130000000-0x000000013fffffff 256M online yes 38-39 0 Movable > > > > Memory block size: 128M > > Total online memory: 4G > > Total offline memory: 0B > > > > so I'll pass "memmap=256M!0x130000000" to the crash kernel. > > > > 2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set > > aside only contains user data, which we don't want to store anyway. We make a > > filesystem in there, and create a kernel crash dump using 'makedumpfile': > > > > mkfs.ext4 /dev/pmem0 > > mount /dev/pmem0 /mnt > > makedumpfile -c -d 31 /proc/vmcore /mnt/kdump > > > > We then set up the next full kernel boot to also have this same PMEM region, > > using the same memmap kernel parameter. We reboot back into a full kernel. > > Btw. How do you ensure that the address range doesn't get reinitialized > by POST? Do you rely on kexec boot here? I've been working under the assumption that I do need to do a full reboot (not just another kexec boot) so that the devices in the system (NICs, disks, etc) are all reinitialized and don't carry over bad state from the crash. I do know about the 'reset_devices' kernel command line parameter, but wasn't sure that would be enough. From looking around it seems like this is very driver + device dependent, so maybe I just need to test more. In any case, you're right, if we do a full reboot and go through POST, it's system dependent on whether BIOS/UEFI/Coreboot/etc will zero memory, and if it does this feature won't work unless we kexec to the 3rd kernel. I've also heard concerns around whether a full reboot will cause the memory controller to reinitialize and potentially cause memory bit flips or similar, though I haven't yet seen this myself. Has anyone seen such bit flips / memory corruption due to system reboot, or is this a non-issue in your experience? Lots to figure out, thanks for the help. :)