From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CA3DEB64DA for ; Fri, 14 Jul 2023 13:48:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7493D6B0071; Fri, 14 Jul 2023 09:48:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F9906B0072; Fri, 14 Jul 2023 09:48:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C0FA6B0074; Fri, 14 Jul 2023 09:48:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4D1E46B0071 for ; Fri, 14 Jul 2023 09:48:20 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0443BC0222 for ; Fri, 14 Jul 2023 13:48:19 +0000 (UTC) X-FDA: 81010346760.19.6431149 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id EC3C810001A for ; Fri, 14 Jul 2023 13:48:17 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F+062NFp; spf=pass (imf14.hostedemail.com: domain of jmoyer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=jmoyer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689342498; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xinDoQRdcE5sxZPT6WiHl1eOgAOzV50L/CpzwNIn2nE=; b=LTuWKYhC7HedJdubhLT3x8/S8asZKsGPPn0xlxFvRcakPRmER+AMRiJjO9czj6irokoTky wKx+UOn00183qTuh8kfq0hU85ugu39IP9hyrLPK2evn7WdMlfE4O//RLicJ2FT7sBGhX+P ZYl6cM3XtcmwzAErfyBrPDbFHb+mNYE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689342498; a=rsa-sha256; cv=none; b=r/fcOrIEkYy0nXFJccGpethjkcASKnH7Mii6A2xSaC4CuxzU//UKZ2q55Tp2rInRRfGBeQ u5bk462NHMOlWpKDzzl/NIkzEMJfqWhPIhSDxuWI+SGQaEJd0+5D7rCsEv/YN6O6a8IqLv NutvknjW46iGhfcMeHQX2nAR5Mpl7os= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F+062NFp; spf=pass (imf14.hostedemail.com: domain of jmoyer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=jmoyer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689342497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xinDoQRdcE5sxZPT6WiHl1eOgAOzV50L/CpzwNIn2nE=; b=F+062NFpQ8JV9QQl7Xw2Tvlbqv3oSLlKFH55XOEeBnc4wZpdrlWzADResF36eJA+UiSiiN OuMUOdU3CqcckCUW/zLB47NKUWKiUMgdlrbJDRWd4gOROQGRU8JazMr9UmemuQ/ogAfB5e y3UMH9JDnaNs86E6jlabC5k6zmSF8G0= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-279-JvoMrE8ZMvyfp9R_BeIRTQ-1; Fri, 14 Jul 2023 09:48:13 -0400 X-MC-Unique: JvoMrE8ZMvyfp9R_BeIRTQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C0E221C0419A; Fri, 14 Jul 2023 13:48:12 +0000 (UTC) Received: from segfault.boston.devel.redhat.com (segfault.boston.devel.redhat.com [10.19.60.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 312B41454142; Fri, 14 Jul 2023 13:48:12 +0000 (UTC) From: Jeff Moyer To: David Hildenbrand Cc: Dan Williams , Vishal Verma , "Rafael J. Wysocki" , Len Brown , Andrew Morton , Oscar Salvador , Dave Jiang , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, Huang Ying , Dave Hansen Subject: Re: [PATCH 0/3] mm: use memmap_on_memory semantics for dax/kmem References: <20230613-vv-kmem_memmap-v1-0-f6de9c6af2c6@intel.com> <29c9b998-f453-59f2-5084-9b4482b489cf@redhat.com> X-PGP-KeyID: 1F78E1B4 X-PGP-CertKey: F6FE 280D 8293 F72C 65FD 5A58 1FF8 A7CA 1F78 E1B4 Date: Fri, 14 Jul 2023 09:54:02 -0400 In-Reply-To: (David Hildenbrand's message of "Fri, 14 Jul 2023 10:35:47 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: EC3C810001A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 9jho3pqwow3m71bnkjhxdhbfgumkf438 X-HE-Tag: 1689342497-807558 X-HE-Meta: U2FsdGVkX1+OkHpUKKkkdz4V5AbR7HCqoCgx9XyxjLofSyJtETdOTbsZsdIj2SQuorMwh4z9Y19IuL6UIs1TPrdsSxaRStXJWC2r2ASCR6+4jhyNkrUa9aPdFqNY3QfApLyXvB5aBqMupBlbjdXPVMbQsmcy5zxEeBB2rKlOIzOICYzFW/k7UIBc9D/kAzT6x+wdDskqG5x16xujm1sw4nAcDDQQCHtK4VNew/8pDT1eOCELMaaLtIF3eDMvejhqS8TtnFjYopF2LpnVmvWePz5G64UeACPoCPLNhEzwlCqXewiu8BUIlYZrcusZEi0fK1X7sIq/VZeprO7sNztHe5kemocSJecL4RQm7OgfG5rIotLNwLFf/uC5sdUBZCdBClNBktY8DCM///w07BVilovC53Mq4NoRHNEs9a8MfClQaHe97qTbd5Md61yGzgrvvjQnR1SfD0376W1TcCY//YyqmM/RGkJAlcL92DEUr/cqUY5FP2+w+A44KTaL8z9rBmBjZRbhzOQWqoIzwKrxU7O1iJgdnR/Pl2Xqhdgcoe7nTa85MInSTMgqojzamtYhRoFrnjVV8T76yC0bPiVUb/BcUsyipbUO+JHrS0nCiB0m7FqER8BzUOYxdZ1+HFBdhwX5SeNH1daZjq7G+oy7QJeHT0KeVOkdohmWW2pQ0kJMqa5eX5tNBn/lcfEB+LGKPSUZ6yNZ8m33sBVvoAaNKptL/6be4n/kgEayAOaP5I1gFyJj8ov4MKcrPmXznhBhyhDI137r6BLnN1q9+wWBb+CMCLkLTUKjOA9KYwRQVQE9z18BxiWImictQ4o0mV16b8kvemCEiP+cZCzN9INipNsfibjjFQj6XjMHIJIPOcRAosyHksj58nIu05X+D05ThcOmy97FYG9ei8EYA9VsoJ6JEjHr/iT1phL8e0v20WELwJ427S7te8r2lJQWhzQW4mwfTSiDnPvIcrg0sb2 SigmuIrr MK0PcToIbrDl3lQMVR0tmJrImBsatofO3FWKQfV07CrWBfX0W8TzmPMK8W4AmBUR35cmZI+2q4BW2yPo1woiGfLK1V1fxEYfeuvrMpIPEiSv3DjaO1YO2l0MGNHYrxt3VSR0lBDIot2q7argUb39Xz8Y7aGalxlXiCwXDxekxisbLG4iixMPO62LtpOsgwSEkRcvgHlo38JNwQJFuYR0/R0VLVlJSsudYSMj7ze6Pn1xo4kjG26EL/dj5pqILMvyXPtSQeE+SUYrehWRCwdNGb2lCh45Qvc1mwOtX5rlP1cKRE12sGJKHCNx+SpR4XdCGqn73 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: David Hildenbrand writes: > On 13.07.23 21:12, Jeff Moyer wrote: >> David Hildenbrand writes: >> >>> On 16.06.23 00:00, Vishal Verma wrote: >>>> The dax/kmem driver can potentially hot-add large amounts of memory >>>> originating from CXL memory expanders, or NVDIMMs, or other 'device >>>> memories'. There is a chance there isn't enough regular system memory >>>> available to fit ythe memmap for this new memory. It's therefore >>>> desirable, if all other conditions are met, for the kmem managed memory >>>> to place its memmap on the newly added memory itself. >>>> >>>> Arrange for this by first allowing for a module parameter override for >>>> the mhp_supports_memmap_on_memory() test using a flag, adjusting the >>>> only other caller of this interface in dirvers/acpi/acpi_memoryhotplug= .c, >>>> exporting the symbol so it can be called by kmem.c, and finally changi= ng >>>> the kmem driver to add_memory() in chunks of memory_block_size_bytes(). >>> >>> 1) Why is the override a requirement here? Just let the admin >>> configure it then then add conditional support for kmem. >>> >>> 2) I recall that there are cases where we don't want the memmap to >>> land on slow memory (which online_movable would achieve). Just imagine >>> the slow PMEM case. So this might need another configuration knob on >>> the kmem side. >> >> From my memory, the case where you don't want the memmap to land on >> *persistent memory* is when the device is small (such as NVDIMM-N), and >> you want to reserve as much space as possible for the application data. >> This has nothing to do with the speed of access. > > Now that you mention it, I also do remember the origin of the altmap -- > to achieve exactly that: place the memmap on the device. > > commit 4b94ffdc4163bae1ec73b6e977ffb7a7da3d06d3 > Author: Dan Williams > Date: Fri Jan 15 16:56:22 2016 -0800 > > x86, mm: introduce vmem_altmap to augment vmemmap_populate() > In support of providing struct page for large persistent memory > capacities, use struct vmem_altmap to change the default policy for > allocating memory for the memmap array. The default vmemmap_populate= () > allocates page table storage area from the page allocator. Given > persistent memory capacities relative to DRAM it may not be feasible = to > store the memmap in 'System Memory'. Instead vmem_altmap represents > pre-allocated "device pages" to satisfy vmemmap_alloc_block_buf() > requests. > > In PFN_MODE_PMEM (and only then), we use the altmap (don't see a way to > configure it). Configuration is done at pmem namespace creation time. The metadata for the namespace indicates where the memmap resides. See the ndctl-create-namespace man page: -M, --map=3D A pmem namespace in "fsdax" or "devdax" mode requires allocation= of per-page metadata. The allocation can be drawn from either: =C2=B7 "mem": typical system memory =C2=B7 "dev": persistent memory reserved from the namespace Given relative capacities of "Persistent Memory" to "Sys= tem RAM" the allocation defaults to reserving space out of t= he namespace directly ("--map=3Ddev"). The overhead is 64-b= ytes per 4K (16GB per 1TB) on x86. > BUT that case is completely different from the "System RAM" mode. The mem= map > of an NVDIMM in pmem mode is barely used by core-mm (i.e., not the buddy). Right. (btw, I don't think system ram mode existed back then.) > In comparison, if the buddy and everybody else works on the memmap in > "System RAM", it's much more significant if that resides on slow memory. Agreed. > Looking at > > commit 9b6e63cbf85b89b2dbffa4955dbf2df8250e5375 > Author: Michal Hocko > Date: Tue Oct 3 16:16:19 2017 -0700 > > mm, page_alloc: add scheduling point to memmap_init_zone > memmap_init_zone gets a pfn range to initialize and it can be > really > large resulting in a soft lockup on non-preemptible kernels > NMI watchdog: BUG: soft lockup - CPU#31 stuck for 23s! > [kworker/u642:5:1720] > [...] > task: ffff88ecd7e902c0 ti: ffff88eca4e50000 task.ti: ffff88eca4e500= 00 > RIP: move_pfn_range_to_zone+0x185/0x1d0 > [...] > Call Trace: > devm_memremap_pages+0x2c7/0x430 > pmem_attach_disk+0x2fd/0x3f0 [nd_pmem] > nvdimm_bus_probe+0x64/0x110 [libnvdimm] > > > It's hard to tell if that was only required due to the memmap for these d= evices > being that large, or also partially because the access to the memmap is s= lower > that it makes a real difference. I believe the main driver was the size. At the time, Intel was advertising 3TiB/socket for pmem. I can't remember the exact DRAM configuration sizes from the time. > I recall that we're also often using ZONE_MOVABLE on such slow memory > to not end up placing other kernel data structures on there: especially, > user space page tables as I've been told. Part of the issue was preserving the media. The page structure gets lots of updates, and that could cause premature wear. > @Dan, any insight on the performance aspects when placing the memmap on > (slow) memory and having that memory be consumed by the buddy where we fr= equently > operate on the memmap? I'm glad you're asking these questions. We definitely want to make sure we don't conflate requirements based on some particular technology/implementation. Also, I wouldn't make any assumptions about the performance of CXL devices. As I understand it, there could be a broad spectrum of performance profiles. And now Dan can correct anything I got wrong. ;-) Cheers, Jeff