From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3FD63A0B25 for ; Thu, 8 Jan 2026 07:31:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767857481; cv=none; b=p3pG2YApstj5jNhXIj0yiGcivq+FpdLTC6OL8+LvtBL75lFk05Qf+yT6Sh/zla68rDPVf7D0ikDd/5OTnkRku3c9SROw9lXFf18LS28RK5GRonH/x8Uy7NQETd9Z790+yAn0QcF5XKTaemMdJv+qo94dnDprYllGDCcClpOcJU4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767857481; c=relaxed/simple; bh=oB0v3TuDQ6bEbwD5Trw3RAt659wtyH7eNQ1hZKZdfPE=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=KDXnh4WHyF3HxYLNcDprV5wRH0bPXvedUIE3EHfQzh+pDo3Ycs/yRRSUpwJJljJaoZd9/W5elKLgu2ExYKZFwcY4ejj91UTop3xJJZUA/aR1I4jYENTBdLayLrQCZXnTjwv91plof4+BIgFDzGDBcL6DG4gtHbBMkiQwvUp4xRM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=aG1hwWgk; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=SUxHWWwd; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=aG1hwWgk; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=SUxHWWwd; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="aG1hwWgk"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="SUxHWWwd"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="aG1hwWgk"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="SUxHWWwd" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id D48733379E; Thu, 8 Jan 2026 07:31:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1767857477; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vWL6JyaCk8ySJrUMJVEuBTQ9IddRkD6tfWdzwFJQ0C0=; b=aG1hwWgk1fqb8Fz8V1d9uQGqJuCZemXKEySSlJbIGsERbUYjp0R1KbsJa4SPy9WbvKl+RI /6nyvI9HR/4uIFWE/akhM4IBYJtm0MEzqtWsDZ/nEoXDHuOz04z+PzU0VtOgmAxEJuulDc uKsw0ydVAfexGRCOnK72mxZtAM8Bci8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1767857477; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vWL6JyaCk8ySJrUMJVEuBTQ9IddRkD6tfWdzwFJQ0C0=; b=SUxHWWwd81XOdH8wjDreTVSH6K687PLkg2U6WzWa3N/Q6jj96c9dPLcvImVuEZ0lI0HSX1 18vdfgSgCYquHdDg== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=aG1hwWgk; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=SUxHWWwd DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1767857477; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vWL6JyaCk8ySJrUMJVEuBTQ9IddRkD6tfWdzwFJQ0C0=; b=aG1hwWgk1fqb8Fz8V1d9uQGqJuCZemXKEySSlJbIGsERbUYjp0R1KbsJa4SPy9WbvKl+RI /6nyvI9HR/4uIFWE/akhM4IBYJtm0MEzqtWsDZ/nEoXDHuOz04z+PzU0VtOgmAxEJuulDc uKsw0ydVAfexGRCOnK72mxZtAM8Bci8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1767857477; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vWL6JyaCk8ySJrUMJVEuBTQ9IddRkD6tfWdzwFJQ0C0=; b=SUxHWWwd81XOdH8wjDreTVSH6K687PLkg2U6WzWa3N/Q6jj96c9dPLcvImVuEZ0lI0HSX1 18vdfgSgCYquHdDg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 605753EA63; Thu, 8 Jan 2026 07:31:17 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id vB27FUVdX2k2KQAAD6G6ig (envelope-from ); Thu, 08 Jan 2026 07:31:17 +0000 Message-ID: <20baab84-c8b0-4c46-a550-21b26b975d07@suse.de> Date: Thu, 8 Jan 2026 08:31:16 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] memory,memory_hotplug: allow restricting memory blocks to zone movable To: "David Hildenbrand (Red Hat)" , Gregory Price Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, osalvador@suse.de, gregkh@linuxfoundation.org, rafael@kernel.org, dakr@kernel.org, akpm@linux-foundation.org, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com References: <20260105203611.4079743-1-gourry@gourry.net> <7f053290-6b9a-4d18-936e-0f28006c79c3@kernel.org> <9575e042-39f4-4f01-80db-34aaaa9312e6@kernel.org> <616f97b7-24e0-4134-a08d-5abaf07a8b09@kernel.org> Content-Language: en-US From: Hannes Reinecke In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Flag: NO X-Spam-Score: -4.51 X-Rspamd-Queue-Id: D48733379E X-Spamd-Result: default: False [-4.51 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-0.999]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_VIA_SMTP_AUTH(0.00)[]; FUZZY_RATELIMITED(0.00)[rspamd.com]; ARC_NA(0.00)[]; RCPT_COUNT_TWELVE(0.00)[16]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:dkim,suse.de:email,imap1.dmz-prg2.suse.org:rdns,imap1.dmz-prg2.suse.org:helo]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.de:+] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Action: no action X-Spam-Level: On 1/6/26 21:22, David Hildenbrand (Red Hat) wrote: > On 1/6/26 20:59, Gregory Price wrote: >> On Tue, Jan 06, 2026 at 07:38:54PM +0100, David Hildenbrand (Red Hat) >> wrote: >>> On 1/6/26 19:06, Gregory Price wrote: >>>> On Tue, Jan 06, 2026 at 06:52:11PM +0100, David Hildenbrand (Red >>>> Hat) wrote: >>>>> On 1/6/26 17:58, Gregory Price wrote: >>>> >>>> Fair, I'll revist this once Hannes gets a chance to chime in. >>>> >>>> This was effective at getting the discussion started though :P >>> >>> Hehe, yes. >>> >>> Another thing to look into would be to provide a way for ndctl to just >>> add+online the memory in one shot, without having to go back to walking >>> memory blocks to online them etc. >>> >> >> I think it's the opposite: offline+remove needing to be done in one step >> while holding the hotplug lock.  Right now, I think you have to do >> something like > > That's what I note below, yes. > > For the udev vs. ndctl race to be handled in a > good way you need add+online be done in one operation. > >> >> daxctl offline-memory ... >> daxctl destroy ... >> >> You can't destroy and have it offline the memory for you in one go IIRC. > > As noted below, we have offline_and_remove_memory(). > > I added the comment: > > /* >  * Try to offline and remove memory. Might take a long time to finish > in case >  * memory is still in use. Primarily useful for memory devices that > logically >  * unplugged all memory (so it's no longer in use) and want to offline > + remove >  * that memory. >  */ > > Nothing speaks against letting dax use that, but the tricky part is that > offlining might take forever, so one has to be prepared to handle that > (and letting user space cancel the operation). > > And for dax devices that consist of multiple ranges, it can be "fun" having > some regions removed and others not. > > Something to think about :) > We had this discussion at LPC. The current interface of having to individually offline every single memory block is not very user-friendly. While it provides the best possible granularity, it really only makes sense for virtual environments where you _can_ hotplug individual blocks. For hardware-based scenarios memory will always be removed in larger entities (eg the CXL device), and it's always an 'all-or-nothing' scenario; you cannot remove individual memory blocks on a CXL device. So there the memory block abstraction makes less sense, and it would be good to have a single 'knob' to remove the entire CXL device and all memory blocks on it. Sure, it might take some time, but one doesn't need to worry about restoring the original state if the operation on one block fails. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich