From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95FC827A462 for ; Fri, 21 Nov 2025 21:07:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763759270; cv=none; b=KArZ0sa42T1m8sgO4VCDB04IgqJjgIM+2DZzi3qIBsBPn/2n7S96KtGQ/V8Xc60bRbs3+Q6VWlB7UoH6rzoho3E4Ufze8mEI6UMLlVeCH7K+FygVAf2BuHEUf2v93ZfFAjkQt3+I7PWpymZyudUFWGe83h0iIpN+abQcpe7ENN8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763759270; c=relaxed/simple; bh=shSHymcVpVsJRS3DnGEq5zOC8PBxp/thlGXIe4I1Liw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=XSIkwuK+N/2zRt0FL0p0Fu/JTHsabCnbOdfwFJe0ws5XNL2+cE5S1lGZSR1ZylEaMa31GE3VV9GRln2cbRBgJMQc2PN1mGbG/PbePlbcbcjgDqjumyQK38yt3PMY1c4W0xqO4zu9UMUd6vs2rIJ+8cBk+DkKpVPh/Kb373qGD3E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=bVeAkH0S; arc=none smtp.client-ip=209.85.160.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="bVeAkH0S" Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-4ee0ce50b95so30302401cf.0 for ; Fri, 21 Nov 2025 13:07:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1763759267; x=1764364067; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Svi+I8ywJ8k5uxtzxqjpEGX4Qh9Fr5IeUT/6ziPIoB4=; b=bVeAkH0Shdp0h8OjS5YQFosmUq1UbuAiSCtBpKGJd649mntc47dWELhSOTZQnaOAjK BCY4D7BXJ5i1hsC1fClTm4PszJCglfGik5ovY+CoJ8ktPXIJO8jpUKKTbvCX2WfqGRw3 bA2H1a0CPlPAyhgwrhn0KSEAyGZt4e+MOmXefgJHOtptodjnXiUQIK0cE/uB+C1MmOj6 9Vv3N5ZkdIweoiAfWLXH0DDPqvoA94Oroxiiu7cs3E8G3STcoHTtPwib7Gxi2uB6ShCe TN0qnP/keU7Y8ly8iP6f6U+hdeJ5CXT5nLs7pWeGnp4edtdM0Sym9I141bXeWTR7F0Zq CZIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763759267; x=1764364067; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Svi+I8ywJ8k5uxtzxqjpEGX4Qh9Fr5IeUT/6ziPIoB4=; b=J4Waa2IK74KmRE3yHFn3TTznOjm/7VMSFiPgwX1gpxeNxmdrGZyb6uJy5RZlBv4GBB PUUCVTWZ38TwUZWPP5owT5WHgOM6dzr7M060XEorSd//tFGgSwCoo8aTTZ8c5jfA1W3q qts9U/KPWqNYmDlTUm+3zW4ABygYC9xbA7+pJfj7F3srGRaizohXzoLFoi5F6qmRS1AY L0Ycsc53vjkmuOiRZ6xNsr/+AV78VsExTaHBE9xXV4swv2SdxoOoZC6tYhpOfszaCCio /BGe6eDiTfCI6FcjVRQUQMaGDzeu6d+Qysa29WzRyW6Yfy3uzYbufAspCnlJKFQfoohV tPew== X-Forwarded-Encrypted: i=1; AJvYcCUBXVuZUpq8QFl+Z33MOVjmoxp5EJY/T6pICbgThZ1eISzgz8Hk17uFbCziUr0YDhHVBvggMIwgw88=@vger.kernel.org X-Gm-Message-State: AOJu0Yw0hy5v59MlcucgzboJjmgh1MEiyfBBf63O4uDjHc/Uycpx8Yod 7lghn8WBHLPkwbfDfinrOvbz+DK2jYJhhnE/N0dZmXIug14rLWaHiPKT7fejkvwilag= X-Gm-Gg: ASbGncttBQW5+Jj+6kov4HBPR9eG+aO7HvgZFdaLrFBFtnXMG7xK4jFzfgrt6zN/Hsk qoBf/uf0zZ6JZ+7DwBqL0Kz5WNWkdDUFJZxiTFZ6Jsktez9aNVRr8UZFeQBp8e4y9tOX5gDYduP /hBAZMWh5vCkfVWRqPA7dolZPY0syoFwCYQPhbu28/vevFbtfLYqugqE0Yq3huv9TDrQMPHQVew 3mcXEGnmMvsZrLScTowndkQn8XLggAPzvtytIE6higCGNG0xsxQfS7B5RW7CV4P+dhJd1aFypuT Sf5oGPyx9pci5WuEB8tUx6gF1u//8M4JjO7XakrzCdnMhIW5nPbLmrA9derBie4IfPMMG149c1u /duaC9dM5Xt7SAjT3GDib3yGyUJm+gXlPPg9BFq8cZBuJFnNxhFQvD1pNdZzbI6BG4pDoWQ78JO C85Mhg8Fvu6plvjyb+QoWPrT6RwLYFZXIP8628HfGvb9JyUft+SeIooltglugVGFMsapxGXQ== X-Google-Smtp-Source: AGHT+IHB54I5/pxWYI0YylL4DbWXryCZhFk7bd7fb71QpMMtzOgbCdkOeFKpCqsEbmreSLGZDE/f8Q== X-Received: by 2002:a05:622a:c6:b0:4eb:a457:394 with SMTP id d75a77b69052e-4ee587d2ef1mr64635291cf.12.1763759267374; Fri, 21 Nov 2025 13:07:47 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-96-255-20-138.washdc.ftas.verizon.net. [96.255.20.138]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4ee48e65e3bsm42420991cf.17.2025.11.21.13.07.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Nov 2025 13:07:46 -0800 (PST) Date: Fri, 21 Nov 2025 16:07:35 -0500 From: Gregory Price To: Alistair Popple Cc: linux-mm@kvack.org, kernel-team@meta.com, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, cgroups@vger.kernel.org, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, longman@redhat.com, akpm@linux-foundation.org, david@redhat.com, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, osalvador@suse.de, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, ying.huang@linux.alibaba.com, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, kees@kernel.org, muchun.song@linux.dev, roman.gushchin@linux.dev, shakeel.butt@linux.dev, rientjes@google.com, jackmanb@google.com, cl@gentwo.org, harry.yoo@oracle.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, fabio.m.de.francesco@linux.intel.com, rrichter@amd.com, ming.li@zohomail.com, usamaarif642@gmail.com, brauner@kernel.org, oleg@redhat.com, namcao@linutronix.de, escape@linux.alibaba.com, dongjoo.seo1@samsung.com Subject: Re: [RFC LPC2026 PATCH v2 00/11] Specific Purpose Memory NUMA Nodes Message-ID: References: <20251112192936.2574429-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Tue, Nov 18, 2025 at 06:02:02PM +1100, Alistair Popple wrote: > > I'm interested in the contrast with zone_device, and in particular why > device_coherent memory doesn't end up being a good fit for this. > > > - Why mempolicy.c and cpusets as-is are insufficient > > - SPM types seeking this form of interface (Accelerator, Compression) > > I'm sure you can guess my interest is in GPUs which also have memory some people > consider should only be used for specific purposes :-) Currently our coherent > GPUs online this as a normal NUMA noode, for which we have also generally > found mempolicy, cpusets, etc. inadequate as well, so it will be interesting to > hear what short comings you have been running into (I'm less familiar with the > Compression cases you talk about here though). > after some thought, talks, and doc readings it seems like the zone_device setups don't allow the CPU to map the devmem into page tables, and instead depends on migrate_device logic (unless the docs are out of sync with the code these days). That's at least what's described in hmm and migrate_device. Assuming this is out of date and ZONE_DEVICE memory is mappable into page tables, assuming you want sparse allocation, ZONE_DEVICE seems to suggest you at least have to re-implement the buddy logic (which isn't that tall of an ask). But I could imagine an (overly simplistic) pattern with SPM Nodes: fd = open("/dev/gpu_mem", ...) buf = mmap(fd, ...) buf[0] 1) driver takes the fault 2) driver calls alloc_page(..., gpu_node, GFP_SPM_NODE) 3) driver manages any special page table masks Like marking pages RO/RW to manage ownership. 4) driver sends the gpu the (mapping_id, pfn, index) information so that gpu can map the region in its page tables. 5) since the memory is cache coherent, gpu and cpu are free to operate directly on the pages without any additional magic (except typical concurrency controls). Driver doesn't have to do much in the way of allocationg management. This is probably less compelling since you don't want general purposes services like reclaim, migration, compaction, tiering - etc. The value is clearly that you get to manage GPU memory like any other memory, but without worry that other parts of the system will touch it. I'm much more focused on the "I have memory that is otherwise general purpose, and wants services like reclaim and compaction, but I want strong controls over how things can land there in the first place". ~Gregory