From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A700D35B13D for ; Mon, 12 Jan 2026 13:45:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768225515; cv=none; b=VK23Aiq8OALN0U5O12taXR8jAcarAfwGAovZ2h1aNQrTzojHO0/2ZCM8LJYFfNK7jBWzqC5vRKQ1ho+FHyrCJSGNPdz1il3vh7XdiWtsoCa9G4MX3CJjtchsI2AN7laxIjGPNeYFcEPji+F/JF28DtzFPRfbyA2mM64b+t+/aqI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768225515; c=relaxed/simple; bh=cUHVEQ4ugeQMuGA4/ocXfnET6Ezlx8k5T0UznoHi1+I=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uo5L6w5dK3yp7z7RjocAl1l2WoHtvOZ7PGWOJKeRMrcHq4xlGdAA4aWWTnjKYyoXtNwgxl9a88de/M3Ud3yHH3L2WPKq69Ns8J2xb3z4lkiEEBXZGNWElm2egF1Nj9YHMFOoQlTtnRat4jbc0hT91yfCScfOlDsiKrgFUysghtc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=GICQ5ZAV; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="GICQ5ZAV" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4ee14ba3d9cso71611141cf.1 for ; Mon, 12 Jan 2026 05:45:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1768225513; x=1768830313; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=oQVWjYpPCtwp4dXF89g7hQK6KDriflVp74tvOjx72Jc=; b=GICQ5ZAV2Gr4ghdUMkl1JWB88WIDy84+ElyztjFepUU/xdAmtbj/1y56tkmQmcKRU0 X+/y+LWb+C0pRWfeXjaugN3pS/SQy0OQjA5gpSLyoW/JwG6f2Y4+JallvDWBEWAyUxDI RJtTlQjGTDES1c1bzMb6fz/FCDYlpegNB4E8dsOLqxDFLzcHLs6d81Y6DWfaiMVbCTIn WOKI9rJVPkWfvcCMD5ava1G93AXb4Wvauh1gWyOKAPCYz+MKp8jYws7itoO2wH4yMX5T u5aDzBEqC15GTvQOHI0XQP5dWjnY8CkNB6hw9eBrF2G1tPMumuT+R0yvDtvL/CGagMWV Rn6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768225513; x=1768830313; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oQVWjYpPCtwp4dXF89g7hQK6KDriflVp74tvOjx72Jc=; b=h713gi7N5hgNHK3/IQKcC9edq55IJgAP7Qk01MLqS9NOOfR1QYU6SAK9LLiGnRiDEw g/K86mpvyyJXIv71CEc96IgbiQ2eAg391kRUBEWlgh0Aj2hNo9IE4d5M8VH2KALGMOeW V9Aq/1nvpuCMFXAQmk2Agc72emHV4tj18FDjNw8R9Ytp8O3kThzeA2hm1jawo1jQdgYm IJjQskriNZUixi+weT9QYGTKGuRFuKcYd/c4Abgpelym9UrLwD6iK4LQZJ2g1eb1faAv rziWLxazH4YJLpCY8OJ2ZHp2+qLVBw3zbi0K5LBas0opWo6qX02EnSLvYQf8L2Xir5PI hTfA== X-Forwarded-Encrypted: i=1; AJvYcCVmfZ2cNz6gTmv3S+67ugXAz0X37vIztlDLzz0ro24LnCAj9W3BBkgLyyxNnah2yDF8cP70GNMuUqQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwVlGovvBQIFDQ7RTjklHlvdP0QqunlUq4P+GaFlstnhBQ0hNzQ v8qxAbR7cAvov70P7BgTWYDf3Y3ZX8NlToIV9B4cctaxZjpqeKLNCC0NLESH5vjdedU= X-Gm-Gg: AY/fxX5c0k5KmJNkUY1fN5danQVso6q5BTztLEe4mxy0mPzegFXrCxTWaMxf1HlT6Jy nT4OYnPlHxKpmgJRmriImAfmoENyTeIIlgx2x0yYB/lyR5J1KdmBeUr6uAb8TCC4EWtsA68AODY cScbtLXvGRBCctJgHZki86Gal+/8cSUwXzoKpjA9Ozm0Y0awWdtVfyPSGbinXIvcTzpEKqz6Q2i KLBdirgpL4MiwYe7N3z9QY6qdUNz+VMXSC7ceo6sUEI5OTklEbw5IVce9EUwRXNjL+yXXPPS9XX Wp6O/1GNmRY4K0JH4fEDE8Bv/iUGH9uJHdYbo+DwKhJkHlYN2Yb2jUdNDP3Ncj8eP63kXPPOPZA 51kD7tNDvpRreCevrZaiWrQAqCOuxbu2cv/j6hjDUhPsr/JRUJcMGjiUSVgHToIgCo1M6dKifLN uClSf/Ojl+JUUaIMkVvKW8IRin687KROBlogsr3o21i/8kz5KR8KEKvJ3QPHjVtXLpn3Ji8yVgt WTxUQ== X-Google-Smtp-Source: AGHT+IG20+wZaR0cdbZzmqo1CbORIU0XkpxQP0kdASvCc3hNxGR2kbLwFGbi9bRoyfQYpJE34s/36Q== X-Received: by 2002:ac8:7f4e:0:b0:4ed:6803:6189 with SMTP id d75a77b69052e-4ffb49998dfmr285683391cf.53.1768225512643; Mon, 12 Jan 2026 05:45:12 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-162-112-119.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.112.119]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4ffa8e36232sm124159891cf.22.2026.01.12.05.45.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jan 2026 05:45:11 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1vfIEI-00000003Q1m-3fo1; Mon, 12 Jan 2026 09:45:10 -0400 Date: Mon, 12 Jan 2026 09:45:10 -0400 From: Jason Gunthorpe To: Zi Yan Cc: Matthew Wilcox , Balbir Singh , Francois Dugast , intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, Matthew Brost , Madhavan Srinivasan , Nicholas Piggin , Michael Ellerman , "Christophe Leroy (CS GROUP)" , Felix Kuehling , Alex Deucher , Christian =?utf-8?B?S8O2bmln?= , David Airlie , Simona Vetter , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , Lyude Paul , Danilo Krummrich , Bjorn Helgaas , Logan Gunthorpe , David Hildenbrand , Oscar Salvador , Andrew Morton , Leon Romanovsky , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Alistair Popple , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, amd-gfx@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org Subject: Re: [PATCH v4 1/7] mm/zone_device: Add order argument to folio_free callback Message-ID: <20260112134510.GC745888@ziepe.ca> References: <20260111205820.830410-1-francois.dugast@intel.com> <20260111205820.830410-2-francois.dugast@intel.com> <874d29da-2008-47e6-9c27-6c00abbf404a@nvidia.com> <0D532F80-6C4D-4800-9473-485B828B55EC@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <0D532F80-6C4D-4800-9473-485B828B55EC@nvidia.com> On Sun, Jan 11, 2026 at 07:51:01PM -0500, Zi Yan wrote: > On 11 Jan 2026, at 19:19, Balbir Singh wrote: > > > On 1/12/26 08:35, Matthew Wilcox wrote: > >> On Sun, Jan 11, 2026 at 09:55:40PM +0100, Francois Dugast wrote: > >>> The core MM splits the folio before calling folio_free, restoring the > >>> zone pages associated with the folio to an initialized state (e.g., > >>> non-compound, pgmap valid, etc...). The order argument represents the > >>> folio’s order prior to the split which can be used driver side to know > >>> how many pages are being freed. > >> > >> This really feels like the wrong way to fix this problem. > >> > > Hi Matthew, > > I think the wording is confusing, since the actual issue is that: > > 1. zone_device_page_init() calls prep_compound_page() to form a large folio, > 2. but free_zone_device_folio() never reverse the course, > 3. the undo of prep_compound_page() in free_zone_device_folio() needs to > be done before driver callback ->folio_free(), since once ->folio_free() > is called, the folio can be reallocated immediately, > 4. after the undo of prep_compound_page(), folio_order() can no longer provide > the original order information, thus, folio_free() needs that for proper > device side ref manipulation. There is something wrong with the driver if the "folio can be reallocated immediately". The flow generally expects there to be a driver allocator linked to folio_free() 1) Allocator finds free memory 2) zone_device_page_init() allocates the memory and makes refcount=1 3) __folio_put() knows the recount 0. 4) free_zone_device_folio() calls folio_free(), but it doesn't actually need to undo prep_compound_page() because *NOTHING* can use the page pointer at this point. 5) Driver puts the memory back into the allocator and now #1 can happen. It knows how much memory to put back because folio->order is valid from #2 6) #1 happens again, then #2 happens again and the folio is in the right state for use. The successor #2 fully undoes the work of the predecessor #2. If you have races where #1 can happen immediately after #3 then the driver design is fundamentally broken and passing around order isn't going to help anything. If the allocator is using the struct page memory then step #5 should also clean up the struct page with the allocator data before returning it to the allocator. I vaugely remember talking about this before in the context of the Xe driver.. You can't just take an existing VRAM allocator and layer it on top of the folios and have it broadly ignore the folio_free callback. Jsaon