From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1089E33F8DC for ; Mon, 8 Jun 2026 23:37:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780961838; cv=none; b=IxRltIxZT/v7WfXUc1Fb71JHCk2FWIOwwcP6o2Ffxhn9Cxq+X4ianjbw/AbYCDY41w6xLGSRx0TurAB7Q81M4grBlSwyLm7f5E+0l9B54vJhVacaxN+9p9FuCwCF0+Z236QRVXAloeHm+jMmKX8bA1XYZ5S3oLP9QlYal6h/oC4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780961838; c=relaxed/simple; bh=DdM8CjuiXZU12OKpECTKcW4MnH6tcaYyclNnnN6UoAA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=H/jDT4w09b46Fk8D7XIXufLRky6aXmrBPeEvvWDneyoi2EPpsKWHU2Dhdh8Sx/fDw7D4d69x8ePkgu/3YB0I9EWMpXVX0u39Frb53/Y9u4PfswTBmnXJJmGiE68bW3Tr8AVFFakgoehxLDaoYYT5nhl7KT92g20kk16PsmJhxto= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=exPJSJni; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="exPJSJni" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2bf20f6be6bso37459015ad.3 for ; Mon, 08 Jun 2026 16:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780961836; x=1781566636; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=DpzGpcAQ40PkxnKmTZfDAjxtQ+CCACqbEdWJNIRNZ7c=; b=exPJSJni8RfKCrUb6m+o1/u4SwofFgznw7sgRDNF6O+z5RegRA2reYb8U3mWF5nUdG QcwjS5Gv7o0ZDZhbjLFpbF+j+X1hGgbGkL8oB8IWtUe5OFP+dIJFUBydN/mI3KsB4SNf 3VqLG9gpA2FGoL+kdlRkX0g0aWK7RsWmzwCETBgeitZRaujMpqgHmQVAO3GoKcGOwDpT +UFBt02BrrhqaAKyDlQ91da6WgsHWaEu7Bjwev+xFthshLipYaZj7/iYyt8pgSIwrxQS o40cYg3S8MAxsZ5gMwqfB4geYPAcXO1DjJglTPIuVQg8DPZ875TmsTLrgbs7OJWMtucq zFig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780961836; x=1781566636; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DpzGpcAQ40PkxnKmTZfDAjxtQ+CCACqbEdWJNIRNZ7c=; b=dkCLQVhnPew7zZO7RlGg5xc+4w5heRL5u4zbEUK5WeeRTZeYX+nuwzPUBU8goSHtbT VGigzpD5Rfwwqos/uD1tdZKpKjxQB+Ej8UE7wWv2a/928s2iF+Z79ygTNenrLfw7Olan AqDrZkTMeX1F6M1b1taVDn47MqA6K0P8MKuHDThl55bn9v1VhxSVOS3+JvJY5u6+qdZu bOqC7ECvv3ziPsYA9PYqxal901HxbO6Senar/bIDoeCIil02u4uqGpygd7ojzMyXylra 3PXNy6VJv6wXp229cNK3T0ie1mzw0BTt3ePO7lhuxruZwsRPVKL5/MQoqCV7VhEhmBNx TMYQ== X-Forwarded-Encrypted: i=1; AFNElJ+tGoBVtZFT58GIb6gekIWKkSTDA9WpxqglNgVLbstn5NE14ozb0mU4OMyFFD3cGNDy6bTQtmaq6b+EiLI=@vger.kernel.org X-Gm-Message-State: AOJu0Yxy4DmRoPU5rFaPsbM9h4pWxKn5AjqJJgQgC+0a2U2MeSXy+wfU uyEVK3splkjJ6Nwli9Fw569elpAd3pyeXyEDotm2dOwZnRGGth+2JKg2/eFjPn6nBA== X-Gm-Gg: Acq92OFUbvUjMAwxIxcsim0+FsNPP3+V8kq86bTb+Ip2+ba2EzOJCQycVhRECKtzCnz R3wfz5Q3nPhLsCmk7nXr0xZz2SC09gOOrOYfoj2kfIG769XlO84RaK95KBnHh/yeY7geJg9ULB8 t4FsNkVUXQQWuJPlYU9y62N1fbvSpjOBLSTJjnGCID22McEt4Z5Tut2MNhW3ayL4yFaO6SD4xkb w11EVC6iQCkPxWKbBW8lax7nd6sqI2iRRFJpBuolbtDHS9OIwQxcFGZA5p6hicm37GcxzuaN+My ue/vsoHoJ7DsF08JOCnMIDaBaDQO5ylBDq2eN9qqfU0ZJojhQWsR6peg5t/YlbwZFkSd2l03mzn N0FyIp8tJogj4ieXD4Mr5Ck6ly5INYFpKV9NQ+x5bwaGG211ECUZ6no4CIVlCsnpkvUGkUOaeID 5IopUmO+AKZRHruU6mFv9jlDpfd+3GDmYHqclvQl6wTvD8X85wVPKhk03Em/jJp3DvEyO4u5Uz X-Received: by 2002:a17:902:da4b:b0:2c1:f262:494d with SMTP id d9443c01a7336-2c1f2624a70mr173213615ad.19.1780961835934; Mon, 08 Jun 2026 16:37:15 -0700 (PDT) Received: from google.com (56.149.168.34.bc.googleusercontent.com. [34.168.149.56]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c164f9f358sm190982245ad.30.2026.06.08.16.37.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Jun 2026 16:37:15 -0700 (PDT) Date: Mon, 8 Jun 2026 23:37:12 +0000 From: David Matlack To: Pratyush Yadav Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Pasha Tatashin Subject: Re: [PATCH 1/2] liveupdate: Reference count outgoing FLB data Message-ID: References: <20260528174140.1921129-1-dmatlack@google.com> <20260528174140.1921129-2-dmatlack@google.com> <2vxzfr34dfty.fsf@kernel.org> <2vxzse6xt8rj.fsf@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2vxzse6xt8rj.fsf@kernel.org> On 2026-06-08 04:19 PM, Pratyush Yadav wrote: > On Tue, Jun 02 2026, David Matlack wrote: > > > On 2026-06-02 07:15 PM, Pratyush Yadav wrote: > >> Hi David, > >> > >> On Thu, May 28 2026, David Matlack wrote: > >> > >> > Increment the outgoing FLB refcount in liveupdate_flb_get_outgoing() so > >> > that the FLB structure cannot be freed while the caller is actively > >> > using it. Add an additional liveupdate_flb_put_outgoing() function so > >> > the caller can explicitly indicate when it is done using the outgoing > >> > FLB. > >> > > >> > During a Live Update, the kernel may need to fetch the outgoing FLB > >> > outside of the scope of a file handler's preserve() and unpreserve() > >> > callbacks. In that situation there is no way for the caller to protect > >> > itself against the outgoing FLB from being freed while it is using it. > >> > Incrementing the reference count in liveupdate_flb_get_outgoing() > >> > ensures it cannot be freed. > >> > >> We grab a reference to the FLB's module when the first file using the > >> FLB is preserved. So the FLB should never go away while preserved files > >> exist. Once all preserved files go away, you normally shouldn't be doing > >> anything with the FLB anyway. > >> > >> Can you please elaborate on the use case and why this is a problem? > >> Using the FLB outside of the standard LUO file callbacks sounds > >> problematic. > > > > The scenario I had in mind was to remove a PCI device from the outgoing > > FLB if the device is forcibly removed while the file is still preserved, > > for example someone writes 1 to /sys/bus/pci/devices/.../remove or a > > device is physically hot-unplugged. > > > > Specifically this call here from the patch below: > > > > +void pci_liveupdate_cleanup_device(struct pci_dev *dev) > > +{ > > + /* > > + * It should be safe to READ_ONCE() outside of the rwsem during cleanup > > + * since there should no longer be any references to @dev on the system. > > + */ > > + if (READ_ONCE(dev->liveupdate.outgoing)) { > > + pci_WARN(dev, 1, "Destroying outgoing-preserved device!\n"); > > + pci_liveupdate_unpreserve(dev); > > + } > > +} > > > > https://lore.kernel.org/linux-pci/20260522202410.3104264-3-dmatlack@google.com/ > > > > I can do this without adding reference counting to > > liveupdate_flb_get_outgoing(), but the reference counting makes it > > obvious that the outgoing FLB will not be freed while I am using it > > here, and also aligns with liveupdate_flb_get_incoming(). > > The lifecycle of FLB is bound to _preserved_ files. So it is only valid > as long as preserved files exist. So I think you should only get the FLB > object when you are inside a file preservation callback for a file which > the FLB is registered. Anywhere outside of that, you are not guaranteed > to get anything sane. LUO should enforce this then, IMO. > This refcounting scheme breaks the inherent "file-lifecycle-bound" part > of FLB, since now anyone can grab a reference and hold the FLB as long > as they like, even when no preserved files exist. > > For the normal case, your the VFIO driver gets probed, it registers its > file handler, then when the device is preserved by VFIO, the VFIO file > handler's callbacks can get the FLB and do whatever. LUO guarantees the > FLB exists. Anywhere outside of that, you should _not_ touch the FLB > because of the reasons above. > > Now for hot-unplug, I think that case is not supported right now. When a > preserved file exists, LUO can only remove it when the user closes the > session. Trying to clean up the file from any other context will leave > dangling references to the file and we currently do not handle those. > Trying to hold the file reference won't help much either since LUO > callbacks will try to proceed as normal, and normal no longer applies. > > For example, say userspace preserved the file for your device in their > session, then you hot-unplug the device, then userspace triggers a > kexec. What is the freeze() callback supposed to do? Sure, the FLB > object still exists, but the device doesn't. Similarly, if you force > remove the module, the freeze() callback itself no longer exists, and > you likely get a panic. > > We might at some point support "invalidating" preserved files. I imagine > when you hot-unplug with a preserved device, you tell LUO to invalidate > all preserved files with that device. They would still exist in their > sessions, but all operations on them fail immediately, including > freeze(), which prevents live update from proceeding until user cleans > them up. > > So unless I am missing something, I think this refcounting is a band-aid > and the real problem is to properly track these "invalidated" files. > > Also, I think the refcounting on the incoming path is also a mistake. > Unfortunately for incoming, there is a need for accessing the FLB > outside of the file handling callbacks, since subsystems needs to use it > to initialize itself. But I suppose we can have a accessor that > subsystems can call once on boot/init to get their object. Then they use > it to initialize their state and refer to the state directly, with all > later calls going through the usual file handler callbacks. > > If you are interested in solving this problem, we can have a chat to > talk in more detail, or perhaps have a discussion at one of the > bi-weeklies? Thanks for the detailed reply but I think it's hard to discuss all these as theoretical situations since we can get bogged down in the parts that aren't clear yet and potential future use-cases. Can you review the use of the outgoing and incoming FLB in the PCI core series and let me know what you think I am doing wrong? https://lore.kernel.org/linux-pci/20260522202410.3104264-1-dmatlack@google.com/ > > > > >> > > >> > This change also aligns the outgoing FLB lifecycle management with the > >> > incoming FLB, since the latter uses the same get/put semantics. > >> > > >> > Fixes: cab056f2aae7 ("liveupdate: luo_flb: introduce File-Lifecycle-Bound global state") > >> > Assisted-by: Gemini:gemini-3-pro-preview > >> > Signed-off-by: David Matlack > >> [...] > >> > >> -- > >> Regards, > >> Pratyush Yadav > > -- > Regards, > Pratyush Yadav