Re: [PATCH 1/2] liveupdate: Reference count outgoing FLB data

[Pratyush Yadav <pratyush@kernel.org>]

On Tue, Jun 09 2026, Pasha Tatashin wrote:

> On 06-08 23:37, David Matlack wrote:
>> On 2026-06-08 04:19 PM, Pratyush Yadav wrote:
>> > On Tue, Jun 02 2026, David Matlack wrote:
>> > 
>> > > On 2026-06-02 07:15 PM, Pratyush Yadav wrote:
>> > >> Hi David,
>> > >> 
>> > >> On Thu, May 28 2026, David Matlack wrote:
>> > >> 
>> > >> > Increment the outgoing FLB refcount in liveupdate_flb_get_outgoing() so
>> > >> > that the FLB structure cannot be freed while the caller is actively
>> > >> > using it. Add an additional liveupdate_flb_put_outgoing() function so
>> > >> > the caller can explicitly indicate when it is done using the outgoing
>> > >> > FLB.
>> > >> >
>> > >> > During a Live Update, the kernel may need to fetch the outgoing FLB
>> > >> > outside of the scope of a file handler's preserve() and unpreserve()
>> > >> > callbacks. In that situation there is no way for the caller to protect
>> > >> > itself against the outgoing FLB from being freed while it is using it.
>> > >> > Incrementing the reference count in liveupdate_flb_get_outgoing()
>> > >> > ensures it cannot be freed.
>> > >> 
>> > >> We grab a reference to the FLB's module when the first file using the
>> > >> FLB is preserved. So the FLB should never go away while preserved files
>> > >> exist. Once all preserved files go away, you normally shouldn't be doing
>> > >> anything with the FLB anyway.
>> > >> 
>> > >> Can you please elaborate on the use case and why this is a problem?
>> > >> Using the FLB outside of the standard LUO file callbacks sounds
>> > >> problematic.
>> > >
>> > > The scenario I had in mind was to remove a PCI device from the outgoing
>> > > FLB if the device is forcibly removed while the file is still preserved,
>> > > for example someone writes 1 to /sys/bus/pci/devices/.../remove or a
>> > > device is physically hot-unplugged.
>> > >
>> > > Specifically this call here from the patch below:
>> > >
>> > > +void pci_liveupdate_cleanup_device(struct pci_dev *dev)
>> > > +{
>> > > +	/*
>> > > +	 * It should be safe to READ_ONCE() outside of the rwsem during cleanup
>> > > +	 * since there should no longer be any references to @dev on the system.
>> > > +	 */
>> > > +	if (READ_ONCE(dev->liveupdate.outgoing)) {
>> > > +		pci_WARN(dev, 1, "Destroying outgoing-preserved device!\n");
>> > > +		pci_liveupdate_unpreserve(dev);
>> > > +	}
>> > > +}
>> > >
>> > > https://lore.kernel.org/linux-pci/20260522202410.3104264-3-dmatlack@google.com/
>> > >
>> > > I can do this without adding reference counting to
>> > > liveupdate_flb_get_outgoing(), but the reference counting makes it
>> > > obvious that the outgoing FLB will not be freed while I am using it
>> > > here, and also aligns with liveupdate_flb_get_incoming().
>> > 
>> > The lifecycle of FLB is bound to _preserved_ files. So it is only valid
>
> An FLB is a shared global resource whose lifecycle is bound to the 
> duration of first creation and final destruction across multiple files. 
> It is not associated with any single preserved file, but rather with the 
> fact that files of a particular type(s) are currently being preserved.
>
> Because the kernel is highly asynchronous, we must handle concurrent 
> access and race conditions during teardown and FLB lookups. Reference 
> counting provides a standard, robust, and methodical way to handle this 
> safely.

But that's my point. As you say, destruction is supposed to happen when
the last file using the FLB goes away. That stops being true when any
random caller can hold a reference. So now you can get a FLB with no
files. It fundamentally breaks the model we started with FLBs.

The FLB model also deals with asynchronicity, but it does so by managing
the lifetime of the FLB directly in LUO instead of leaving it to
subsystems. This reduces cognitive load on the users of the FLB API.
They do not need to care about managing the lifecycle themselves.

If users of FLB are directly managing references, then why does LUO even
need to manage anything? Then subsystems can just create their own
global data structure, grab a reference and release it as they please.
That is a perfectly valid way of managing this global object, but we
decided to _not_ do this because letting LUO manage the lifecycle (and
thus the refcount) results in code that is easier to understand and
reason about.

>
>> > as long as preserved files exist. So I think you should only get the FLB
>> > object when you are inside a file preservation callback for a file which
>
> Limiting access to only within a file preservation callback is not 
> feasible. FLBs are global resources and may be queried early during boot 
> or late during teardown, completely outside of a specific file's 
> preservation callback.

To clarify, I don't mean only the preservation callback. I mean during
any of the file handler callbacks.

And if the subsystem needs to do something with the FLB after all its
files are frozen, we can provide a freeze() callback to
liveupdate_flb_ops.

Why do you need to access the FLB anywhere else outside of these
callbacks? What's the use case?

>
>> > the FLB is registered. Anywhere outside of that, you are not guaranteed
>> > to get anything sane.
>> 
>> LUO should enforce this then, IMO.
>> 
>> > This refcounting scheme breaks the inherent "file-lifecycle-bound" part
>> > of FLB, since now anyone can grab a reference and hold the FLB as long
>> > as they like, even when no preserved files exist.
>
> This is the standard design pattern for shared resources in the kernel. 
> When the last file of that type is unpreserved, the FLB's reference 
> count reaches zero, and it is cleanly destroyed.
>
> If there is a bug where someone holds onto a reference indefinitely, 
> that is simply a leak/bug to be fixed. It is not a flaw in the 
> reference-counting pattern itself, which is crucial for preventing 
> use-after-free and race conditions under concurrent access.

Going to my previous point, the whole reason for FLB to exist is that we
don't want to burden subsystems with managing these references. If we do
that, then I think it would be way easier to just get rid of FLBs
entirely. It would be almost as easy for each subsystem to allocate a
global struct with its state and a refcount.

I don't get why we want to mix these approaches. FLB makes my life as a
user of the API easier because I don't need to worry about the
lifetimes. LUO takes care of that for me.

>
>> > For the normal case, your the VFIO driver gets probed, it registers its
>> > file handler, then when the device is preserved by VFIO, the VFIO file
>> > handler's callbacks can get the FLB and do whatever. LUO guarantees the
>> > FLB exists. Anywhere outside of that, you should _not_ touch the FLB
>> > because of the reasons above.
>> > 
>> > Now for hot-unplug, I think that case is not supported right now. When a
>> > preserved file exists, LUO can only remove it when the user closes the
>> > session. Trying to clean up the file from any other context will leave
>> > dangling references to the file and we currently do not handle those.
>> > Trying to hold the file reference won't help much either since LUO
>> > callbacks will try to proceed as normal, and normal no longer applies.
>> > 
>> > For example, say userspace preserved the file for your device in their
>> > session, then you hot-unplug the device, then userspace triggers a
>> > kexec. What is the freeze() callback supposed to do? Sure, the FLB
>> > object still exists, but the device doesn't. Similarly, if you force
>> > remove the module, the freeze() callback itself no longer exists, and
>> > you likely get a panic.
>> > 
>> > We might at some point support "invalidating" preserved files. I imagine
>> > when you hot-unplug with a preserved device, you tell LUO to invalidate
>> > all preserved files with that device. They would still exist in their
>> > sessions, but all operations on them fail immediately, including
>> > freeze(), which prevents live update from proceeding until user cleans
>> > them up.
>> > 
>> > So unless I am missing something, I think this refcounting is a band-aid
>> > and the real problem is to properly track these "invalidated" files.
>> > 
>> > Also, I think the refcounting on the incoming path is also a mistake.
>
> On incoming path FLBs may be accessed during early boot and later 
> including once we enter userspace. We must have a way to ensure valid, 
> race-free access to these shared resources. Without reference counting, 
> querying the existence of an FLB or retrieving it would require 
> introducing global locking, which is highly undesirable.

Ideally, on the incoming path LUO should provide a subsystem init hook
that lets them get their FLB data and use it during init. Unfortunately
it isn't easy because of boot ordering. You might need the FLB before
LUO comes up.

But I think we should still tighten up access to the FLB on the incoming
side as well. You can use the FLB early until it gets registered (or
something similar) with LUO. Then you hand over the ownership to LUO and
can only use it from the standard LUO callbacks. At this point the
access model of incoming and outgoing become the same.

Overall, my point is that we should pick one usage model. Either LUO
manages the lifetime of the object or the subsystem. Mixing both this
way sounds wrong.

>
> Also, having both the incoming and outgoing FLB paths behave 
> symmetrically using the same reference-counting API makes the overall 
> design uniform, clean, and less error-prone.
>
>> > Unfortunately for incoming, there is a need for accessing the FLB
>> > outside of the file handling callbacks, since subsystems needs to use it
>> > to initialize itself. But I suppose we can have a accessor that
>> > subsystems can call once on boot/init to get their object. Then they use
>> > it to initialize their state and refer to the state directly, with all
>> > later calls going through the usual file handler callbacks.
>> > 
>> > If you are interested in solving this problem, we can have a chat to
>> > talk in more detail, or perhaps have a discussion at one of the
>> > bi-weeklies?
>> 
>> Thanks for the detailed reply but I think it's hard to discuss all these
>> as theoretical situations since we can get bogged down in the parts that
>> aren't clear yet and potential future use-cases.
>> 
>> Can you review the use of the outgoing and incoming FLB in the PCI core
>> series and let me know what you think I am doing wrong?
>> 
>>   https://lore.kernel.org/linux-pci/20260522202410.3104264-1-dmatlack@google.com/

Yes, good idea. Will do.

-- 
Regards,
Pratyush Yadav