Re: iommu/virtio: design question: hot-removal allowed?

[Jean-Philippe Brucker <jpb@kernel.org>]

Hi Cam,

Thanks for investigating this

On Thu, Nov 05, 2026 at 03:54:20PM -0500, Cam Miller wrote:
> Hi,
> 
> 
> Testing on x86, I noticed that userspace supplies a mechanism for hot-removing a virtio-iommu device.
> 
>   echo 1 > /sys/bus/pci/devices/<virtio_iommu_dev>/remove
> 
> Hot-removing virtio-iommu should NOT be supported, right? The intention is clear in the QEMU source,
> which fences off hot(un)plugging virtio-iommu devices.

We would also have to prevent:

* unbinding the driver. For that we can use .suppress_bind_attrs

* unbinding the transport driver: virtio-pci, virtio-mmio

* possibly other weird cases like PCI bridges upstream of the IOMMU
  getting removed

Module removal is fine because the IOMMU core holds a module ref for
each endpoint.

> If I am correct, then shall we fence off hot-removal (perhaps by simply omitting this remove file
> from sysfs)?

That would be a PCI core change, I guess?

I'd much prefer handling forced removal as gently as possible in the
viommu driver, but as you've noticed, we're not there.

> Else, keep reading...
> 
> ...I believe I have found two bugs in viommu_remove() path that I would like to bring attention to.
> The first bug is that the echo above causes the following kernel panic.

Yes please send a patch

[...]
> After applying the fix shown above, a second problem is exposed. Unregistering the IOMMU while
> endpoints remain active is like pulling out the rug from underneath those endpoints. The following
> warning catches this case.
> 
>   WARNING: drivers/iommu/iommu.c:743 at __iommu_group_free_device.constprop.0+0xb9/0xe0, CPU#1:
> bash/1250
>   --- snip ---
>   Call Trace:
>    <TASK>
>    __iommu_group_remove_device+0x95/0xe0
>    ? __pfx_remove_iommu_group+0x10/0x10
>    remove_iommu_group+0x39/0x60
>    bus_for_each_dev+0x85/0xd0
>    iommu_device_unregister+0x3d/0xb0
>    viommu_remove+0x1d/0x60
>   --- snip ---

Device links would likely prevent this sort of thing. I worked on it a
while ago but at the time I was hoping the new fw_devlink would
automatically create those links. Unfortunately there was (is?) at least
a problem with having the IOMMU on the same bus as the endpoints. So we
should add explicit device_link_add() in the driver, to force endpoint
unbind before viommu unbind.

> ...Point being that hot-removing virtio-iommu is currently broken.
> 
> In summary:
> - Should hot-removal be supported/allowed?

Reluctantly allowed, I think

> - If no, then how shall we fence it off? Via sysfs?
> - Else, there are some bugs that need to be fixed (and I would be happy to work on that if that is
> not already evident).

That would be really welcome!

Thanks,
Jean