Re: [PATCH RFC] net/mlx5: check whether VFs are assigned before disabling SR-IOV

["Boone, Max" <mboone@akamai.com>]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]



> On Apr 29, 2026, at 2:38 PM, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> 
> !-------------------------------------------------------------------|
>  This Message Is From an External Sender
>  This message came from outside your organization.
> |-------------------------------------------------------------------!
> 
> On Tue, Apr 28, 2026 at 08:04:14PM +0200, Max Boone via B4 Relay wrote:
>> From: Max Boone <mboone@akamai.com>
>> 
>> When MLX5 cards are passed through to a VM, disabling SR-IOV by
>> setting the sriov_numvfs to 0 will render the machine unstable.
> 
> What? How does that happen?

Unstable is maybe a bit confusing phrasing on my part, “locks up”
might be a better description?

In short:
- Enable by setting sriov_numvfs to positive
- vfio-pci passthrough to QEMU (or other process)
- Disable by setting sriov_numvfs to zero
- QEMU processes freeze, shell that was writing to sysfs freezes
- SIGKILL doesn’t seem to have much effect, shutdown never completes

Python script to reproduce without QEMU:
- https://github.com/akamaxb/repro-vfio-sriov-removal/blob/main/vfio-sriov-bind.py

Does:
  1. Require sriov_numvfs == 0 on the PF (report any existing users and exit if not)
  2. Add one SR-IOV VF
  3. Bind the VF to vfio-pci via driver_override + drivers_probe
  4. Open VFIO container + group, get device fd
  5. Create a KVM VM (registers an MMU notifier — required to trigger the race)
  6. Hold and wait for user input

To trigger the bug while the script is waiting, in another terminal:
    echo 0 > /sys/bus/pci/devices/<pf_device>/sriov_numvfs

On the vfio-pci end of it all, it prints these two lines to dmesg before it hangs:
- https://elixir.bootlin.com/linux/v7.0.1/source/drivers/vfio/pci/vfio_pci_core.c#L1826
- https://elixir.bootlin.com/linux/v7.0.1/source/drivers/vfio/vfio_main.c#L421

>> -void mlx5_sriov_disable(struct pci_dev *pdev, bool num_vf_change)
>> +int mlx5_sriov_disable(struct pci_dev *pdev, bool num_vf_change)
>> {
>> struct mlx5_core_dev *dev  = pci_get_drvdata(pdev);
>> struct devlink *devlink = priv_to_devlink(dev);
>> int num_vfs = pci_num_vf(dev->pdev);
>> 
>> + if (pci_vfs_assigned(dev->pdev)) {
>> + mlx5_core_warn(dev, "can't disable sriov, VFs are assigned\n");
>> + return -EPERM;
>> + }
> 
> *barf* WTF did this come from?

Hahaha, take your pick:
- https://elixir.bootlin.com/linux/v7.0.1/C/ident/pci_vfs_assigned

I followed the sysfs sriov_numvfs op for a couple drivers and saw
that ixgbe (and others) had it plumbed in, so presumed (sorry)
that this would fix it / was an obvious omission if the rest is doing
it. My bad for cargo culting an artifact from Xen.

> Grep says only Xen makes this true, so this is all working around some
> Xen brokenness in their "assignment" ?

Yeap, I see, looks like it.

> If people care about Xen pci_is_dev_assigned() should be be purged and
> pciback should be fixed to not "make the machine unstable" when it is
> removed during a VF teardown.
> 
> Or at the very least this nasty Xen intrustion should be placed in the
> PCI core code and removed from the drivers.
> 
> Also, no, you can't fail mlx5_sriov_disable() it is called during
> driver remove and cannot fail in that flow.

Check. I can do some further digging and build a kernel with lockdep
to try and find what it is hanging on specifically. Unless something pops
to mind?

> 
> Jason


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3061 bytes --]