Re: 4.14.0-rc3 is auto-reloading PCIe SR-IOV Virtual Function device drivers when VFs are attached to VMs ...

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 17 Oct 2017 18:36:21 +0000
Casey Leedom <leedom@chelsio.com> wrote:

> Hi there,
> 
>   I hope this is the right Linux list for this issue.  One of our QA staff
> has noticed a new behavior starting about 4.14.0-rc3.  We instantiate PCIe
> SR-IOV Virtual Functions and the corresponding driver (cxgb4vf in this case)
> is automatically loaded.  That's fine and has been the case for some time.
> But, we remove the driver module (rmmod cxgb4vf) and then attach one of the
> VFs to a KVM Virtual Machine and the cxgb4vf driver module gets reloaded in
> the KVM Hypervisor.  This is new behavior.  I see that there was a pretty
> big change done by Luis R. Rodriguez in kernel.org commit revision 2355869
> but we haven't yet isolated the new behavior to that commit.  That'll be our
> next test but I wanted to get this out there for comment.

I'm not sure I understand the problem, generally when doing device
assignment you're using vfio-pci, which I'll assume is the case since
we're talking about an upstream kernel and Legacy KVM device assignment
no longer exists upstream.  That means the VF being assigned needs to
be bound to the vfio-pci driver, which libvirt might do for you if the
hostdev XML is specified with managed='yes'.  At what point is cxgb4vf
being re-loaded?  Is it correlated with the VM starting or VF being
hot-added to a VM or the reverse, VM shutdown or VF removed?  Is the
cxgb4vf driver replacing vfio-pci for the device being assigned?
libvirt might do a drivers_probe on remove, but that behavior shouldn't
have changed.  Being a VF, it needs to support FLR, so that would be
the means vfio would use to reset the device.  I wouldn't expect an FLR
to trigger any sort of device remove/re-add that might cause udev to
reload the driver.  Presumably adding a modprobe.d blacklist entry for
cxgb4vf would resolve it, if so, maybe look at udev events.  If there
are multiple PFs in the host, any creation of new VFs matching the
cxgb4vf driver could trigger udev to re-load the driver.  Thanks,

Alex