Re: Correct way to enable BusMaster with VFIO?

[Alex Williamson <alex.williamson@redhat.com>]

Hi Wei,

On Tue, 2015-04-07 at 21:02 -0700, Wei Hu wrote:
> Hi Alex,
> 
> With your change "Release devices with BusMaster disabled", I've found
> that my VFIO device driver is no longer receiving MSI interrupts.
> After reviewing the code I think it makes sense. But I had two
> questions below while debugging my issue.
> 
> 1.  If I had set the bus master bit in the command register by hand
> before opening the vfio device, the kernel would actually leave
> BusMaster enabled. This seems to contradict the call to
> pci_clear_master() from vfio_pci_enable(). What's going on here, is
> something else enabling BusMaster?

I don't see this behavior.  If I use a test program that simply opens
the vfio group, configures the IOMMU container and gets a device file
descriptor, BusMaster is clear at that point regardless of whether the
device has it enabled or disabled beforehand.  Are you perhaps attaching
bridges to vfio-pci and the bridge is getting enabled as part of the
pci_enable_device() call of the endpoint?  Newer versions of vfio-pci
won't allow binding bridges, it was a bug that it was ever allowed.  Our
config space manipulation only knows about normal header types.

> 2. What's the recommended way to enable BusMaster with your change
> now? Should my driver map the config space region and set the
> BusMaster bit? Or should I have a separate command to enable the bit
> before opening the device?

VFIO is designed to be a complete interface to the device, why would you
write a driver that makes use of VFIO that has pre-conditions of
manipulating the device state outside of VFIO?  There's no additional
mapping of config space required, you already have access to it through
the device file descriptor.  It should simply be a matter of doing a
read-modify-write at the proper offset to toggle the command register.
QEMU uses pread(2) and pwrite(2) for this, but any equivalent tool
should work as well.  Thanks,

Alex