Re: [ANNOUNCE] New driver vxge for Neterion's X3100 series10GbEPCIe adapter

[Yu Zhao <yu.zhao@intel.com>]

On Wed, Apr 01, 2009 at 01:44:45PM +0800, Leonid Grossman wrote:
> > On Wed, Apr 01, 2009 at 11:36:11AM +0800, Ramkrishna Vepa wrote:
> > > > Yes, and that's what the PCI subsystem does. If the vxge VF is
> > > identical
> > > > to its PF, then vxge should be able to drive both PF and VF
> without
> > > any
> > > > modification.
> > > [Ram] Ok. In that case, is the call to pci_enable/disable_sriov
> still
> > > required for vxge?
> > 
> > Yes, the vxge driver first binds the PF once it's loaded (VF doesn't
> > exist at this time) and calls the SR-IOV API. The VF appears after the
> > SR-IOV is enabled and then the same copy of the vxge driver can bind
> > the VF too if you want to use the VF in the native Linux. Though the
> > hardware is in the SR-IOV mode in this case, it would be equal to the
> > multi-function mode. Or you can assign the VF to the Xen/KVM guest and
> > let another copy of vxge driver (may be vxge for Windows, Solaris,
> BSD,
> > etc.) running in the guest bind it.
> 
> Yu, could you pl. explain why this call is not optional - SR-IOV pci-e
> code should be able to find SR-IOV capable device and enable all VFs
> based upon pci-e config space alone, without any help from
> device-specific PF driver.

Yes, this is true in certain cases. However, there are several things
that prevent us to enable the SR-IOV implicitly by the PCI subsystem.

First, the SR-IOV spec says "Once the SRIOV capability is configured
enabling VF to be assigned to individual SI, the PF takes on a more
supervisory role. For example, the PF can be used to manage device
specific functionality such as internal resource allocation to each
VF, VF arbitration to shared resources such as the PCIe Link or the
Function-specific Link (e.g., a network or storage Link), etc." And
some SR-IOV devices follow this suggestion thus their VF cannot work
without PF driver pre-exits. Intel 82576 is an example -- it requires
the PF driver to allocate tx/rx queues for the VF so the VF can be
functional. Only enabling the SR-IOV in the PF PCI config space will
end up with VF appearing useless even its PCI config space is visible.

Second, the SR-IOV depends on somethings that are not available before
the PCI subsystem is fully initialized. This mean we cannot enable the
SR-IOV capability before all PCI device are enumerated. For example,
if the VF resides on the bus different than the PF bus, then we can't
enable the VF before the bus used by the VF is scan because we don't
know if the bus is reserved by the BIOS for the VF or not. Another
example is the dependency link used by the PF -- we can't create the
sysfs symbolic link indicating the dependency before all PFs in a
device are enumerated.

And some SR-IOV devices can support multiple modes at same time.
The 82576 can support N VFs + M VMDq modes (N + M = 8), which means
sometimes people may want to only enable arbitrary number of VFs.
The PCI subsystem can't get value to config the NumVFs unless some
one calls the API. I guess only the PF driver can do this.

> Once VFs appear, vxge or any other native netdev driver should be able
> to bind a VF regardless of PF driver being loaded first (or at all) -
> there are some use cases that do not assume PF driver presence...

So the PF will not be binded to any driver in these use cases? Can you
please elaborate?

Thanks,
Yu