Re: PV passthrough of sibling igbvf's

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 16/10/12 15:21, Laszlo Ersek wrote:
> Hi,
>
> this is with reference to
> <https://bugzilla.redhat.com/show_bug.cgi?id=865736> -- RHEL-5.9 Beta
> host & guest. Nonetheless I think my question applies to current
> upstream Linux -- if not, I'd greatly appreciate commit hashes.
>
> Consider several igbvf's that belong to the same PF (port): they are
> functions that share a (bus, device) pair (aka "slot") in dom0. The VPCI
> implementation of pciback_add_pci_dev() [drivers/xen/pciback/vpci.c]
> will assign these sibling functions to the same virtual slot. In other
> words, VFs that are siblings in dom0 end up as siblings in the PV domU.
>
> (Upstream path and function: "drivers/xen/xen-pciback/vpci.c",
> __xen_pcibk_add_pci_dev().)
>
> This logic appears to date back to
> <http://xenbits.xensource.com/xen-unstable.hg/rev/5b433b4fca34>.
>
>
> The RHEL-5.9 Beta PV domU does something like this:
>
>   pci_scan_slot
>     /* for each function: */
>       pci_scan_single_device
>          pci_scan_device
>            pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, &l)
>            pci_setup_device
>              pci_read_config_byte(dev, PCI_HEADER_TYPE, &hdr_type)
>              dev->multifunction = !!(hdr_type & 0x80);
>       if (dev_scanned && !dev->multifunc && func == 0): break
>
> Current upstream Linux has gone through several changes here, but the
> gist is the same: if function 0 is successfully scanned and it
> explicitly reports itself non-multi (--> no_next_fn()), then the rest of
> the functions on the slot is skipped.
>
> Problem is, function 0 of the igbvf I'm looking at reports itself as
> non-multifunction, and thus the domU doesn't find the rest of the
> passed-through functions.

Which is correct, because the virtual function itself is only a single
function.

I would hazard a guess that the real bug is trying to fake up 8
individual virtual functions as an 8-fuction device, which seems like a
toolstack bug to me.

While it would certainly be possible to trap and emulate reads like
this, I would think it would be decidedly hacky, thus preferably avoided.

~Andrew

>
>
> pciback seems to have no overlay for PCI_HEADER_TYPE in array
> "header_common" [upstream: drivers/xen/xen-pciback/conf_space_header.c],
> thus pciback_config_read() / xen_pcibk_config_read() pass through the
> header type transparently when the domU reads it in pci_setup_device().
>
>
> I wonder if
>
> - a new dom0 overlay should be introduced for PCI_HEADER_TYPE, to the
> tune of upstream Linux commit fd5b221b (ie. vendor-id/device-id), that
> would perhaps check the # of sibling functions in the given vpci slot,
> and fake the MSB in "hdr_type",
>
> - or the domU's slot scanning logic should be changed,
>
> - or the igbvf I'm looking at reports an incorrect "hdr_type" (in which
> case we'd still have to fake something). The legacy
> "/etc/xen/xend-pci-quirks.sxp" interface is only suitable for giving
> extra write access to the domU, thus not good enough here.
>
> Thanks a lot!
> Laszlo
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com