Re: [PATCH v10 13/17] vpci: add initial support for virtual PCI bus topology

[Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>]

Hi Stefano,

Stefano Stabellini <sstabellini@kernel.org> writes:

> On Thu, 16 Nov 2023, Julien Grall wrote:
>> IIUC, this means that Xen will allocate the BDF. I think this will become a
>> problem quite quickly as some of the PCI may need to be assigned at a specific
>> vBDF (I have the intel graphic card in mind).
>> 
>> Also, xl allows you to specificy the slot (e.g. <bdf>@<vslot>) which would not
>> work with this approach.
>> 
>> For dom0less passthrough, I feel the virtual BDF should always be specified in
>> device-tree. When a domain is created after boot, then I think you want to
>> support <bdf>@<vslot> where <vslot> is optional.
>
> Hi Julien,
>
> I also think there should be a way to specify the virtual BDF, but if
> possible (meaning: it is not super difficult to implement) I think it
> would be very convenient if we could let Xen pick whatever virtual BDF
> Xen wants when the user doesn't specify the virtual BDF. That's
> because it would make it easier to specify the configuration for the
> user. Typically the user doesn't care about the virtual BDF, only to
> expose a specific host device to the VM. There are exceptions of course
> and that's why I think we should also have a way for the user to
> request a specific virtual BDF. One of these exceptions are integrated
> GPUs: the OS drivers used to have hardcoded BDFs. So it wouldn't work if
> the device shows up at a different virtual BDF compared to the host.
>
> Thinking more about this, one way to simplify the problem would be if we
> always reuse the physical BDF as virtual BDF for passthrough devices. I
> think that would solve the problem and makes it much more unlikely to
> run into drivers bugs.

I'm not sure that this is possible. AFAIK, if we have device with B>0,
we need to have bridge device for it. So, if I want to passthrough
device 08:00.0, I need to provide a virtual bridge with BDF 0:NN.0. This
unnecessary complicates things.

Also, there can be funny situation with conflicting BFD numbers exposed
by different domains. I know that this is not your typical setup, but
imagine that Domain A acts as a driver domain for PCI controller A and
Domain B acts as a driver domain for PCI controller B. They may expose
devices with same BDFs but with different segments.

> And we allocate a "special" virtual BDF space for emulated devices, with
> the Root Complex still emulated in Xen. For instance, we could reserve
> ff:xx:xx and in case of clashes we could refuse to continue. Or we could
> allocate the first free virtual BDF, after all the pasthrough devices.

Again, I may be wrong there, but we need an emulated PCI bridge device if we
want to use Bus numbers > 0.

>
> Example:
> - the user wants to assign physical 00:11.5 and b3:00.1 to the guest
> - Xen create virtual BDFs 00:11.5 and b3:00.1 for the passthrough devices
> - Xen allocates the next virtual BDF for emulated devices: b4:xx.x
> - If more virtual BDFs are needed for emulated devices, Xen allocates
>   b5:xx.x
>
> I still think, no matter the BDF allocation scheme, that we should try
> to avoid as much as possible to have two different PCI Root Complex
> emulators. Ideally we would have only one PCI Root Complex emulated by
> Xen. Having 2 PCI Root Complexes both of them emulated by Xen would be
> tolerable but not ideal.

But what is exactly wrong with this setup?

> The worst case I would like to avoid is to have
> two PCI Root Complexes, one emulated by Xen and one emulated by QEMU.

This is how our setup works right now.

I agree that we need some way to provide static vBDF numbers. But I am
wondering what is the best way to do this. We need some entity that
manages and assigns those vBDFs. It should reside in Xen, because there
is Dom0less use case. Probably we need to extend
xen_domctl_assign_device so we can either request a free vBDF or a
specific vBDF. And in the first case, Xen should return assigned vBDF so
toolstack can give it to a backend, if PCI device is purely virtual.

-- 
WBR, Volodymyr