Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap <george.dunlap@eu.citrix.com>]

On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
>> On 05/07/13 14:48, Andrew Cooper wrote:
>>> On 05/07/13 14:45, George Dunlap wrote:
>>>> On 05/07/13 14:39, Andrew Cooper wrote:
>>>>> On 05/07/13 12:01, George Dunlap wrote:
>>>>>> I've been doing some work to try to make driver domains easier to set
>>>>>> up and use.  At the moment, in order to pass a device through to a
>>>>>> guest, you first need to assign it to pciback.  This involves doing
>>>>>> one of three things:
>>>>>> * Running xl pci-assignable-add for the device
>>>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>
>>>>>> None of these are very satisfying.  What I think would be better is if
>>>>>> there was a way to specify in the guest config file, "If device X is
>>>>>> not assignable, try to make it assignable".  That way you can have a
>>>>>> driver domain grab the appropriate device just by running "xl create
>>>>>> domnet"; and once we have the xendomains script up and running with
>>>>>> xl, you can simply configure your domnet appropriately, and then put
>>>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>>>
>>>>>> My initial idea was to add a parameter to the pci argument in the
>>>>>> config file; for example:
>>>>>>
>>>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>>>
>>>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>>>> assignable, that xl should try to make is assignable.
>>>>>>
>>>>>> The problem here is that this would need to be parsed by
>>>>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>>>>> libxl_device_pci.
>>>>>>
>>>>>> Now it seems to me that the right place to do this "seizing" is in xl,
>>>>>> not inside libxl -- the functions for doing assignment exist already,
>>>>>> and are simple and straightforward.  But doing it in xl, but as a
>>>>>> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
>>>>>> pass something else back, which begins to get awkward.
>>>>>>
>>>>>> So it seems to me we have a couple of options:
>>>>>> 1. Create a new argument, "pci_seize" or something like that, which
>>>>>> would be processed separately from pci
>>>>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>>>>>> arguments directed at xl rather than libxl
>>>>>> 3. Add "seize" to libxl_device_pci, but have it only used by xl
>>>>>> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>>>>>>
>>>>>> Any preference -- or any other ideas?
>>>>>>
>>>>>>    -George
>>>>> How about a setting in xl.conf of "auto-seize pci devices" ?  That way
>>>>> the seizing is entirely part of xl
>>>> Auto-seizing is fairly dangerous; you could easily accidentally yank
>>>> out the ethernet card, or even the disk that dom0 is using.  I really
>>>> think it should have to be enabled on a device-by-device basis.
>>>>
>>>> I suppose another option would be to be able to set, in xl.conf, a
>>>> list of auto-seizeable devices.  I don't really like that option as
>>>> well, though.  I'd rather be able to keep all the configuration in one
>>>> place.
>>>>
>>>>   -George
>>> Or a slight less extreme version.
>>>
>>> If xl sees that it would need seize a device, it could ask "You are
>>> trying to create a domain with device $FOO.  Would you like to seize it
>> >from dom0 ?"
>>
>> That won't work for driver domains, as we want it all to happen
>> automatically when the host is booting. :-)
>
> The high-level goal is that we want to put the network devices with a
> network backend and storage devices with storage backend. Ignorning
> that for network devices you might want seperate backends for each
> device (say one backend for Wireless, one for Ethernet, etc).
>
> Perhaps the logic ought to do grouping - so you say:
>   a) "backends:all-network" (which would created one backend with all of the
>     wireless, ethernet, etc PCI devices), or
>   b) "backends:all-network,seperate-storage", which  create one backend with
>    all of the wireless, ethernet in one backend; and one backend domain for each
>    storage device?
>
> Naturally the user gets to chose which grouping they would like?

We seem to be talking about different things.  You seem to be talking 
about automatically starting some pre-made VMs and assigning devices and 
backends to them?  But I'm not really sure.

I was assuming that the user was going to be installing and configuring 
their own driver domains.  The user already has to specify 
"pci=['$BDF']" in their config file to get specific devices passed 
through -- this would just be making it easy to have the device assigned 
to pciback as well.

I suspect that a lot of people will want to have one network card 
assigned to domain 0 as a "management network", and only have other 
devices assigned to driver domains.  I think that having one device per 
domain is probably the best recommendation; although we obviously want 
to support someone who wants a single "manage all the devices" domain, 
we should assume that people are going to have one device per driver domain.

  -George