Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Laine Stump <laine@redhat.com>]

On 08/18/2016 03:41 AM, Andrew Jones wrote:
> On Wed, Aug 17, 2016 at 01:00:05PM -0400, Laine Stump wrote:
>> On 08/17/2016 12:13 PM, Andrew Jones wrote:
>>> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>>>> Hi all,
>>>>       Now I'm investigating net device hot plug and disk hotplug for
>>>> AArch64. For virtio , the default address is virtio-mmio. After Libvirt
>>>> 1.3.5, user can explicitly specify the address-type to pci and so libvirt
>>>> will pass the virtio-pci parameters to the Qemu.
>>>>       Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
>>>> Libvirt version is 1.3.5.
>>>>       For net-device, I change the address-type to pci, and libvirt pass the
>>>> command below:
>>>>       -device
>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
>>>>
>>>>       After booting, the eth0 device disappear(eth0 occur when the address
>>>> is virtio-mmio),
>>>> but I can find another net-device enp2s1, also it can't work for dhcp:
>>>> Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
>>>> device
>>>> I'm not sure whether it worked.
>>>>
>>>>       For disk device,* when I change the address-type to pci, the whole
>>>> qemu command is :*
>>>> https://paste.fedoraproject.org/409553/,  but the VM can not boot
>>>> successfully. Does Qemu not support device disk of virtio-pci in AArch64
>>>> just as it in X86_64?
>>>>       Thanks~Since I am not very familiar with Qemu, really looking forward
>>>> to your response.
>>>>
>>>> Best Regards,
>>>> Kevin Zhao
>>> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
>>> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
>>> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
>>> go in order to configure a base/standard mach-virt PCIe machine.
>>
>> Well, you can do it now, but you have to manually assign the PCI addresses
>> of devices (and if you want hotplug you need to live with Intel/TI-specific
>> PCIe controllers).
>>
>>
>>>
>>> 1) If we want to support both PCIe devices and PCI, then things are messy.
>>>     Currently we propose dropping PCI support. mach-virt pretty much
>>>     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
>>
>> I have a libvirt patch just about ACKed for pushing upstream that will
>> automatically assign virtio-pci devices to a PCIe slot (if the qemu binary
>> supports virtio-1.0):
>>
>> https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html
>>
>> Separate patches do the same for the e1000e emulated network device (which
>> you probably don't care about) and the nec-usb-xhci (USB3) controller (more
>> useful):
>>
>> https://www.redhat.com/archives/libvir-list/2016-August/msg00732.html
>>
>
> Thanks for the update Laine. This sounds great to me. With those patches
> we can switch from virtio-mmio to virtio-pci easily, even if we're still
> missing hotplug a bit longer. What limit do we have for the number of
> devices, when we don't have any switches? I think I experimented once and
> found it to be 7.

Theoretically you should be able to put something in each slot of 
pcie-root, and there are 31 slots (but slot 0x1f is always used by 
integrated devices, slot 1 is usually used by video, and slot 0x1D is 
usually used by a USB controller). Anyway, you should be able to get a 
lot more than 7 devices (one potential problem is that if you add a PCI 
controller that has a device plugged into it which requests IO port 
space, that will get chewed up very quickly. That's not an issue if you 
are connecting endpoints directly to pcie-root though.


>
>> Once these are in place, the only type of device of any consequence that I
>> can see still having no PCIe alternative is audio (even though only the
>> virgl video device is PCIe, libvirt has always assigned the primary video to
>> slot 1 on pcie-root anyway (although you shouldn't put a legacy PCI device
>> on a pcie-root-port or pcie-switch-downstream-port, it is acceptable to plug
>> it directly into pcie-root (as long as you know you won't need to hotplug
>> it).
>>
>>> 2) root complex ports, switches (upstream/downstream ports) are currently
>>>     based on Intel parts. Marcel is thinking about creating generic models.
>>
>> I say this every time it comes up, so just to be consistent: +1 :-)
>>
>>> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>>>     fashion, leaving holes for hotplug.
>>
>> See above about virtio, although that doesn't cover the whole story. The
>> other part (which I'm working on right now) is that libvirt needs to
>> automatically add pcie-root-port, pcie-switch-upstream-port, and
>> pcie-switch-downstream-port devices as necessary. With the patches I
>> mentioned above, you still have to manually add enough pcie-*-port
>> controllers to the config, and then libvirt will plug the PCIe devices into
>> the right place. This is simple enough to do, but it does require
>> intervention.
>
> OK, so we want this to support hotplug and eventually chain switches,
> bumping our device limit up higher and higher. To what? I'm not sure,
> I guess we're still limited by address space.

As long as the endpoint devices don't require IO port space, the 
BIOS/firmware shouldn't try to reserve it, and the limit will be "more 
than you would ever need".

>
>>
>> As far as leaving holes for hotplug, there's actually still a bit of an open
>> question there - with machinetypes that use only legacy-PCI, *all* slots are
>> hotpluggable, and they're added 31 at a time, so there was never any
>> question about which slots were hotpluggable, and it would be very rare to
>> end up with a configuration that had 0 free slots available for hotplug
>> (actually libvirt would always make sure there was at least one, but in
>> practice there would be many more open slots). With PCIe-capable
>> machinetypes that is changed, since the root complex (pcie-root) doesn't
>> support hotplug, and new slots are added 1 at a time (pcie-*-port) rather
>> than 31 at a time. This means you have to really go out of your way if you
>> want open slots for hotplug (and even if you want devices in the machine at
>> boot time to be hot-unpluggable).
>>
>> I'm still not sure just how far we need to go in this regard.  We've already
>> decided that, unless manually set to an address on pcie-root by the
>> user/management application, all PCI devices will be auto-assigned to a slot
>> that supports hotplug. What I'm not sure about is whether we should always
>> auto-add an extra pcie-*-root to be sure a device can be hotplugged, or if
>> we should admit that 1 available slot isn't good enough for all situations,
>> so we should instead just leave it up to the user/management to manually add
>> extra ports if they think they'll want to hotplug something later.
>
> Hmm... Maybe the tools can make this easier by offering an option to
> provide N extra ports.

I've thought about this, and there have even been emails about it, but I 
don't know if it could ever be accepted into libvirt. My idea was to 
have an attribute that says "always maintain X hotpluggable slots when 
coldplugging new devices". I have a feeling that will get as warm of a 
reception as my proposed "hotpluggable='no'" attribute for devices (i.e. 
not warm at all :-))

The problem is that if you just tell the user to put in extra 
pcie-*-port controllers to allow for hotplug, those will eventually be 
eaten up, and then the next time you start the guest you'll again have 
no open hotpluggable slots again (even though restarting would have 
given you the opportunity to automatically add new *-port controllers).


>
> Hmm2... I think I agree that we don't need to worry too much about
> providing free ports for hotplug (maybe just one for fun). With
> virtio-scsi we can plug disks already.


until you run out of open units/buses/whatever they're called in 
virtio-scsi (I'm too lazy to look it up right now). And then you have 
the same problem.

We really can never solve the problem, we can only *delay* it, and the 
more we try to delay it the more convoluted the code gets. And of course 
the same number that is "not enough" for some users is "too much" for 
others. That's why I've been wondering if maybe we should just throw up 
our hands and punt (i.e. unabashedly document "if you want slots 
available for hotplug, you'll need to add them. Of course *even that* 
doesn't work unless you add the hotpluggable slots in a separate change 
from any endpoint devices; otherwise the new endpoint devices would just 
be immediately assigned to the new slots you added "for future hotplug" 
:-/).


> If we want to provide multiple
> virtio-net devices for the price of one port, we can enable mutlifunction.


Except that you can't hotplug/unplug the functions independent of each 
other - all the functions in one slot must be plugged/unplugged together.

> And, iirc, there's work to get ARI functioning, allowing mutlifunction to
> go nuts (assuming I understand its purpose correctly)

I keep forgetting all the acronyms. Is ARI the one that allows you (on 
controllers that only have a single slot, e.g. pcie-root-port) to 
interpret the byte that's normally a 5 bit slot# + 3 bit function# as a 
single 8 bit function#? (The idea being that you still have only a 
single slot, but that slot has 256 functions). There again, you still 
have the "hotplug as a single unit" problem, only the effects are 
multiplied because there are so many devices.


>
> So maybe the default config just needs 3 ports?
>  1 virtio-scsi with as many disks as requested
>  1 virtio-net with as many functions as nics are requested
>  1 extra port

None of those are in any default config created by libvirt. In a libvirt 
config with no specified devices (and no specified "anti-devices" such 
as "<controller type='usb' model='none'/>"), there is just the 
integrated ACHI (sata) controller, a USB controller (usually), and a 
<balloon> device (not sure why that's considered so important that it's 
in there by default, but it is).