Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Marcel Apfelbaum <marcel.apfelbaum@gmail.com>]

On 07/26/2018 07:14 PM, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
>> Hi
>>
>> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
>>> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>>>> Hey,
>>>>
>>>> Our team is adding a NIC functional  emulation to QEMU.
>>>> One of the features we are adding to this NIC is SRIOV.
>>>>
>>>> Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
>>>> "
>>>> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
>> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
>> not sure ACS is a prerequisite
>> for SR-IOV.
> Looks like windows blocks dev assignment in nested VMs without it.
> Thinking about it, doesn't vfio do the same by default?
vfio requires vIOMMU even for nested, as far as I know. They do have a 
"no-iommu"
option, but I don't know how  it can help in this case.

Elijah, what about implementing ACS for QEMU's generic PCIe Root Ports?
Maybe is not too complicated and it seems like a handy feature to have 
anyway.
Patches would be welcomed :)

Thanks,
Marcel

>   I think vfio has
> a flag to override this though.
>
>> We would need ARI support for allowing more than 8VFs, but QEMU
>> doesn't support that either (yet).
>>
>> Knut Omag has some working patches, he successfully implemented SR-IOV with
>> QEMU, see:
>>       https://github.com/knuto/qemu/tree/sriov_patches_v7
>>
>> The code was not merged since we need at least a device with SR-IOV support
>> to justify the addition.
>>
>>>> Contact your system vendor for further information.
>>>> "
>>> I'm not sure what the status of emulated SR-IOV is so I have CCed
>>> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>> Thanks,
>> Marcel
>>
>>>> Could you please advise about what could be the issue here?
>>>>
>>>> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>>>>
>>>> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
>>>> "
>>>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>>>>     <name>nst105</name>
>>>>     <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>>>     <memory unit='KiB'>8388608</memory>
>>>>     <currentMemory unit='KiB'>8388608</currentMemory>
>>>>     <vcpu placement='static'>8</vcpu>
>>>>     <resource>
>>>>       <partition>/machine</partition>
>>>>     </resource>
>>>>     <os>
>>>>       <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>>>     </os>
>>>>     <features>
>>>>       <acpi/>
>>>>       <apic/>
>>>>       <hyperv>
>>>>         <relaxed state='on'/>
>>>>         <vapic state='on'/>
>>>>         <spinlocks state='on' retries='8191'/>
>>>>       </hyperv>
>>>>     </features>
>>>>     <cpu mode='custom' match='exact' check='none'>
>>>>       <model fallback='forbid'>SandyBridge</model>
>>>>       <feature policy='require' name='hypervisor'/>
>>>>       <feature policy='require' name='vmx'/>
>>>>     </cpu>
>>>>     <clock offset='localtime'>
>>>>       <timer name='rtc' tickpolicy='catchup'/>
>>>>       <timer name='pit' tickpolicy='delay'/>
>>>>       <timer name='hpet' present='no'/>
>>>>       <timer name='hypervclock' present='yes'/>
>>>>     </clock>
>>>>     <on_poweroff>destroy</on_poweroff>
>>>>     <on_reboot>restart</on_reboot>
>>>>     <on_crash>destroy</on_crash>
>>>>     <pm>
>>>>       <suspend-to-mem enabled='no'/>
>>>>       <suspend-to-disk enabled='no'/>
>>>>     </pm>
>>>>     <devices>
>>>>       <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>>>       <disk type='file' device='disk'>
>>>>         <driver name='qemu' type='qcow2'/>
>>>>         <source file='/images/nst105.qcow2'/>
>>>>         <target dev='sda' bus='sata'/>
>>>>         <boot order='2'/>
>>>>         <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>       </disk>
>>>>       <controller type='usb' index='0' model='ich9-ehci1'>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>>>>       </controller>
>>>>       <controller type='usb' index='0' model='ich9-uhci1'>
>>>>         <master startport='0'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>>>>       </controller>
>>>>       <controller type='usb' index='0' model='ich9-uhci2'>
>>>>         <master startport='2'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>>>>       </controller>
>>>>       <controller type='usb' index='0' model='ich9-uhci3'>
>>>>         <master startport='4'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>>>>       </controller>
>>>>       <controller type='sata' index='0'>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>>>>       </controller>
>>>>       <controller type='pci' index='0' model='pcie-root'/>
>>>>       <controller type='pci' index='1' model='pcie-root-port'>
>>>>         <model name='pcie-root-port'/>
>>>>         <target chassis='1' port='0x10'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>>>>       </controller>
>>>>       <controller type='pci' index='2' model='pcie-root-port'>
>>>>         <model name='pcie-root-port'/>
>>>>         <target chassis='2' port='0x11'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>>>>       </controller>
>>>>       <controller type='pci' index='3' model='pcie-root-port'>
>>>>         <model name='pcie-root-port'/>
>>>>         <target chassis='3' port='0x12'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>>>>       </controller>
>>>>       <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>>>         <model name='i82801b11-bridge'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>>>>       </controller>
>>>>       <controller type='pci' index='5' model='pci-bridge'>
>>>>         <model name='pci-bridge'/>
>>>>         <target chassisNr='5'/>
>>>>         <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>>>>       </controller>
>>>>       <interface type='bridge'>
>>>>         <mac address='00:50:56:1b:dd:6f'/>
>>>>         <source bridge='br0'/>
>>>>         <model type='virtio'/>
>>>>         <boot order='1'/>
>>>>         <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>>>       </interface>
>>>>       <serial type='pty'>
>>>>         <target port='0'/>
>>>>       </serial>
>>>>       <console type='pty'>
>>>>         <target type='serial' port='0'/>
>>>>       </console>
>>>>       <input type='tablet' bus='usb'>
>>>>         <address type='usb' bus='0' port='1'/>
>>>>       </input>
>>>>       <input type='mouse' bus='ps2'/>
>>>>       <input type='keyboard' bus='ps2'/>
>>>>       <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>>>>         <listen type='address' address='127.0.0.1'/>
>>>>       </graphics>
>>>>       <video>
>>>>         <model type='vga' vram='16384' heads='1' primary='yes'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>>>>       </video>
>>>>       <memballoon model='virtio'>
>>>>         <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>>>>       </memballoon>
>>>>     </devices>
>>>>     <seclabel type='none' model='none'/>
>>>>     <seclabel type='dynamic' model='dac' relabel='yes'/>
>>>>     <qemu:commandline>
>>>>       <qemu:arg value='-device'/>
>>>>       <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>>>>       <qemu:arg value='-netdev'/>
>>>>       <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>>>       <qemu:arg value='-device'/>
>>>>       <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>>>     </qemu:commandline>
>>>> </domain>
>>>> "
>>>> ______________________
>>>> General info:
>>>> Host OS: RH7.0 (Kernel: 4.14.13)
>>>> QEMU version: 2.11
>>>> libvirt version: 3.2.0
>>>> Running the following on the host shows that both nested and IOMMU are
>>>> enabled:
>>>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>>>> Y
>>>> ~]# dmesg | grep -e DMAR -e IOMMU
>>>> [    0.000000] DMAR: IOMMU enabled
>>>>
>>>> Thanks,
>>>> Elijah
>>>>