Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Marcel Apfelbaum <marcel.apfelbaum@gmail.com>]

Hi

On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>> Hey,
>>
>> Our team is adding a NIC functional  emulation to QEMU.
>> One of the features we are adding to this NIC is SRIOV.
>>
>> Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
>> "
>> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.

QEMU's emulated PCI Express Root Ports do not support ACS yet, however I 
am not sure ACS is a prerequisite
for SR-IOV. We would need ARI support for allowing more than 8VFs, but 
QEMU doesn't support that either (yet).

Knut Omag has some working patches, he successfully implemented SR-IOV 
with QEMU, see:
      https://github.com/knuto/qemu/tree/sriov_patches_v7

The code was not merged since we need at least a device with SR-IOV 
support to justify the addition.

>> Contact your system vendor for further information.
>> "
> I'm not sure what the status of emulated SR-IOV is so I have CCed
> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.

Thanks,
Marcel

>> Could you please advise about what could be the issue here?
>>
>> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>>
>> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
>> "
>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>>    <name>nst105</name>
>>    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>    <memory unit='KiB'>8388608</memory>
>>    <currentMemory unit='KiB'>8388608</currentMemory>
>>    <vcpu placement='static'>8</vcpu>
>>    <resource>
>>      <partition>/machine</partition>
>>    </resource>
>>    <os>
>>      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>    </os>
>>    <features>
>>      <acpi/>
>>      <apic/>
>>      <hyperv>
>>        <relaxed state='on'/>
>>        <vapic state='on'/>
>>        <spinlocks state='on' retries='8191'/>
>>      </hyperv>
>>    </features>
>>    <cpu mode='custom' match='exact' check='none'>
>>      <model fallback='forbid'>SandyBridge</model>
>>      <feature policy='require' name='hypervisor'/>
>>      <feature policy='require' name='vmx'/>
>>    </cpu>
>>    <clock offset='localtime'>
>>      <timer name='rtc' tickpolicy='catchup'/>
>>      <timer name='pit' tickpolicy='delay'/>
>>      <timer name='hpet' present='no'/>
>>      <timer name='hypervclock' present='yes'/>
>>    </clock>
>>    <on_poweroff>destroy</on_poweroff>
>>    <on_reboot>restart</on_reboot>
>>    <on_crash>destroy</on_crash>
>>    <pm>
>>      <suspend-to-mem enabled='no'/>
>>      <suspend-to-disk enabled='no'/>
>>    </pm>
>>    <devices>
>>      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>      <disk type='file' device='disk'>
>>        <driver name='qemu' type='qcow2'/>
>>        <source file='/images/nst105.qcow2'/>
>>        <target dev='sda' bus='sata'/>
>>        <boot order='2'/>
>>        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>      </disk>
>>      <controller type='usb' index='0' model='ich9-ehci1'>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>>      </controller>
>>      <controller type='usb' index='0' model='ich9-uhci1'>
>>        <master startport='0'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>>      </controller>
>>      <controller type='usb' index='0' model='ich9-uhci2'>
>>        <master startport='2'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>>      </controller>
>>      <controller type='usb' index='0' model='ich9-uhci3'>
>>        <master startport='4'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>>      </controller>
>>      <controller type='sata' index='0'>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>>      </controller>
>>      <controller type='pci' index='0' model='pcie-root'/>
>>      <controller type='pci' index='1' model='pcie-root-port'>
>>        <model name='pcie-root-port'/>
>>        <target chassis='1' port='0x10'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>>      </controller>
>>      <controller type='pci' index='2' model='pcie-root-port'>
>>        <model name='pcie-root-port'/>
>>        <target chassis='2' port='0x11'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>>      </controller>
>>      <controller type='pci' index='3' model='pcie-root-port'>
>>        <model name='pcie-root-port'/>
>>        <target chassis='3' port='0x12'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>>      </controller>
>>      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>        <model name='i82801b11-bridge'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>>      </controller>
>>      <controller type='pci' index='5' model='pci-bridge'>
>>        <model name='pci-bridge'/>
>>        <target chassisNr='5'/>
>>        <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>>      </controller>
>>      <interface type='bridge'>
>>        <mac address='00:50:56:1b:dd:6f'/>
>>        <source bridge='br0'/>
>>        <model type='virtio'/>
>>        <boot order='1'/>
>>        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>      </interface>
>>      <serial type='pty'>
>>        <target port='0'/>
>>      </serial>
>>      <console type='pty'>
>>        <target type='serial' port='0'/>
>>      </console>
>>      <input type='tablet' bus='usb'>
>>        <address type='usb' bus='0' port='1'/>
>>      </input>
>>      <input type='mouse' bus='ps2'/>
>>      <input type='keyboard' bus='ps2'/>
>>      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>>        <listen type='address' address='127.0.0.1'/>
>>      </graphics>
>>      <video>
>>        <model type='vga' vram='16384' heads='1' primary='yes'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>>      </video>
>>      <memballoon model='virtio'>
>>        <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>>      </memballoon>
>>    </devices>
>>    <seclabel type='none' model='none'/>
>>    <seclabel type='dynamic' model='dac' relabel='yes'/>
>>    <qemu:commandline>
>>      <qemu:arg value='-device'/>
>>      <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>>      <qemu:arg value='-netdev'/>
>>      <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>      <qemu:arg value='-device'/>
>>      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>    </qemu:commandline>
>> </domain>
>> "
>> ______________________
>> General info:
>> Host OS: RH7.0 (Kernel: 4.14.13)
>> QEMU version: 2.11
>> libvirt version: 3.2.0
>> Running the following on the host shows that both nested and IOMMU are
>> enabled:
>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>> Y
>> ~]# dmesg | grep -e DMAR -e IOMMU
>> [    0.000000] DMAR: IOMMU enabled
>>
>> Thanks,
>> Elijah
>>