[Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Elijah Shakkour]

Hey,

Our team is adding a NIC functional  emulation to QEMU.
One of the features we are adding to this NIC is SRIOV.

Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
"
SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port. Contact your system vendor for further information.
"

Could you please advise about what could be the issue here?

BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.

Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
"
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>nst105</name>
  <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>SandyBridge</model>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='vmx'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/images/nst105.qcow2'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='5'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='00:50:56:1b:dd:6f'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
    <qemu:arg value='-netdev'/>
    <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
  </qemu:commandline>
</domain>
"
______________________
General info:
Host OS: RH7.0 (Kernel: 4.14.13)
QEMU version: 2.11
libvirt version: 3.2.0
Running the following on the host shows that both nested and IOMMU are
enabled:
~]#: cat /sys/module/kvm_intel/parameters/nested
Y
~]# dmesg | grep -e DMAR -e IOMMU
[    0.000000] DMAR: IOMMU enabled

Thanks,
Elijah

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6871 bytes --]

On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> Hey,
> 
> Our team is adding a NIC functional  emulation to QEMU.
> One of the features we are adding to this NIC is SRIOV.
> 
> Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
> "
> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port. Contact your system vendor for further information.
> "

I'm not sure what the status of emulated SR-IOV is so I have CCed
Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.

> 
> Could you please advise about what could be the issue here?
> 
> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
> 
> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
> "
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>   <name>nst105</name>
>   <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>   <memory unit='KiB'>8388608</memory>
>   <currentMemory unit='KiB'>8388608</currentMemory>
>   <vcpu placement='static'>8</vcpu>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <hyperv>
>       <relaxed state='on'/>
>       <vapic state='on'/>
>       <spinlocks state='on' retries='8191'/>
>     </hyperv>
>   </features>
>   <cpu mode='custom' match='exact' check='none'>
>     <model fallback='forbid'>SandyBridge</model>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='require' name='vmx'/>
>   </cpu>
>   <clock offset='localtime'>
>     <timer name='rtc' tickpolicy='catchup'/>
>     <timer name='pit' tickpolicy='delay'/>
>     <timer name='hpet' present='no'/>
>     <timer name='hypervclock' present='yes'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <pm>
>     <suspend-to-mem enabled='no'/>
>     <suspend-to-disk enabled='no'/>
>   </pm>
>   <devices>
>     <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/images/nst105.qcow2'/>
>       <target dev='sda' bus='sata'/>
>       <boot order='2'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <controller type='usb' index='0' model='ich9-ehci1'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci1'>
>       <master startport='0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci2'>
>       <master startport='2'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci3'>
>       <master startport='4'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>     </controller>
>     <controller type='sata' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pcie-root'/>
>     <controller type='pci' index='1' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='1' port='0x10'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>     </controller>
>     <controller type='pci' index='2' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='2' port='0x11'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>     </controller>
>     <controller type='pci' index='3' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='3' port='0x12'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>     </controller>
>     <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>       <model name='i82801b11-bridge'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>     </controller>
>     <controller type='pci' index='5' model='pci-bridge'>
>       <model name='pci-bridge'/>
>       <target chassisNr='5'/>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='00:50:56:1b:dd:6f'/>
>       <source bridge='br0'/>
>       <model type='virtio'/>
>       <boot order='1'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <input type='tablet' bus='usb'>
>       <address type='usb' bus='0' port='1'/>
>     </input>
>     <input type='mouse' bus='ps2'/>
>     <input type='keyboard' bus='ps2'/>
>     <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>       <listen type='address' address='127.0.0.1'/>
>     </graphics>
>     <video>
>       <model type='vga' vram='16384' heads='1' primary='yes'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>     </memballoon>
>   </devices>
>   <seclabel type='none' model='none'/>
>   <seclabel type='dynamic' model='dac' relabel='yes'/>
>   <qemu:commandline>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>     <qemu:arg value='-netdev'/>
>     <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>     <qemu:arg value='-device'/>
>     <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>   </qemu:commandline>
> </domain>
> "
> ______________________
> General info:
> Host OS: RH7.0 (Kernel: 4.14.13)
> QEMU version: 2.11
> libvirt version: 3.2.0
> Running the following on the host shows that both nested and IOMMU are
> enabled:
> ~]#: cat /sys/module/kvm_intel/parameters/nested
> Y
> ~]# dmesg | grep -e DMAR -e IOMMU
> [    0.000000] DMAR: IOMMU enabled
> 
> Thanks,
> Elijah
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Marcel Apfelbaum]

Hi

On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>> Hey,
>>
>> Our team is adding a NIC functional  emulation to QEMU.
>> One of the features we are adding to this NIC is SRIOV.
>>
>> Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
>> "
>> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.

QEMU's emulated PCI Express Root Ports do not support ACS yet, however I 
am not sure ACS is a prerequisite
for SR-IOV. We would need ARI support for allowing more than 8VFs, but 
QEMU doesn't support that either (yet).

Knut Omag has some working patches, he successfully implemented SR-IOV 
with QEMU, see:
      https://github.com/knuto/qemu/tree/sriov_patches_v7

The code was not merged since we need at least a device with SR-IOV 
support to justify the addition.

>> Contact your system vendor for further information.
>> "
> I'm not sure what the status of emulated SR-IOV is so I have CCed
> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.

Thanks,
Marcel

>> Could you please advise about what could be the issue here?
>>
>> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>>
>> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
>> "
>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>>    <name>nst105</name>
>>    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>    <memory unit='KiB'>8388608</memory>
>>    <currentMemory unit='KiB'>8388608</currentMemory>
>>    <vcpu placement='static'>8</vcpu>
>>    <resource>
>>      <partition>/machine</partition>
>>    </resource>
>>    <os>
>>      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>    </os>
>>    <features>
>>      <acpi/>
>>      <apic/>
>>      <hyperv>
>>        <relaxed state='on'/>
>>        <vapic state='on'/>
>>        <spinlocks state='on' retries='8191'/>
>>      </hyperv>
>>    </features>
>>    <cpu mode='custom' match='exact' check='none'>
>>      <model fallback='forbid'>SandyBridge</model>
>>      <feature policy='require' name='hypervisor'/>
>>      <feature policy='require' name='vmx'/>
>>    </cpu>
>>    <clock offset='localtime'>
>>      <timer name='rtc' tickpolicy='catchup'/>
>>      <timer name='pit' tickpolicy='delay'/>
>>      <timer name='hpet' present='no'/>
>>      <timer name='hypervclock' present='yes'/>
>>    </clock>
>>    <on_poweroff>destroy</on_poweroff>
>>    <on_reboot>restart</on_reboot>
>>    <on_crash>destroy</on_crash>
>>    <pm>
>>      <suspend-to-mem enabled='no'/>
>>      <suspend-to-disk enabled='no'/>
>>    </pm>
>>    <devices>
>>      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>      <disk type='file' device='disk'>
>>        <driver name='qemu' type='qcow2'/>
>>        <source file='/images/nst105.qcow2'/>
>>        <target dev='sda' bus='sata'/>
>>        <boot order='2'/>
>>        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>      </disk>
>>      <controller type='usb' index='0' model='ich9-ehci1'>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>>      </controller>
>>      <controller type='usb' index='0' model='ich9-uhci1'>
>>        <master startport='0'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>>      </controller>
>>      <controller type='usb' index='0' model='ich9-uhci2'>
>>        <master startport='2'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>>      </controller>
>>      <controller type='usb' index='0' model='ich9-uhci3'>
>>        <master startport='4'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>>      </controller>
>>      <controller type='sata' index='0'>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>>      </controller>
>>      <controller type='pci' index='0' model='pcie-root'/>
>>      <controller type='pci' index='1' model='pcie-root-port'>
>>        <model name='pcie-root-port'/>
>>        <target chassis='1' port='0x10'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>>      </controller>
>>      <controller type='pci' index='2' model='pcie-root-port'>
>>        <model name='pcie-root-port'/>
>>        <target chassis='2' port='0x11'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>>      </controller>
>>      <controller type='pci' index='3' model='pcie-root-port'>
>>        <model name='pcie-root-port'/>
>>        <target chassis='3' port='0x12'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>>      </controller>
>>      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>        <model name='i82801b11-bridge'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>>      </controller>
>>      <controller type='pci' index='5' model='pci-bridge'>
>>        <model name='pci-bridge'/>
>>        <target chassisNr='5'/>
>>        <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>>      </controller>
>>      <interface type='bridge'>
>>        <mac address='00:50:56:1b:dd:6f'/>
>>        <source bridge='br0'/>
>>        <model type='virtio'/>
>>        <boot order='1'/>
>>        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>      </interface>
>>      <serial type='pty'>
>>        <target port='0'/>
>>      </serial>
>>      <console type='pty'>
>>        <target type='serial' port='0'/>
>>      </console>
>>      <input type='tablet' bus='usb'>
>>        <address type='usb' bus='0' port='1'/>
>>      </input>
>>      <input type='mouse' bus='ps2'/>
>>      <input type='keyboard' bus='ps2'/>
>>      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>>        <listen type='address' address='127.0.0.1'/>
>>      </graphics>
>>      <video>
>>        <model type='vga' vram='16384' heads='1' primary='yes'/>
>>        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>>      </video>
>>      <memballoon model='virtio'>
>>        <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>>      </memballoon>
>>    </devices>
>>    <seclabel type='none' model='none'/>
>>    <seclabel type='dynamic' model='dac' relabel='yes'/>
>>    <qemu:commandline>
>>      <qemu:arg value='-device'/>
>>      <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>>      <qemu:arg value='-netdev'/>
>>      <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>      <qemu:arg value='-device'/>
>>      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>    </qemu:commandline>
>> </domain>
>> "
>> ______________________
>> General info:
>> Host OS: RH7.0 (Kernel: 4.14.13)
>> QEMU version: 2.11
>> libvirt version: 3.2.0
>> Running the following on the host shows that both nested and IOMMU are
>> enabled:
>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>> Y
>> ~]# dmesg | grep -e DMAR -e IOMMU
>> [    0.000000] DMAR: IOMMU enabled
>>
>> Thanks,
>> Elijah
>>

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Michael S. Tsirkin]

On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> Hi
> 
> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > Hey,
> > > 
> > > Our team is adding a NIC functional  emulation to QEMU.
> > > One of the features we are adding to this NIC is SRIOV.
> > > 
> > > Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
> > > "
> > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
> 
> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> not sure ACS is a prerequisite
> for SR-IOV.

Looks like windows blocks dev assignment in nested VMs without it.
Thinking about it, doesn't vfio do the same by default? I think vfio has
a flag to override this though.

> We would need ARI support for allowing more than 8VFs, but QEMU
> doesn't support that either (yet).
> 
> Knut Omag has some working patches, he successfully implemented SR-IOV with
> QEMU, see:
>      https://github.com/knuto/qemu/tree/sriov_patches_v7
> 
> The code was not merged since we need at least a device with SR-IOV support
> to justify the addition.
> 
> > > Contact your system vendor for further information.
> > > "
> > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> 
> Thanks,
> Marcel
> 
> > > Could you please advise about what could be the issue here?
> > > 
> > > BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
> > > 
> > > Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
> > > "
> > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> > >    <name>nst105</name>
> > >    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > >    <memory unit='KiB'>8388608</memory>
> > >    <currentMemory unit='KiB'>8388608</currentMemory>
> > >    <vcpu placement='static'>8</vcpu>
> > >    <resource>
> > >      <partition>/machine</partition>
> > >    </resource>
> > >    <os>
> > >      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > >    </os>
> > >    <features>
> > >      <acpi/>
> > >      <apic/>
> > >      <hyperv>
> > >        <relaxed state='on'/>
> > >        <vapic state='on'/>
> > >        <spinlocks state='on' retries='8191'/>
> > >      </hyperv>
> > >    </features>
> > >    <cpu mode='custom' match='exact' check='none'>
> > >      <model fallback='forbid'>SandyBridge</model>
> > >      <feature policy='require' name='hypervisor'/>
> > >      <feature policy='require' name='vmx'/>
> > >    </cpu>
> > >    <clock offset='localtime'>
> > >      <timer name='rtc' tickpolicy='catchup'/>
> > >      <timer name='pit' tickpolicy='delay'/>
> > >      <timer name='hpet' present='no'/>
> > >      <timer name='hypervclock' present='yes'/>
> > >    </clock>
> > >    <on_poweroff>destroy</on_poweroff>
> > >    <on_reboot>restart</on_reboot>
> > >    <on_crash>destroy</on_crash>
> > >    <pm>
> > >      <suspend-to-mem enabled='no'/>
> > >      <suspend-to-disk enabled='no'/>
> > >    </pm>
> > >    <devices>
> > >      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > >      <disk type='file' device='disk'>
> > >        <driver name='qemu' type='qcow2'/>
> > >        <source file='/images/nst105.qcow2'/>
> > >        <target dev='sda' bus='sata'/>
> > >        <boot order='2'/>
> > >        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > >      </disk>
> > >      <controller type='usb' index='0' model='ich9-ehci1'>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
> > >      </controller>
> > >      <controller type='usb' index='0' model='ich9-uhci1'>
> > >        <master startport='0'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
> > >      </controller>
> > >      <controller type='usb' index='0' model='ich9-uhci2'>
> > >        <master startport='2'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
> > >      </controller>
> > >      <controller type='usb' index='0' model='ich9-uhci3'>
> > >        <master startport='4'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
> > >      </controller>
> > >      <controller type='sata' index='0'>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
> > >      </controller>
> > >      <controller type='pci' index='0' model='pcie-root'/>
> > >      <controller type='pci' index='1' model='pcie-root-port'>
> > >        <model name='pcie-root-port'/>
> > >        <target chassis='1' port='0x10'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
> > >      </controller>
> > >      <controller type='pci' index='2' model='pcie-root-port'>
> > >        <model name='pcie-root-port'/>
> > >        <target chassis='2' port='0x11'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
> > >      </controller>
> > >      <controller type='pci' index='3' model='pcie-root-port'>
> > >        <model name='pcie-root-port'/>
> > >        <target chassis='3' port='0x12'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
> > >      </controller>
> > >      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > >        <model name='i82801b11-bridge'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
> > >      </controller>
> > >      <controller type='pci' index='5' model='pci-bridge'>
> > >        <model name='pci-bridge'/>
> > >        <target chassisNr='5'/>
> > >        <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
> > >      </controller>
> > >      <interface type='bridge'>
> > >        <mac address='00:50:56:1b:dd:6f'/>
> > >        <source bridge='br0'/>
> > >        <model type='virtio'/>
> > >        <boot order='1'/>
> > >        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> > >      </interface>
> > >      <serial type='pty'>
> > >        <target port='0'/>
> > >      </serial>
> > >      <console type='pty'>
> > >        <target type='serial' port='0'/>
> > >      </console>
> > >      <input type='tablet' bus='usb'>
> > >        <address type='usb' bus='0' port='1'/>
> > >      </input>
> > >      <input type='mouse' bus='ps2'/>
> > >      <input type='keyboard' bus='ps2'/>
> > >      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
> > >        <listen type='address' address='127.0.0.1'/>
> > >      </graphics>
> > >      <video>
> > >        <model type='vga' vram='16384' heads='1' primary='yes'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
> > >      </video>
> > >      <memballoon model='virtio'>
> > >        <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
> > >      </memballoon>
> > >    </devices>
> > >    <seclabel type='none' model='none'/>
> > >    <seclabel type='dynamic' model='dac' relabel='yes'/>
> > >    <qemu:commandline>
> > >      <qemu:arg value='-device'/>
> > >      <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
> > >      <qemu:arg value='-netdev'/>
> > >      <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > >      <qemu:arg value='-device'/>
> > >      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > >    </qemu:commandline>
> > > </domain>
> > > "
> > > ______________________
> > > General info:
> > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > QEMU version: 2.11
> > > libvirt version: 3.2.0
> > > Running the following on the host shows that both nested and IOMMU are
> > > enabled:
> > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > Y
> > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > [    0.000000] DMAR: IOMMU enabled
> > > 
> > > Thanks,
> > > Elijah
> > > 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Marcel Apfelbaum]

On 07/26/2018 07:14 PM, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
>> Hi
>>
>> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
>>> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>>>> Hey,
>>>>
>>>> Our team is adding a NIC functional  emulation to QEMU.
>>>> One of the features we are adding to this NIC is SRIOV.
>>>>
>>>> Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
>>>> "
>>>> SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
>> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
>> not sure ACS is a prerequisite
>> for SR-IOV.
> Looks like windows blocks dev assignment in nested VMs without it.
> Thinking about it, doesn't vfio do the same by default?
vfio requires vIOMMU even for nested, as far as I know. They do have a 
"no-iommu"
option, but I don't know how  it can help in this case.

Elijah, what about implementing ACS for QEMU's generic PCIe Root Ports?
Maybe is not too complicated and it seems like a handy feature to have 
anyway.
Patches would be welcomed :)

Thanks,
Marcel

>   I think vfio has
> a flag to override this though.
>
>> We would need ARI support for allowing more than 8VFs, but QEMU
>> doesn't support that either (yet).
>>
>> Knut Omag has some working patches, he successfully implemented SR-IOV with
>> QEMU, see:
>>       https://github.com/knuto/qemu/tree/sriov_patches_v7
>>
>> The code was not merged since we need at least a device with SR-IOV support
>> to justify the addition.
>>
>>>> Contact your system vendor for further information.
>>>> "
>>> I'm not sure what the status of emulated SR-IOV is so I have CCed
>>> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>> Thanks,
>> Marcel
>>
>>>> Could you please advise about what could be the issue here?
>>>>
>>>> BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
>>>>
>>>> Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
>>>> "
>>>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>>>>     <name>nst105</name>
>>>>     <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>>>     <memory unit='KiB'>8388608</memory>
>>>>     <currentMemory unit='KiB'>8388608</currentMemory>
>>>>     <vcpu placement='static'>8</vcpu>
>>>>     <resource>
>>>>       <partition>/machine</partition>
>>>>     </resource>
>>>>     <os>
>>>>       <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>>>     </os>
>>>>     <features>
>>>>       <acpi/>
>>>>       <apic/>
>>>>       <hyperv>
>>>>         <relaxed state='on'/>
>>>>         <vapic state='on'/>
>>>>         <spinlocks state='on' retries='8191'/>
>>>>       </hyperv>
>>>>     </features>
>>>>     <cpu mode='custom' match='exact' check='none'>
>>>>       <model fallback='forbid'>SandyBridge</model>
>>>>       <feature policy='require' name='hypervisor'/>
>>>>       <feature policy='require' name='vmx'/>
>>>>     </cpu>
>>>>     <clock offset='localtime'>
>>>>       <timer name='rtc' tickpolicy='catchup'/>
>>>>       <timer name='pit' tickpolicy='delay'/>
>>>>       <timer name='hpet' present='no'/>
>>>>       <timer name='hypervclock' present='yes'/>
>>>>     </clock>
>>>>     <on_poweroff>destroy</on_poweroff>
>>>>     <on_reboot>restart</on_reboot>
>>>>     <on_crash>destroy</on_crash>
>>>>     <pm>
>>>>       <suspend-to-mem enabled='no'/>
>>>>       <suspend-to-disk enabled='no'/>
>>>>     </pm>
>>>>     <devices>
>>>>       <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>>>       <disk type='file' device='disk'>
>>>>         <driver name='qemu' type='qcow2'/>
>>>>         <source file='/images/nst105.qcow2'/>
>>>>         <target dev='sda' bus='sata'/>
>>>>         <boot order='2'/>
>>>>         <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>       </disk>
>>>>       <controller type='usb' index='0' model='ich9-ehci1'>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
>>>>       </controller>
>>>>       <controller type='usb' index='0' model='ich9-uhci1'>
>>>>         <master startport='0'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
>>>>       </controller>
>>>>       <controller type='usb' index='0' model='ich9-uhci2'>
>>>>         <master startport='2'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
>>>>       </controller>
>>>>       <controller type='usb' index='0' model='ich9-uhci3'>
>>>>         <master startport='4'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
>>>>       </controller>
>>>>       <controller type='sata' index='0'>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
>>>>       </controller>
>>>>       <controller type='pci' index='0' model='pcie-root'/>
>>>>       <controller type='pci' index='1' model='pcie-root-port'>
>>>>         <model name='pcie-root-port'/>
>>>>         <target chassis='1' port='0x10'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
>>>>       </controller>
>>>>       <controller type='pci' index='2' model='pcie-root-port'>
>>>>         <model name='pcie-root-port'/>
>>>>         <target chassis='2' port='0x11'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
>>>>       </controller>
>>>>       <controller type='pci' index='3' model='pcie-root-port'>
>>>>         <model name='pcie-root-port'/>
>>>>         <target chassis='3' port='0x12'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
>>>>       </controller>
>>>>       <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>>>         <model name='i82801b11-bridge'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
>>>>       </controller>
>>>>       <controller type='pci' index='5' model='pci-bridge'>
>>>>         <model name='pci-bridge'/>
>>>>         <target chassisNr='5'/>
>>>>         <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
>>>>       </controller>
>>>>       <interface type='bridge'>
>>>>         <mac address='00:50:56:1b:dd:6f'/>
>>>>         <source bridge='br0'/>
>>>>         <model type='virtio'/>
>>>>         <boot order='1'/>
>>>>         <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>>>       </interface>
>>>>       <serial type='pty'>
>>>>         <target port='0'/>
>>>>       </serial>
>>>>       <console type='pty'>
>>>>         <target type='serial' port='0'/>
>>>>       </console>
>>>>       <input type='tablet' bus='usb'>
>>>>         <address type='usb' bus='0' port='1'/>
>>>>       </input>
>>>>       <input type='mouse' bus='ps2'/>
>>>>       <input type='keyboard' bus='ps2'/>
>>>>       <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
>>>>         <listen type='address' address='127.0.0.1'/>
>>>>       </graphics>
>>>>       <video>
>>>>         <model type='vga' vram='16384' heads='1' primary='yes'/>
>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>>>>       </video>
>>>>       <memballoon model='virtio'>
>>>>         <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
>>>>       </memballoon>
>>>>     </devices>
>>>>     <seclabel type='none' model='none'/>
>>>>     <seclabel type='dynamic' model='dac' relabel='yes'/>
>>>>     <qemu:commandline>
>>>>       <qemu:arg value='-device'/>
>>>>       <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
>>>>       <qemu:arg value='-netdev'/>
>>>>       <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>>>       <qemu:arg value='-device'/>
>>>>       <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>>>     </qemu:commandline>
>>>> </domain>
>>>> "
>>>> ______________________
>>>> General info:
>>>> Host OS: RH7.0 (Kernel: 4.14.13)
>>>> QEMU version: 2.11
>>>> libvirt version: 3.2.0
>>>> Running the following on the host shows that both nested and IOMMU are
>>>> enabled:
>>>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>>>> Y
>>>> ~]# dmesg | grep -e DMAR -e IOMMU
>>>> [    0.000000] DMAR: IOMMU enabled
>>>>
>>>> Thanks,
>>>> Elijah
>>>>

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Michael S. Tsirkin]

On Thu, Jul 26, 2018 at 07:34:41PM +0300, Marcel Apfelbaum wrote:
> 
> 
> On 07/26/2018 07:14 PM, Michael S. Tsirkin wrote:
> > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > Hi
> > > 
> > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > Hey,
> > > > > 
> > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > 
> > > > > Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
> > > > > "
> > > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.
> > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > > not sure ACS is a prerequisite
> > > for SR-IOV.
> > Looks like windows blocks dev assignment in nested VMs without it.
> > Thinking about it, doesn't vfio do the same by default?
> vfio requires vIOMMU even for nested, as far as I know.

Absolutely.

> They do have a
> "no-iommu"
> option, but I don't know how  it can help in this case.
> 
> Elijah, what about implementing ACS for QEMU's generic PCIe Root Ports?
> Maybe is not too complicated and it seems like a handy feature to have
> anyway.
> Patches would be welcomed :)
> 
> Thanks,
> Marcel
> 
> >   I think vfio has
> > a flag to override this though.
> > 
> > > We would need ARI support for allowing more than 8VFs, but QEMU
> > > doesn't support that either (yet).
> > > 
> > > Knut Omag has some working patches, he successfully implemented SR-IOV with
> > > QEMU, see:
> > >       https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > 
> > > The code was not merged since we need at least a device with SR-IOV support
> > > to justify the addition.
> > > 
> > > > > Contact your system vendor for further information.
> > > > > "
> > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > Thanks,
> > > Marcel
> > > 
> > > > > Could you please advise about what could be the issue here?
> > > > > 
> > > > > BTW: I use same configuration (VM XML file attached) when running linux VM (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is working for me there.
> > > > > 
> > > > > Here the XML file I use to define the VM (our emulated NIC is added at the end of XML):
> > > > > "
> > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> > > > >     <name>nst105</name>
> > > > >     <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > >     <memory unit='KiB'>8388608</memory>
> > > > >     <currentMemory unit='KiB'>8388608</currentMemory>
> > > > >     <vcpu placement='static'>8</vcpu>
> > > > >     <resource>
> > > > >       <partition>/machine</partition>
> > > > >     </resource>
> > > > >     <os>
> > > > >       <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > >     </os>
> > > > >     <features>
> > > > >       <acpi/>
> > > > >       <apic/>
> > > > >       <hyperv>
> > > > >         <relaxed state='on'/>
> > > > >         <vapic state='on'/>
> > > > >         <spinlocks state='on' retries='8191'/>
> > > > >       </hyperv>
> > > > >     </features>
> > > > >     <cpu mode='custom' match='exact' check='none'>
> > > > >       <model fallback='forbid'>SandyBridge</model>
> > > > >       <feature policy='require' name='hypervisor'/>
> > > > >       <feature policy='require' name='vmx'/>
> > > > >     </cpu>
> > > > >     <clock offset='localtime'>
> > > > >       <timer name='rtc' tickpolicy='catchup'/>
> > > > >       <timer name='pit' tickpolicy='delay'/>
> > > > >       <timer name='hpet' present='no'/>
> > > > >       <timer name='hypervclock' present='yes'/>
> > > > >     </clock>
> > > > >     <on_poweroff>destroy</on_poweroff>
> > > > >     <on_reboot>restart</on_reboot>
> > > > >     <on_crash>destroy</on_crash>
> > > > >     <pm>
> > > > >       <suspend-to-mem enabled='no'/>
> > > > >       <suspend-to-disk enabled='no'/>
> > > > >     </pm>
> > > > >     <devices>
> > > > >       <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > >       <disk type='file' device='disk'>
> > > > >         <driver name='qemu' type='qcow2'/>
> > > > >         <source file='/images/nst105.qcow2'/>
> > > > >         <target dev='sda' bus='sata'/>
> > > > >         <boot order='2'/>
> > > > >         <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > > >       </disk>
> > > > >       <controller type='usb' index='0' model='ich9-ehci1'>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
> > > > >       </controller>
> > > > >       <controller type='usb' index='0' model='ich9-uhci1'>
> > > > >         <master startport='0'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
> > > > >       </controller>
> > > > >       <controller type='usb' index='0' model='ich9-uhci2'>
> > > > >         <master startport='2'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
> > > > >       </controller>
> > > > >       <controller type='usb' index='0' model='ich9-uhci3'>
> > > > >         <master startport='4'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
> > > > >       </controller>
> > > > >       <controller type='sata' index='0'>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
> > > > >       </controller>
> > > > >       <controller type='pci' index='0' model='pcie-root'/>
> > > > >       <controller type='pci' index='1' model='pcie-root-port'>
> > > > >         <model name='pcie-root-port'/>
> > > > >         <target chassis='1' port='0x10'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
> > > > >       </controller>
> > > > >       <controller type='pci' index='2' model='pcie-root-port'>
> > > > >         <model name='pcie-root-port'/>
> > > > >         <target chassis='2' port='0x11'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
> > > > >       </controller>
> > > > >       <controller type='pci' index='3' model='pcie-root-port'>
> > > > >         <model name='pcie-root-port'/>
> > > > >         <target chassis='3' port='0x12'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
> > > > >       </controller>
> > > > >       <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > >         <model name='i82801b11-bridge'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
> > > > >       </controller>
> > > > >       <controller type='pci' index='5' model='pci-bridge'>
> > > > >         <model name='pci-bridge'/>
> > > > >         <target chassisNr='5'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
> > > > >       </controller>
> > > > >       <interface type='bridge'>
> > > > >         <mac address='00:50:56:1b:dd:6f'/>
> > > > >         <source bridge='br0'/>
> > > > >         <model type='virtio'/>
> > > > >         <boot order='1'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> > > > >       </interface>
> > > > >       <serial type='pty'>
> > > > >         <target port='0'/>
> > > > >       </serial>
> > > > >       <console type='pty'>
> > > > >         <target type='serial' port='0'/>
> > > > >       </console>
> > > > >       <input type='tablet' bus='usb'>
> > > > >         <address type='usb' bus='0' port='1'/>
> > > > >       </input>
> > > > >       <input type='mouse' bus='ps2'/>
> > > > >       <input type='keyboard' bus='ps2'/>
> > > > >       <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
> > > > >         <listen type='address' address='127.0.0.1'/>
> > > > >       </graphics>
> > > > >       <video>
> > > > >         <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
> > > > >       </video>
> > > > >       <memballoon model='virtio'>
> > > > >         <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
> > > > >       </memballoon>
> > > > >     </devices>
> > > > >     <seclabel type='none' model='none'/>
> > > > >     <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > >     <qemu:commandline>
> > > > >       <qemu:arg value='-device'/>
> > > > >       <qemu:arg value='pcie-root-port,pref64-reserve=500M,slot=0,id=pcie_port.1'/>
> > > > >       <qemu:arg value='-netdev'/>
> > > > >       <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > >       <qemu:arg value='-device'/>
> > > > >       <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > >     </qemu:commandline>
> > > > > </domain>
> > > > > "
> > > > > ______________________
> > > > > General info:
> > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > QEMU version: 2.11
> > > > > libvirt version: 3.2.0
> > > > > Running the following on the host shows that both nested and IOMMU are
> > > > > enabled:
> > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > Y
> > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > [    0.000000] DMAR: IOMMU enabled
> > > > > 
> > > > > Thanks,
> > > > > Elijah
> > > > > 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Alex Williamson]

On Thu, 26 Jul 2018 19:14:45 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > Hi
> > 
> > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:  
> > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:  
> > > > Hey,
> > > > 
> > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > One of the features we are adding to this NIC is SRIOV.
> > > > 
> > > > Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
> > > > "
> > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.  
> > 
> > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > not sure ACS is a prerequisite
> > for SR-IOV.

ACS is certainly not a prerequisite for SR-IOV.
 
> Looks like windows blocks dev assignment in nested VMs without it.
> Thinking about it, doesn't vfio do the same by default? I think vfio has
> a flag to override this though.

IOMMU grouping in Linux takes isolation via ACS and device specific
mechanisms into account, limiting the granularity with which a
userspace driver can claim ownership of devices, but it doesn't
actually prevent enabling SR-IOV on the endpoint.  It just makes it
less useful if your intention is to use SR-IOV for device assignment.
The only overrides for this are out-of-tree.  Thanks,

Alex

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Michael S. Tsirkin]

On Thu, Jul 26, 2018 at 10:42:52AM -0600, Alex Williamson wrote:
> On Thu, 26 Jul 2018 19:14:45 +0300
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > Hi
> > > 
> > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:  
> > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:  
> > > > > Hey,
> > > > > 
> > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > 
> > > > > Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
> > > > > "
> > > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.  
> > > 
> > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > > not sure ACS is a prerequisite
> > > for SR-IOV.
> 
> ACS is certainly not a prerequisite for SR-IOV.
>  
> > Looks like windows blocks dev assignment in nested VMs without it.
> > Thinking about it, doesn't vfio do the same by default? I think vfio has
> > a flag to override this though.
> 
> IOMMU grouping in Linux takes isolation via ACS and device specific
> mechanisms into account, limiting the granularity with which a
> userspace driver can claim ownership of devices,

In that you must assign all devices behind this port as a group?

> but it doesn't
> actually prevent enabling SR-IOV on the endpoint.  It just makes it
> less useful if your intention is to use SR-IOV for device assignment.
> The only overrides for this are out-of-tree.  Thanks,
> 
> Alex

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Knut Omang]

On Thu, 2018-07-26 at 20:11 +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 10:42:52AM -0600, Alex Williamson wrote:
> > On Thu, 26 Jul 2018 19:14:45 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:
> > > > Hi
> > > > 
> > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:  
> > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:  
> > > > > > Hey,
> > > > > > 
> > > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > 
> > > > > > Here is the error message I get when checking SRIOV support of  our
> > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > "
> > > > > > SR-IOV cannot be used on this system as the PCI Express hardware
> > > > > > does not support Access Control Services (ACS) at any root port.  
> > > > 
> > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
> > > > am
> > > > not sure ACS is a prerequisite
> > > > for SR-IOV.
> > 
> > ACS is certainly not a prerequisite for SR-IOV.
> >  
> > > Looks like windows blocks dev assignment in nested VMs without it.
> > > Thinking about it, doesn't vfio do the same by default? I think vfio has
> > > a flag to override this though.
> > 
> > IOMMU grouping in Linux takes isolation via ACS and device specific
> > mechanisms into account, limiting the granularity with which a
> > userspace driver can claim ownership of devices,
> 
> In that you must assign all devices behind this port as a group?

Yes, that's why I needed to add PCIe config space ACS support in ioh3420:
to be able to use VFIO to pass through individual VFs to different L2 VMs with
"plain" kernels on the L1 host which owns the SR/IOV device.

Knut

> > but it doesn't
> > actually prevent enabling SR-IOV on the endpoint.  It just makes it
> > less useful if your intention is to use SR-IOV for device assignment.
> > The only overrides for this are out-of-tree.  Thanks,
> >
> > 
> > Alex
> 
> 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Alex Williamson]

On Thu, 26 Jul 2018 20:11:44 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Thu, Jul 26, 2018 at 10:42:52AM -0600, Alex Williamson wrote:
> > On Thu, 26 Jul 2018 19:14:45 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Thu, Jul 26, 2018 at 06:51:13PM +0300, Marcel Apfelbaum wrote:  
> > > > Hi
> > > > 
> > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:    
> > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:    
> > > > > > Hey,
> > > > > > 
> > > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > 
> > > > > > Here is the error message I get when checking SRIOV support of  our emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > "
> > > > > > SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port.    
> > > > 
> > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I am
> > > > not sure ACS is a prerequisite
> > > > for SR-IOV.  
> > 
> > ACS is certainly not a prerequisite for SR-IOV.
> >    
> > > Looks like windows blocks dev assignment in nested VMs without it.
> > > Thinking about it, doesn't vfio do the same by default? I think vfio has
> > > a flag to override this though.  
> > 
> > IOMMU grouping in Linux takes isolation via ACS and device specific
> > mechanisms into account, limiting the granularity with which a
> > userspace driver can claim ownership of devices,  
> 
> In that you must assign all devices behind this port as a group?

Yup.  Not very useful with SR-IOV.  Thanks,

Alex

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Knut Omang]

On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> Hi
> 
> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > Hey,
> > > 
> > > Our team is adding a NIC functional  emulation to QEMU.
> > > One of the features we are adding to this NIC is SRIOV.
> > > 
> > > Here is the error message I get when checking SRIOV support of  our
> > > emulated NIC on Win2016 server (the hyper-v VM).
> > > "
> > > SR-IOV cannot be used on this system as the PCI Express hardware does not
> > > support Access Control Services (ACS) at any root port.
> 
> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I 
> am not sure ACS is a prerequisite
> for SR-IOV. We would need ARI support for allowing more than 8VFs, but 
> QEMU doesn't support that either (yet).
> 
> Knut Omag has some working patches, he successfully implemented SR-IOV 
> with QEMU, see:
>       https://github.com/knuto/qemu/tree/sriov_patches_v7
> 
> The code was not merged since we need at least a device with SR-IOV 
> support to justify the addition.

FYI, I recently rebased these to latest master but just didn't get to push them
out. I did just now - they are available here:

   https://github.com/knuto/qemu/tree/sriov_patches_v8

As far as I know ARI support with my patch set works just fine - I have tested
it with lots of VFs.

One of the patches in the series (pci: Make use of the devfn property when
registering new devices) is necessary to make ARI work as it should with SR/IOV.

For the hardware model I developed the SR/IOV patches for, I also added enough
ACS support in the root port (PCIe capability helper patch + usage in
ioh3420) to make VFIO "happy". I haven't submitted them because they
are "questionable" since they likely do not reflect the actual features of the
ioh3420. I can make those available if interesting.

Thanks,
Knut

> > > Contact your system vendor for further information.
> > > "
> > 
> > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> 
> Thanks,
> Marcel
> 
> > > Could you please advise about what could be the issue here?
> > > 
> > > BTW: I use same configuration (VM XML file attached) when running linux VM
> > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > working for me there.
> > > 
> > > Here the XML file I use to define the VM (our emulated NIC is added at the
> > > end of XML):
> > > "
> > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > > >;
> > >    <name>nst105</name>
> > >    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > >    <memory unit='KiB'>8388608</memory>
> > >    <currentMemory unit='KiB'>8388608</currentMemory>
> > >    <vcpu placement='static'>8</vcpu>
> > >    <resource>
> > >      <partition>/machine</partition>
> > >    </resource>
> > >    <os>
> > >      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > >    </os>
> > >    <features>
> > >      <acpi/>
> > >      <apic/>
> > >      <hyperv>
> > >        <relaxed state='on'/>
> > >        <vapic state='on'/>
> > >        <spinlocks state='on' retries='8191'/>
> > >      </hyperv>
> > >    </features>
> > >    <cpu mode='custom' match='exact' check='none'>
> > >      <model fallback='forbid'>SandyBridge</model>
> > >      <feature policy='require' name='hypervisor'/>
> > >      <feature policy='require' name='vmx'/>
> > >    </cpu>
> > >    <clock offset='localtime'>
> > >      <timer name='rtc' tickpolicy='catchup'/>
> > >      <timer name='pit' tickpolicy='delay'/>
> > >      <timer name='hpet' present='no'/>
> > >      <timer name='hypervclock' present='yes'/>
> > >    </clock>
> > >    <on_poweroff>destroy</on_poweroff>
> > >    <on_reboot>restart</on_reboot>
> > >    <on_crash>destroy</on_crash>
> > >    <pm>
> > >      <suspend-to-mem enabled='no'/>
> > >      <suspend-to-disk enabled='no'/>
> > >    </pm>
> > >    <devices>
> > >      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > >      <disk type='file' device='disk'>
> > >        <driver name='qemu' type='qcow2'/>
> > >        <source file='/images/nst105.qcow2'/>
> > >        <target dev='sda' bus='sata'/>
> > >        <boot order='2'/>
> > >        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > >      </disk>
> > >      <controller type='usb' index='0' model='ich9-ehci1'>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x7'/>
> > >      </controller>
> > >      <controller type='usb' index='0' model='ich9-uhci1'>
> > >        <master startport='0'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x0' multifunction='on'/>
> > >      </controller>
> > >      <controller type='usb' index='0' model='ich9-uhci2'>
> > >        <master startport='2'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x1'/>
> > >      </controller>
> > >      <controller type='usb' index='0' model='ich9-uhci3'>
> > >        <master startport='4'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > function='0x2'/>
> > >      </controller>
> > >      <controller type='sata' index='0'>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > function='0x2'/>
> > >      </controller>
> > >      <controller type='pci' index='0' model='pcie-root'/>
> > >      <controller type='pci' index='1' model='pcie-root-port'>
> > >        <model name='pcie-root-port'/>
> > >        <target chassis='1' port='0x10'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x0' multifunction='on'/>
> > >      </controller>
> > >      <controller type='pci' index='2' model='pcie-root-port'>
> > >        <model name='pcie-root-port'/>
> > >        <target chassis='2' port='0x11'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x1'/>
> > >      </controller>
> > >      <controller type='pci' index='3' model='pcie-root-port'>
> > >        <model name='pcie-root-port'/>
> > >        <target chassis='3' port='0x12'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > function='0x2'/>
> > >      </controller>
> > >      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > >        <model name='i82801b11-bridge'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > function='0x0'/>
> > >      </controller>
> > >      <controller type='pci' index='5' model='pci-bridge'>
> > >        <model name='pci-bridge'/>
> > >        <target chassisNr='5'/>
> > >        <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > function='0x0'/>
> > >      </controller>
> > >      <interface type='bridge'>
> > >        <mac address='00:50:56:1b:dd:6f'/>
> > >        <source bridge='br0'/>
> > >        <model type='virtio'/>
> > >        <boot order='1'/>
> > >        <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > function='0x0'/>
> > >      </interface>
> > >      <serial type='pty'>
> > >        <target port='0'/>
> > >      </serial>
> > >      <console type='pty'>
> > >        <target type='serial' port='0'/>
> > >      </console>
> > >      <input type='tablet' bus='usb'>
> > >        <address type='usb' bus='0' port='1'/>
> > >      </input>
> > >      <input type='mouse' bus='ps2'/>
> > >      <input type='keyboard' bus='ps2'/>
> > >      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > keymap='en-us'>
> > >        <listen type='address' address='127.0.0.1'/>
> > >      </graphics>
> > >      <video>
> > >        <model type='vga' vram='16384' heads='1' primary='yes'/>
> > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > function='0x0'/>
> > >      </video>
> > >      <memballoon model='virtio'>
> > >        <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > function='0x0'/>
> > >      </memballoon>
> > >    </devices>
> > >    <seclabel type='none' model='none'/>
> > >    <seclabel type='dynamic' model='dac' relabel='yes'/>
> > >    <qemu:commandline>
> > >      <qemu:arg value='-device'/>
> > >      <qemu:arg value='pcie-root-port,pref64-
> > > reserve=500M,slot=0,id=pcie_port.1'/>
> > >      <qemu:arg value='-netdev'/>
> > >      <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > >      <qemu:arg value='-device'/>
> > >      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > >    </qemu:commandline>
> > > </domain>
> > > "
> > > ______________________
> > > General info:
> > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > QEMU version: 2.11
> > > libvirt version: 3.2.0
> > > Running the following on the host shows that both nested and IOMMU are
> > > enabled:
> > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > Y
> > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > [    0.000000] DMAR: IOMMU enabled
> > > 
> > > Thanks,
> > > Elijah
> > > 
> 
> 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Michael S. Tsirkin]

On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > Hi
> > 
> > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > Hey,
> > > > 
> > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > One of the features we are adding to this NIC is SRIOV.
> > > > 
> > > > Here is the error message I get when checking SRIOV support of  our
> > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > "
> > > > SR-IOV cannot be used on this system as the PCI Express hardware does not
> > > > support Access Control Services (ACS) at any root port.
> > 
> > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I 
> > am not sure ACS is a prerequisite
> > for SR-IOV. We would need ARI support for allowing more than 8VFs, but 
> > QEMU doesn't support that either (yet).
> > 
> > Knut Omag has some working patches, he successfully implemented SR-IOV 
> > with QEMU, see:
> >       https://github.com/knuto/qemu/tree/sriov_patches_v7
> > 
> > The code was not merged since we need at least a device with SR-IOV 
> > support to justify the addition.
> 
> FYI, I recently rebased these to latest master but just didn't get to push them
> out. I did just now - they are available here:
> 
>    https://github.com/knuto/qemu/tree/sriov_patches_v8
> 
> As far as I know ARI support with my patch set works just fine - I have tested
> it with lots of VFs.
> 
> One of the patches in the series (pci: Make use of the devfn property when
> registering new devices) is necessary to make ARI work as it should with SR/IOV.
> 
> For the hardware model I developed the SR/IOV patches for, I also added enough
> ACS support in the root port (PCIe capability helper patch + usage in
> ioh3420) to make VFIO "happy". I haven't submitted them because they
> are "questionable" since they likely do not reflect the actual features of the
> ioh3420.

In that the actual ioh3420 doesn't support ACS?

> I can make those available if interesting.
> 
> Thanks,
> Knut
> 
> > > > Contact your system vendor for further information.
> > > > "
> > > 
> > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > 
> > Thanks,
> > Marcel
> > 
> > > > Could you please advise about what could be the issue here?
> > > > 
> > > > BTW: I use same configuration (VM XML file attached) when running linux VM
> > > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > > working for me there.
> > > > 
> > > > Here the XML file I use to define the VM (our emulated NIC is added at the
> > > > end of XML):
> > > > "
> > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > > > >;
> > > >    <name>nst105</name>
> > > >    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > >    <memory unit='KiB'>8388608</memory>
> > > >    <currentMemory unit='KiB'>8388608</currentMemory>
> > > >    <vcpu placement='static'>8</vcpu>
> > > >    <resource>
> > > >      <partition>/machine</partition>
> > > >    </resource>
> > > >    <os>
> > > >      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > >    </os>
> > > >    <features>
> > > >      <acpi/>
> > > >      <apic/>
> > > >      <hyperv>
> > > >        <relaxed state='on'/>
> > > >        <vapic state='on'/>
> > > >        <spinlocks state='on' retries='8191'/>
> > > >      </hyperv>
> > > >    </features>
> > > >    <cpu mode='custom' match='exact' check='none'>
> > > >      <model fallback='forbid'>SandyBridge</model>
> > > >      <feature policy='require' name='hypervisor'/>
> > > >      <feature policy='require' name='vmx'/>
> > > >    </cpu>
> > > >    <clock offset='localtime'>
> > > >      <timer name='rtc' tickpolicy='catchup'/>
> > > >      <timer name='pit' tickpolicy='delay'/>
> > > >      <timer name='hpet' present='no'/>
> > > >      <timer name='hypervclock' present='yes'/>
> > > >    </clock>
> > > >    <on_poweroff>destroy</on_poweroff>
> > > >    <on_reboot>restart</on_reboot>
> > > >    <on_crash>destroy</on_crash>
> > > >    <pm>
> > > >      <suspend-to-mem enabled='no'/>
> > > >      <suspend-to-disk enabled='no'/>
> > > >    </pm>
> > > >    <devices>
> > > >      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > >      <disk type='file' device='disk'>
> > > >        <driver name='qemu' type='qcow2'/>
> > > >        <source file='/images/nst105.qcow2'/>
> > > >        <target dev='sda' bus='sata'/>
> > > >        <boot order='2'/>
> > > >        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> > > >      </disk>
> > > >      <controller type='usb' index='0' model='ich9-ehci1'>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x7'/>
> > > >      </controller>
> > > >      <controller type='usb' index='0' model='ich9-uhci1'>
> > > >        <master startport='0'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x0' multifunction='on'/>
> > > >      </controller>
> > > >      <controller type='usb' index='0' model='ich9-uhci2'>
> > > >        <master startport='2'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x1'/>
> > > >      </controller>
> > > >      <controller type='usb' index='0' model='ich9-uhci3'>
> > > >        <master startport='4'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > function='0x2'/>
> > > >      </controller>
> > > >      <controller type='sata' index='0'>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > function='0x2'/>
> > > >      </controller>
> > > >      <controller type='pci' index='0' model='pcie-root'/>
> > > >      <controller type='pci' index='1' model='pcie-root-port'>
> > > >        <model name='pcie-root-port'/>
> > > >        <target chassis='1' port='0x10'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > function='0x0' multifunction='on'/>
> > > >      </controller>
> > > >      <controller type='pci' index='2' model='pcie-root-port'>
> > > >        <model name='pcie-root-port'/>
> > > >        <target chassis='2' port='0x11'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > function='0x1'/>
> > > >      </controller>
> > > >      <controller type='pci' index='3' model='pcie-root-port'>
> > > >        <model name='pcie-root-port'/>
> > > >        <target chassis='3' port='0x12'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > function='0x2'/>
> > > >      </controller>
> > > >      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > >        <model name='i82801b11-bridge'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > function='0x0'/>
> > > >      </controller>
> > > >      <controller type='pci' index='5' model='pci-bridge'>
> > > >        <model name='pci-bridge'/>
> > > >        <target chassisNr='5'/>
> > > >        <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > function='0x0'/>
> > > >      </controller>
> > > >      <interface type='bridge'>
> > > >        <mac address='00:50:56:1b:dd:6f'/>
> > > >        <source bridge='br0'/>
> > > >        <model type='virtio'/>
> > > >        <boot order='1'/>
> > > >        <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > function='0x0'/>
> > > >      </interface>
> > > >      <serial type='pty'>
> > > >        <target port='0'/>
> > > >      </serial>
> > > >      <console type='pty'>
> > > >        <target type='serial' port='0'/>
> > > >      </console>
> > > >      <input type='tablet' bus='usb'>
> > > >        <address type='usb' bus='0' port='1'/>
> > > >      </input>
> > > >      <input type='mouse' bus='ps2'/>
> > > >      <input type='keyboard' bus='ps2'/>
> > > >      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > > keymap='en-us'>
> > > >        <listen type='address' address='127.0.0.1'/>
> > > >      </graphics>
> > > >      <video>
> > > >        <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > function='0x0'/>
> > > >      </video>
> > > >      <memballoon model='virtio'>
> > > >        <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > function='0x0'/>
> > > >      </memballoon>
> > > >    </devices>
> > > >    <seclabel type='none' model='none'/>
> > > >    <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > >    <qemu:commandline>
> > > >      <qemu:arg value='-device'/>
> > > >      <qemu:arg value='pcie-root-port,pref64-
> > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > >      <qemu:arg value='-netdev'/>
> > > >      <qemu:arg value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > >      <qemu:arg value='-device'/>
> > > >      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > >    </qemu:commandline>
> > > > </domain>
> > > > "
> > > > ______________________
> > > > General info:
> > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > QEMU version: 2.11
> > > > libvirt version: 3.2.0
> > > > Running the following on the host shows that both nested and IOMMU are
> > > > enabled:
> > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > Y
> > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > [    0.000000] DMAR: IOMMU enabled
> > > > 
> > > > Thanks,
> > > > Elijah
> > > > 
> > 
> > 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Knut Omang]

On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > Hi
> > > 
> > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > Hey,
> > > > > 
> > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > 
> > > > > Here is the error message I get when checking SRIOV support of  our
> > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > "
> > > > > SR-IOV cannot be used on this system as the PCI Express hardware does
> > > > > not
> > > > > support Access Control Services (ACS) at any root port.
> > > 
> > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I 
> > > am not sure ACS is a prerequisite
> > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but 
> > > QEMU doesn't support that either (yet).
> > > 
> > > Knut Omag has some working patches, he successfully implemented SR-IOV 
> > > with QEMU, see:
> > >       https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > 
> > > The code was not merged since we need at least a device with SR-IOV 
> > > support to justify the addition.
> > 
> > FYI, I recently rebased these to latest master but just didn't get to push
> > them
> > out. I did just now - they are available here:
> > 
> >    https://github.com/knuto/qemu/tree/sriov_patches_v8
> > 
> > As far as I know ARI support with my patch set works just fine - I have
> > tested
> > it with lots of VFs.
> > 
> > One of the patches in the series (pci: Make use of the devfn property when
> > registering new devices) is necessary to make ARI work as it should with
> > SR/IOV.
> > 
> > For the hardware model I developed the SR/IOV patches for, I also added
> > enough
> > ACS support in the root port (PCIe capability helper patch + usage in
> > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > are "questionable" since they likely do not reflect the actual features of
> > the
> > ioh3420.
> 
> In that the actual ioh3420 doesn't support ACS?

yes.. I don't have one so I don't know but that was my assumption..

Knut

> > I can make those available if interesting.
> > 
> > Thanks,
> > Knut
> > 
> > > > > Contact your system vendor for further information.
> > > > > "
> > > > 
> > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > 
> > > Thanks,
> > > Marcel
> > > 
> > > > > Could you please advise about what could be the issue here?
> > > > > 
> > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > linux VM
> > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > > > working for me there.
> > > > > 
> > > > > Here the XML file I use to define the VM (our emulated NIC is added at
> > > > > the
> > > > > end of XML):
> > > > > "
> > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/
> > > > > 1.0'
> > > > > > ;
> > > > > 
> > > > >    <name>nst105</name>
> > > > >    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > >    <memory unit='KiB'>8388608</memory>
> > > > >    <currentMemory unit='KiB'>8388608</currentMemory>
> > > > >    <vcpu placement='static'>8</vcpu>
> > > > >    <resource>
> > > > >      <partition>/machine</partition>
> > > > >    </resource>
> > > > >    <os>
> > > > >      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > >    </os>
> > > > >    <features>
> > > > >      <acpi/>
> > > > >      <apic/>
> > > > >      <hyperv>
> > > > >        <relaxed state='on'/>
> > > > >        <vapic state='on'/>
> > > > >        <spinlocks state='on' retries='8191'/>
> > > > >      </hyperv>
> > > > >    </features>
> > > > >    <cpu mode='custom' match='exact' check='none'>
> > > > >      <model fallback='forbid'>SandyBridge</model>
> > > > >      <feature policy='require' name='hypervisor'/>
> > > > >      <feature policy='require' name='vmx'/>
> > > > >    </cpu>
> > > > >    <clock offset='localtime'>
> > > > >      <timer name='rtc' tickpolicy='catchup'/>
> > > > >      <timer name='pit' tickpolicy='delay'/>
> > > > >      <timer name='hpet' present='no'/>
> > > > >      <timer name='hypervclock' present='yes'/>
> > > > >    </clock>
> > > > >    <on_poweroff>destroy</on_poweroff>
> > > > >    <on_reboot>restart</on_reboot>
> > > > >    <on_crash>destroy</on_crash>
> > > > >    <pm>
> > > > >      <suspend-to-mem enabled='no'/>
> > > > >      <suspend-to-disk enabled='no'/>
> > > > >    </pm>
> > > > >    <devices>
> > > > >      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > >      <disk type='file' device='disk'>
> > > > >        <driver name='qemu' type='qcow2'/>
> > > > >        <source file='/images/nst105.qcow2'/>
> > > > >        <target dev='sda' bus='sata'/>
> > > > >        <boot order='2'/>
> > > > >        <address type='drive' controller='0' bus='0' target='0'
> > > > > unit='0'/>
> > > > >      </disk>
> > > > >      <controller type='usb' index='0' model='ich9-ehci1'>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x7'/>
> > > > >      </controller>
> > > > >      <controller type='usb' index='0' model='ich9-uhci1'>
> > > > >        <master startport='0'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x0' multifunction='on'/>
> > > > >      </controller>
> > > > >      <controller type='usb' index='0' model='ich9-uhci2'>
> > > > >        <master startport='2'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x1'/>
> > > > >      </controller>
> > > > >      <controller type='usb' index='0' model='ich9-uhci3'>
> > > > >        <master startport='4'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > function='0x2'/>
> > > > >      </controller>
> > > > >      <controller type='sata' index='0'>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > function='0x2'/>
> > > > >      </controller>
> > > > >      <controller type='pci' index='0' model='pcie-root'/>
> > > > >      <controller type='pci' index='1' model='pcie-root-port'>
> > > > >        <model name='pcie-root-port'/>
> > > > >        <target chassis='1' port='0x10'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > function='0x0' multifunction='on'/>
> > > > >      </controller>
> > > > >      <controller type='pci' index='2' model='pcie-root-port'>
> > > > >        <model name='pcie-root-port'/>
> > > > >        <target chassis='2' port='0x11'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > function='0x1'/>
> > > > >      </controller>
> > > > >      <controller type='pci' index='3' model='pcie-root-port'>
> > > > >        <model name='pcie-root-port'/>
> > > > >        <target chassis='3' port='0x12'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > function='0x2'/>
> > > > >      </controller>
> > > > >      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > >        <model name='i82801b11-bridge'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > function='0x0'/>
> > > > >      </controller>
> > > > >      <controller type='pci' index='5' model='pci-bridge'>
> > > > >        <model name='pci-bridge'/>
> > > > >        <target chassisNr='5'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > function='0x0'/>
> > > > >      </controller>
> > > > >      <interface type='bridge'>
> > > > >        <mac address='00:50:56:1b:dd:6f'/>
> > > > >        <source bridge='br0'/>
> > > > >        <model type='virtio'/>
> > > > >        <boot order='1'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > function='0x0'/>
> > > > >      </interface>
> > > > >      <serial type='pty'>
> > > > >        <target port='0'/>
> > > > >      </serial>
> > > > >      <console type='pty'>
> > > > >        <target type='serial' port='0'/>
> > > > >      </console>
> > > > >      <input type='tablet' bus='usb'>
> > > > >        <address type='usb' bus='0' port='1'/>
> > > > >      </input>
> > > > >      <input type='mouse' bus='ps2'/>
> > > > >      <input type='keyboard' bus='ps2'/>
> > > > >      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > > > keymap='en-us'>
> > > > >        <listen type='address' address='127.0.0.1'/>
> > > > >      </graphics>
> > > > >      <video>
> > > > >        <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > function='0x0'/>
> > > > >      </video>
> > > > >      <memballoon model='virtio'>
> > > > >        <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > function='0x0'/>
> > > > >      </memballoon>
> > > > >    </devices>
> > > > >    <seclabel type='none' model='none'/>
> > > > >    <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > >    <qemu:commandline>
> > > > >      <qemu:arg value='-device'/>
> > > > >      <qemu:arg value='pcie-root-port,pref64-
> > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > >      <qemu:arg value='-netdev'/>
> > > > >      <qemu:arg
> > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > >      <qemu:arg value='-device'/>
> > > > >      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > >    </qemu:commandline>
> > > > > </domain>
> > > > > "
> > > > > ______________________
> > > > > General info:
> > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > QEMU version: 2.11
> > > > > libvirt version: 3.2.0
> > > > > Running the following on the host shows that both nested and IOMMU are
> > > > > enabled:
> > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > Y
> > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > [    0.000000] DMAR: IOMMU enabled
> > > > > 
> > > > > Thanks,
> > > > > Elijah
> > > > > 
> > > 
> > > 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Marcel Apfelbaum]

On 07/26/2018 07:42 PM, Knut Omang wrote:
> On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
>> On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
>>> On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
>>>> Hi
>>>>
>>>> On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
>>>>> On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
>>>>>> Hey,
>>>>>>
>>>>>> Our team is adding a NIC functional  emulation to QEMU.
>>>>>> One of the features we are adding to this NIC is SRIOV.
>>>>>>
>>>>>> Here is the error message I get when checking SRIOV support of  our
>>>>>> emulated NIC on Win2016 server (the hyper-v VM).
>>>>>> "
>>>>>> SR-IOV cannot be used on this system as the PCI Express hardware does
>>>>>> not
>>>>>> support Access Control Services (ACS) at any root port.
>>>> QEMU's emulated PCI Express Root Ports do not support ACS yet, however I
>>>> am not sure ACS is a prerequisite
>>>> for SR-IOV. We would need ARI support for allowing more than 8VFs, but
>>>> QEMU doesn't support that either (yet).
>>>>
>>>> Knut Omag has some working patches, he successfully implemented SR-IOV
>>>> with QEMU, see:
>>>>        https://github.com/knuto/qemu/tree/sriov_patches_v7
>>>>
>>>> The code was not merged since we need at least a device with SR-IOV
>>>> support to justify the addition.
>>> FYI, I recently rebased these to latest master but just didn't get to push
>>> them
>>> out. I did just now - they are available here:
>>>
>>>     https://github.com/knuto/qemu/tree/sriov_patches_v8
>>>
>>> As far as I know ARI support with my patch set works just fine - I have
>>> tested
>>> it with lots of VFs.
>>>
>>> One of the patches in the series (pci: Make use of the devfn property when
>>> registering new devices) is necessary to make ARI work as it should with
>>> SR/IOV.
>>>
>>> For the hardware model I developed the SR/IOV patches for, I also added
>>> enough
>>> ACS support in the root port (PCIe capability helper patch + usage in
>>> ioh3420) to make VFIO "happy". I haven't submitted them because they
>>> are "questionable" since they likely do not reflect the actual features of
>>> the
>>> ioh3420.
>> In that the actual ioh3420 doesn't support ACS?
> yes.. I don't have one so I don't know but that was my assumption..

Hi Knut,

We have now a generic PCIe Root Port we can add whatever we want to it.
See please hw/pci-bridge/gen_pcie_root_port.c.

So your patches add both ARI and ACS support, nice!
Maybe it worth merging at least these features.

Thanks,
Marcel

>
> Knut
>
>>> I can make those available if interesting.
>>>
>>> Thanks,
>>> Knut
>>>
>>>>>> Contact your system vendor for further information.
>>>>>> "
>>>>> I'm not sure what the status of emulated SR-IOV is so I have CCed
>>>>> Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
>>>> Thanks,
>>>> Marcel
>>>>
>>>>>> Could you please advise about what could be the issue here?
>>>>>>
>>>>>> BTW: I use same configuration (VM XML file attached) when running
>>>>>> linux VM
>>>>>> (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
>>>>>> working for me there.
>>>>>>
>>>>>> Here the XML file I use to define the VM (our emulated NIC is added at
>>>>>> the
>>>>>> end of XML):
>>>>>> "
>>>>>> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/
>>>>>> 1.0'
>>>>>>> ;
>>>>>>     <name>nst105</name>
>>>>>>     <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
>>>>>>     <memory unit='KiB'>8388608</memory>
>>>>>>     <currentMemory unit='KiB'>8388608</currentMemory>
>>>>>>     <vcpu placement='static'>8</vcpu>
>>>>>>     <resource>
>>>>>>       <partition>/machine</partition>
>>>>>>     </resource>
>>>>>>     <os>
>>>>>>       <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
>>>>>>     </os>
>>>>>>     <features>
>>>>>>       <acpi/>
>>>>>>       <apic/>
>>>>>>       <hyperv>
>>>>>>         <relaxed state='on'/>
>>>>>>         <vapic state='on'/>
>>>>>>         <spinlocks state='on' retries='8191'/>
>>>>>>       </hyperv>
>>>>>>     </features>
>>>>>>     <cpu mode='custom' match='exact' check='none'>
>>>>>>       <model fallback='forbid'>SandyBridge</model>
>>>>>>       <feature policy='require' name='hypervisor'/>
>>>>>>       <feature policy='require' name='vmx'/>
>>>>>>     </cpu>
>>>>>>     <clock offset='localtime'>
>>>>>>       <timer name='rtc' tickpolicy='catchup'/>
>>>>>>       <timer name='pit' tickpolicy='delay'/>
>>>>>>       <timer name='hpet' present='no'/>
>>>>>>       <timer name='hypervclock' present='yes'/>
>>>>>>     </clock>
>>>>>>     <on_poweroff>destroy</on_poweroff>
>>>>>>     <on_reboot>restart</on_reboot>
>>>>>>     <on_crash>destroy</on_crash>
>>>>>>     <pm>
>>>>>>       <suspend-to-mem enabled='no'/>
>>>>>>       <suspend-to-disk enabled='no'/>
>>>>>>     </pm>
>>>>>>     <devices>
>>>>>>       <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
>>>>>>       <disk type='file' device='disk'>
>>>>>>         <driver name='qemu' type='qcow2'/>
>>>>>>         <source file='/images/nst105.qcow2'/>
>>>>>>         <target dev='sda' bus='sata'/>
>>>>>>         <boot order='2'/>
>>>>>>         <address type='drive' controller='0' bus='0' target='0'
>>>>>> unit='0'/>
>>>>>>       </disk>
>>>>>>       <controller type='usb' index='0' model='ich9-ehci1'>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x7'/>
>>>>>>       </controller>
>>>>>>       <controller type='usb' index='0' model='ich9-uhci1'>
>>>>>>         <master startport='0'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x0' multifunction='on'/>
>>>>>>       </controller>
>>>>>>       <controller type='usb' index='0' model='ich9-uhci2'>
>>>>>>         <master startport='2'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x1'/>
>>>>>>       </controller>
>>>>>>       <controller type='usb' index='0' model='ich9-uhci3'>
>>>>>>         <master startport='4'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>>>>>> function='0x2'/>
>>>>>>       </controller>
>>>>>>       <controller type='sata' index='0'>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
>>>>>> function='0x2'/>
>>>>>>       </controller>
>>>>>>       <controller type='pci' index='0' model='pcie-root'/>
>>>>>>       <controller type='pci' index='1' model='pcie-root-port'>
>>>>>>         <model name='pcie-root-port'/>
>>>>>>         <target chassis='1' port='0x10'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>>>>> function='0x0' multifunction='on'/>
>>>>>>       </controller>
>>>>>>       <controller type='pci' index='2' model='pcie-root-port'>
>>>>>>         <model name='pcie-root-port'/>
>>>>>>         <target chassis='2' port='0x11'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>>>>> function='0x1'/>
>>>>>>       </controller>
>>>>>>       <controller type='pci' index='3' model='pcie-root-port'>
>>>>>>         <model name='pcie-root-port'/>
>>>>>>         <target chassis='3' port='0x12'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>>>>> function='0x2'/>
>>>>>>       </controller>
>>>>>>       <controller type='pci' index='4' model='dmi-to-pci-bridge'>
>>>>>>         <model name='i82801b11-bridge'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
>>>>>> function='0x0'/>
>>>>>>       </controller>
>>>>>>       <controller type='pci' index='5' model='pci-bridge'>
>>>>>>         <model name='pci-bridge'/>
>>>>>>         <target chassisNr='5'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
>>>>>> function='0x0'/>
>>>>>>       </controller>
>>>>>>       <interface type='bridge'>
>>>>>>         <mac address='00:50:56:1b:dd:6f'/>
>>>>>>         <source bridge='br0'/>
>>>>>>         <model type='virtio'/>
>>>>>>         <boot order='1'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
>>>>>> function='0x0'/>
>>>>>>       </interface>
>>>>>>       <serial type='pty'>
>>>>>>         <target port='0'/>
>>>>>>       </serial>
>>>>>>       <console type='pty'>
>>>>>>         <target type='serial' port='0'/>
>>>>>>       </console>
>>>>>>       <input type='tablet' bus='usb'>
>>>>>>         <address type='usb' bus='0' port='1'/>
>>>>>>       </input>
>>>>>>       <input type='mouse' bus='ps2'/>
>>>>>>       <input type='keyboard' bus='ps2'/>
>>>>>>       <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
>>>>>> keymap='en-us'>
>>>>>>         <listen type='address' address='127.0.0.1'/>
>>>>>>       </graphics>
>>>>>>       <video>
>>>>>>         <model type='vga' vram='16384' heads='1' primary='yes'/>
>>>>>>         <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>>>>>> function='0x0'/>
>>>>>>       </video>
>>>>>>       <memballoon model='virtio'>
>>>>>>         <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
>>>>>> function='0x0'/>
>>>>>>       </memballoon>
>>>>>>     </devices>
>>>>>>     <seclabel type='none' model='none'/>
>>>>>>     <seclabel type='dynamic' model='dac' relabel='yes'/>
>>>>>>     <qemu:commandline>
>>>>>>       <qemu:arg value='-device'/>
>>>>>>       <qemu:arg value='pcie-root-port,pref64-
>>>>>> reserve=500M,slot=0,id=pcie_port.1'/>
>>>>>>       <qemu:arg value='-netdev'/>
>>>>>>       <qemu:arg
>>>>>> value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
>>>>>>       <qemu:arg value='-device'/>
>>>>>>       <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
>>>>>>     </qemu:commandline>
>>>>>> </domain>
>>>>>> "
>>>>>> ______________________
>>>>>> General info:
>>>>>> Host OS: RH7.0 (Kernel: 4.14.13)
>>>>>> QEMU version: 2.11
>>>>>> libvirt version: 3.2.0
>>>>>> Running the following on the host shows that both nested and IOMMU are
>>>>>> enabled:
>>>>>> ~]#: cat /sys/module/kvm_intel/parameters/nested
>>>>>> Y
>>>>>> ~]# dmesg | grep -e DMAR -e IOMMU
>>>>>> [    0.000000] DMAR: IOMMU enabled
>>>>>>
>>>>>> Thanks,
>>>>>> Elijah
>>>>>>
>>>>

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Knut Omang]

On Thu, 2018-07-26 at 20:05 +0300, Marcel Apfelbaum wrote:
> 
> On 07/26/2018 07:42 PM, Knut Omang wrote:
> > On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > > > Hi
> > > > > 
> > > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > > Hey,
> > > > > > > 
> > > > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > > 
> > > > > > > Here is the error message I get when checking SRIOV support
> > > > > > > of  our
> > > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > > "
> > > > > > > SR-IOV cannot be used on this system as the PCI Express hardware
> > > > > > > does
> > > > > > > not
> > > > > > > support Access Control Services (ACS) at any root port.
> > > > > 
> > > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however
> > > > > I
> > > > > am not sure ACS is a prerequisite
> > > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > > > > QEMU doesn't support that either (yet).
> > > > > 
> > > > > Knut Omag has some working patches, he successfully implemented SR-IOV
> > > > > with QEMU, see:
> > > > >        https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > > > 
> > > > > The code was not merged since we need at least a device with SR-IOV
> > > > > support to justify the addition.
> > > > 
> > > > FYI, I recently rebased these to latest master but just didn't get to
> > > > push
> > > > them
> > > > out. I did just now - they are available here:
> > > > 
> > > >     https://github.com/knuto/qemu/tree/sriov_patches_v8
> > > > 
> > > > As far as I know ARI support with my patch set works just fine - I have
> > > > tested
> > > > it with lots of VFs.
> > > > 
> > > > One of the patches in the series (pci: Make use of the devfn property
> > > > when
> > > > registering new devices) is necessary to make ARI work as it should with
> > > > SR/IOV.
> > > > 
> > > > For the hardware model I developed the SR/IOV patches for, I also added
> > > > enough
> > > > ACS support in the root port (PCIe capability helper patch + usage in
> > > > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > > > are "questionable" since they likely do not reflect the actual features
> > > > of
> > > > the
> > > > ioh3420.
> > > 
> > > In that the actual ioh3420 doesn't support ACS?
> > 
> > yes.. I don't have one so I don't know but that was my assumption..
> 
> Hi Knut,
> 
> We have now a generic PCIe Root Port we can add whatever we want to it.
> See please hw/pci-bridge/gen_pcie_root_port.c.

Ok, I see! - I wasn't aware of that. I was thinking more in terms of just 
emulating a slightly newer Intel root port but never got around to that.

> So your patches add both ARI and ACS support, nice!
> Maybe it worth merging at least these features.

I'll have a look at the generic root port, it is probably a natural extension 
for v9 of the sriov patch set :-)

Thanks,
Knut

> 
> Thanks,
> Marcel
> 
> > 
> > Knut
> > 
> > > > I can make those available if interesting.
> > > > 
> > > > Thanks,
> > > > Knut
> > > > 
> > > > > > > Contact your system vendor for further information.
> > > > > > > "
> > > > > > 
> > > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > > > 
> > > > > Thanks,
> > > > > Marcel
> > > > > 
> > > > > > > Could you please advise about what could be the issue here?
> > > > > > > 
> > > > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > > > linux VM
> > > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and
> > > > > > > SRIOV is
> > > > > > > working for me there.
> > > > > > > 
> > > > > > > Here the XML file I use to define the VM (our emulated NIC is
> > > > > > > added at
> > > > > > > the
> > > > > > > end of XML):
> > > > > > > "
> > > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/q
> > > > > > > emu/
> > > > > > > 1.0'
> > > > > > > > ;
> > > > > > > 
> > > > > > >     <name>nst105</name>
> > > > > > >     <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > > >     <memory unit='KiB'>8388608</memory>
> > > > > > >     <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > > >     <vcpu placement='static'>8</vcpu>
> > > > > > >     <resource>
> > > > > > >       <partition>/machine</partition>
> > > > > > >     </resource>
> > > > > > >     <os>
> > > > > > >       <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > > >     </os>
> > > > > > >     <features>
> > > > > > >       <acpi/>
> > > > > > >       <apic/>
> > > > > > >       <hyperv>
> > > > > > >         <relaxed state='on'/>
> > > > > > >         <vapic state='on'/>
> > > > > > >         <spinlocks state='on' retries='8191'/>
> > > > > > >       </hyperv>
> > > > > > >     </features>
> > > > > > >     <cpu mode='custom' match='exact' check='none'>
> > > > > > >       <model fallback='forbid'>SandyBridge</model>
> > > > > > >       <feature policy='require' name='hypervisor'/>
> > > > > > >       <feature policy='require' name='vmx'/>
> > > > > > >     </cpu>
> > > > > > >     <clock offset='localtime'>
> > > > > > >       <timer name='rtc' tickpolicy='catchup'/>
> > > > > > >       <timer name='pit' tickpolicy='delay'/>
> > > > > > >       <timer name='hpet' present='no'/>
> > > > > > >       <timer name='hypervclock' present='yes'/>
> > > > > > >     </clock>
> > > > > > >     <on_poweroff>destroy</on_poweroff>
> > > > > > >     <on_reboot>restart</on_reboot>
> > > > > > >     <on_crash>destroy</on_crash>
> > > > > > >     <pm>
> > > > > > >       <suspend-to-mem enabled='no'/>
> > > > > > >       <suspend-to-disk enabled='no'/>
> > > > > > >     </pm>
> > > > > > >     <devices>
> > > > > > >       <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > > >       <disk type='file' device='disk'>
> > > > > > >         <driver name='qemu' type='qcow2'/>
> > > > > > >         <source file='/images/nst105.qcow2'/>
> > > > > > >         <target dev='sda' bus='sata'/>
> > > > > > >         <boot order='2'/>
> > > > > > >         <address type='drive' controller='0' bus='0' target='0'
> > > > > > > unit='0'/>
> > > > > > >       </disk>
> > > > > > >       <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x7'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > > >         <master startport='0'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > > >         <master startport='2'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x1'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > > >         <master startport='4'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x2'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='sata' index='0'>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > > > function='0x2'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='0' model='pcie-root'/>
> > > > > > >       <controller type='pci' index='1' model='pcie-root-port'>
> > > > > > >         <model name='pcie-root-port'/>
> > > > > > >         <target chassis='1' port='0x10'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='2' model='pcie-root-port'>
> > > > > > >         <model name='pcie-root-port'/>
> > > > > > >         <target chassis='2' port='0x11'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x1'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='3' model='pcie-root-port'>
> > > > > > >         <model name='pcie-root-port'/>
> > > > > > >         <target chassis='3' port='0x12'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x2'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > > >         <model name='i82801b11-bridge'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > > > function='0x0'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='5' model='pci-bridge'>
> > > > > > >         <model name='pci-bridge'/>
> > > > > > >         <target chassisNr='5'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > >       </controller>
> > > > > > >       <interface type='bridge'>
> > > > > > >         <mac address='00:50:56:1b:dd:6f'/>
> > > > > > >         <source bridge='br0'/>
> > > > > > >         <model type='virtio'/>
> > > > > > >         <boot order='1'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > >       </interface>
> > > > > > >       <serial type='pty'>
> > > > > > >         <target port='0'/>
> > > > > > >       </serial>
> > > > > > >       <console type='pty'>
> > > > > > >         <target type='serial' port='0'/>
> > > > > > >       </console>
> > > > > > >       <input type='tablet' bus='usb'>
> > > > > > >         <address type='usb' bus='0' port='1'/>
> > > > > > >       </input>
> > > > > > >       <input type='mouse' bus='ps2'/>
> > > > > > >       <input type='keyboard' bus='ps2'/>
> > > > > > >       <graphics type='vnc' port='-1' autoport='yes'
> > > > > > > listen='127.0.0.1'
> > > > > > > keymap='en-us'>
> > > > > > >         <listen type='address' address='127.0.0.1'/>
> > > > > > >       </graphics>
> > > > > > >       <video>
> > > > > > >         <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > > > function='0x0'/>
> > > > > > >       </video>
> > > > > > >       <memballoon model='virtio'>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > >       </memballoon>
> > > > > > >     </devices>
> > > > > > >     <seclabel type='none' model='none'/>
> > > > > > >     <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > > >     <qemu:commandline>
> > > > > > >       <qemu:arg value='-device'/>
> > > > > > >       <qemu:arg value='pcie-root-port,pref64-
> > > > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > > >       <qemu:arg value='-netdev'/>
> > > > > > >       <qemu:arg
> > > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > > >       <qemu:arg value='-device'/>
> > > > > > >       <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > > >     </qemu:commandline>
> > > > > > > </domain>
> > > > > > > "
> > > > > > > ______________________
> > > > > > > General info:
> > > > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > > > QEMU version: 2.11
> > > > > > > libvirt version: 3.2.0
> > > > > > > Running the following on the host shows that both nested and IOMMU
> > > > > > > are
> > > > > > > enabled:
> > > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > > > Y
> > > > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > > > [    0.000000] DMAR: IOMMU enabled
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Elijah
> > > > > > > 
> 
> 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Knut Omang]

On Thu, 2018-07-26 at 20:05 +0300, Marcel Apfelbaum wrote:
> 
> On 07/26/2018 07:42 PM, Knut Omang wrote:
> > On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> > > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > > > Hi
> > > > > 
> > > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > > Hey,
> > > > > > > 
> > > > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > > 
> > > > > > > Here is the error message I get when checking SRIOV support
> > > > > > > of  our
> > > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > > "
> > > > > > > SR-IOV cannot be used on this system as the PCI Express hardware
> > > > > > > does
> > > > > > > not
> > > > > > > support Access Control Services (ACS) at any root port.
> > > > > 
> > > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however
> > > > > I
> > > > > am not sure ACS is a prerequisite
> > > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but
> > > > > QEMU doesn't support that either (yet).
> > > > > 
> > > > > Knut Omag has some working patches, he successfully implemented SR-IOV
> > > > > with QEMU, see:
> > > > >        https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > > > 
> > > > > The code was not merged since we need at least a device with SR-IOV
> > > > > support to justify the addition.
> > > > 
> > > > FYI, I recently rebased these to latest master but just didn't get to
> > > > push
> > > > them
> > > > out. I did just now - they are available here:
> > > > 
> > > >     https://github.com/knuto/qemu/tree/sriov_patches_v8
> > > > 
> > > > As far as I know ARI support with my patch set works just fine - I have
> > > > tested
> > > > it with lots of VFs.
> > > > 
> > > > One of the patches in the series (pci: Make use of the devfn property
> > > > when
> > > > registering new devices) is necessary to make ARI work as it should with
> > > > SR/IOV.
> > > > 
> > > > For the hardware model I developed the SR/IOV patches for, I also added
> > > > enough
> > > > ACS support in the root port (PCIe capability helper patch + usage in
> > > > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > > > are "questionable" since they likely do not reflect the actual features
> > > > of
> > > > the
> > > > ioh3420.
> > > 
> > > In that the actual ioh3420 doesn't support ACS?
> > 
> > yes.. I don't have one so I don't know but that was my assumption..
> 
> Hi Knut,
> 
> We have now a generic PCIe Root Port we can add whatever we want to it.
> See please hw/pci-bridge/gen_pcie_root_port.c.

Ok, I see! - I wasn't aware of that. I was thinking more in terms of just 
emulating a slightly newer Intel root port but never got around to that.

> So your patches add both ARI and ACS support, nice!
> Maybe it worth merging at least these features.

I'll have a look at the generic root port, it is probably a natural extension 
for v9 of the sriov patch set :-)

Thanks,
Knut

> 
> Thanks,
> Marcel
> 
> > 
> > Knut
> > 
> > > > I can make those available if interesting.
> > > > 
> > > > Thanks,
> > > > Knut
> > > > 
> > > > > > > Contact your system vendor for further information.
> > > > > > > "
> > > > > > 
> > > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > > > 
> > > > > Thanks,
> > > > > Marcel
> > > > > 
> > > > > > > Could you please advise about what could be the issue here?
> > > > > > > 
> > > > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > > > linux VM
> > > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and
> > > > > > > SRIOV is
> > > > > > > working for me there.
> > > > > > > 
> > > > > > > Here the XML file I use to define the VM (our emulated NIC is
> > > > > > > added at
> > > > > > > the
> > > > > > > end of XML):
> > > > > > > "
> > > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/q
> > > > > > > emu/
> > > > > > > 1.0'
> > > > > > > > ;
> > > > > > > 
> > > > > > >     <name>nst105</name>
> > > > > > >     <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > > >     <memory unit='KiB'>8388608</memory>
> > > > > > >     <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > > >     <vcpu placement='static'>8</vcpu>
> > > > > > >     <resource>
> > > > > > >       <partition>/machine</partition>
> > > > > > >     </resource>
> > > > > > >     <os>
> > > > > > >       <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > > >     </os>
> > > > > > >     <features>
> > > > > > >       <acpi/>
> > > > > > >       <apic/>
> > > > > > >       <hyperv>
> > > > > > >         <relaxed state='on'/>
> > > > > > >         <vapic state='on'/>
> > > > > > >         <spinlocks state='on' retries='8191'/>
> > > > > > >       </hyperv>
> > > > > > >     </features>
> > > > > > >     <cpu mode='custom' match='exact' check='none'>
> > > > > > >       <model fallback='forbid'>SandyBridge</model>
> > > > > > >       <feature policy='require' name='hypervisor'/>
> > > > > > >       <feature policy='require' name='vmx'/>
> > > > > > >     </cpu>
> > > > > > >     <clock offset='localtime'>
> > > > > > >       <timer name='rtc' tickpolicy='catchup'/>
> > > > > > >       <timer name='pit' tickpolicy='delay'/>
> > > > > > >       <timer name='hpet' present='no'/>
> > > > > > >       <timer name='hypervclock' present='yes'/>
> > > > > > >     </clock>
> > > > > > >     <on_poweroff>destroy</on_poweroff>
> > > > > > >     <on_reboot>restart</on_reboot>
> > > > > > >     <on_crash>destroy</on_crash>
> > > > > > >     <pm>
> > > > > > >       <suspend-to-mem enabled='no'/>
> > > > > > >       <suspend-to-disk enabled='no'/>
> > > > > > >     </pm>
> > > > > > >     <devices>
> > > > > > >       <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > > >       <disk type='file' device='disk'>
> > > > > > >         <driver name='qemu' type='qcow2'/>
> > > > > > >         <source file='/images/nst105.qcow2'/>
> > > > > > >         <target dev='sda' bus='sata'/>
> > > > > > >         <boot order='2'/>
> > > > > > >         <address type='drive' controller='0' bus='0' target='0'
> > > > > > > unit='0'/>
> > > > > > >       </disk>
> > > > > > >       <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x7'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > > >         <master startport='0'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > > >         <master startport='2'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x1'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > > >         <master startport='4'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > > function='0x2'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='sata' index='0'>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > > > function='0x2'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='0' model='pcie-root'/>
> > > > > > >       <controller type='pci' index='1' model='pcie-root-port'>
> > > > > > >         <model name='pcie-root-port'/>
> > > > > > >         <target chassis='1' port='0x10'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x0' multifunction='on'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='2' model='pcie-root-port'>
> > > > > > >         <model name='pcie-root-port'/>
> > > > > > >         <target chassis='2' port='0x11'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x1'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='3' model='pcie-root-port'>
> > > > > > >         <model name='pcie-root-port'/>
> > > > > > >         <target chassis='3' port='0x12'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > > function='0x2'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > > >         <model name='i82801b11-bridge'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > > > function='0x0'/>
> > > > > > >       </controller>
> > > > > > >       <controller type='pci' index='5' model='pci-bridge'>
> > > > > > >         <model name='pci-bridge'/>
> > > > > > >         <target chassisNr='5'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > >       </controller>
> > > > > > >       <interface type='bridge'>
> > > > > > >         <mac address='00:50:56:1b:dd:6f'/>
> > > > > > >         <source bridge='br0'/>
> > > > > > >         <model type='virtio'/>
> > > > > > >         <boot order='1'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > >       </interface>
> > > > > > >       <serial type='pty'>
> > > > > > >         <target port='0'/>
> > > > > > >       </serial>
> > > > > > >       <console type='pty'>
> > > > > > >         <target type='serial' port='0'/>
> > > > > > >       </console>
> > > > > > >       <input type='tablet' bus='usb'>
> > > > > > >         <address type='usb' bus='0' port='1'/>
> > > > > > >       </input>
> > > > > > >       <input type='mouse' bus='ps2'/>
> > > > > > >       <input type='keyboard' bus='ps2'/>
> > > > > > >       <graphics type='vnc' port='-1' autoport='yes'
> > > > > > > listen='127.0.0.1'
> > > > > > > keymap='en-us'>
> > > > > > >         <listen type='address' address='127.0.0.1'/>
> > > > > > >       </graphics>
> > > > > > >       <video>
> > > > > > >         <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > > > function='0x0'/>
> > > > > > >       </video>
> > > > > > >       <memballoon model='virtio'>
> > > > > > >         <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > > > function='0x0'/>
> > > > > > >       </memballoon>
> > > > > > >     </devices>
> > > > > > >     <seclabel type='none' model='none'/>
> > > > > > >     <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > > >     <qemu:commandline>
> > > > > > >       <qemu:arg value='-device'/>
> > > > > > >       <qemu:arg value='pcie-root-port,pref64-
> > > > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > > >       <qemu:arg value='-netdev'/>
> > > > > > >       <qemu:arg
> > > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > > >       <qemu:arg value='-device'/>
> > > > > > >       <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > > >     </qemu:commandline>
> > > > > > > </domain>
> > > > > > > "
> > > > > > > ______________________
> > > > > > > General info:
> > > > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > > > QEMU version: 2.11
> > > > > > > libvirt version: 3.2.0
> > > > > > > Running the following on the host shows that both nested and IOMMU
> > > > > > > are
> > > > > > > enabled:
> > > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > > > Y
> > > > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > > > [    0.000000] DMAR: IOMMU enabled
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Elijah
> > > > > > > 
> 
> 

Re: [Qemu-devel] Question: SRIOV support over Win Hyper-V VM running in QEMU process on Linux host

[Michael S. Tsirkin]

On Thu, Jul 26, 2018 at 06:42:51PM +0200, Knut Omang wrote:
> On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote:
> > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote:
> > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote:
> > > > Hi
> > > > 
> > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote:
> > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote:
> > > > > > Hey,
> > > > > > 
> > > > > > Our team is adding a NIC functional  emulation to QEMU.
> > > > > > One of the features we are adding to this NIC is SRIOV.
> > > > > > 
> > > > > > Here is the error message I get when checking SRIOV support of  our
> > > > > > emulated NIC on Win2016 server (the hyper-v VM).
> > > > > > "
> > > > > > SR-IOV cannot be used on this system as the PCI Express hardware does
> > > > > > not
> > > > > > support Access Control Services (ACS) at any root port.
> > > > 
> > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however I 
> > > > am not sure ACS is a prerequisite
> > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but 
> > > > QEMU doesn't support that either (yet).
> > > > 
> > > > Knut Omag has some working patches, he successfully implemented SR-IOV 
> > > > with QEMU, see:
> > > >       https://github.com/knuto/qemu/tree/sriov_patches_v7
> > > > 
> > > > The code was not merged since we need at least a device with SR-IOV 
> > > > support to justify the addition.
> > > 
> > > FYI, I recently rebased these to latest master but just didn't get to push
> > > them
> > > out. I did just now - they are available here:
> > > 
> > >    https://github.com/knuto/qemu/tree/sriov_patches_v8
> > > 
> > > As far as I know ARI support with my patch set works just fine - I have
> > > tested
> > > it with lots of VFs.
> > > 
> > > One of the patches in the series (pci: Make use of the devfn property when
> > > registering new devices) is necessary to make ARI work as it should with
> > > SR/IOV.
> > > 
> > > For the hardware model I developed the SR/IOV patches for, I also added
> > > enough
> > > ACS support in the root port (PCIe capability helper patch + usage in
> > > ioh3420) to make VFIO "happy". I haven't submitted them because they
> > > are "questionable" since they likely do not reflect the actual features of
> > > the
> > > ioh3420.
> > 
> > In that the actual ioh3420 doesn't support ACS?
> 
> yes.. I don't have one so I don't know but that was my assumption..
> 
> Knut

As Marcel said, just add it to the generic port, we can
upstream that.

> > > I can make those available if interesting.
> > > 
> > > Thanks,
> > > Knut
> > > 
> > > > > > Contact your system vendor for further information.
> > > > > > "
> > > > > 
> > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed
> > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU.
> > > > 
> > > > Thanks,
> > > > Marcel
> > > > 
> > > > > > Could you please advise about what could be the issue here?
> > > > > > 
> > > > > > BTW: I use same configuration (VM XML file attached) when running
> > > > > > linux VM
> > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and SRIOV is
> > > > > > working for me there.
> > > > > > 
> > > > > > Here the XML file I use to define the VM (our emulated NIC is added at
> > > > > > the
> > > > > > end of XML):
> > > > > > "
> > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/
> > > > > > 1.0'
> > > > > > > ;
> > > > > > 
> > > > > >    <name>nst105</name>
> > > > > >    <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid>
> > > > > >    <memory unit='KiB'>8388608</memory>
> > > > > >    <currentMemory unit='KiB'>8388608</currentMemory>
> > > > > >    <vcpu placement='static'>8</vcpu>
> > > > > >    <resource>
> > > > > >      <partition>/machine</partition>
> > > > > >    </resource>
> > > > > >    <os>
> > > > > >      <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
> > > > > >    </os>
> > > > > >    <features>
> > > > > >      <acpi/>
> > > > > >      <apic/>
> > > > > >      <hyperv>
> > > > > >        <relaxed state='on'/>
> > > > > >        <vapic state='on'/>
> > > > > >        <spinlocks state='on' retries='8191'/>
> > > > > >      </hyperv>
> > > > > >    </features>
> > > > > >    <cpu mode='custom' match='exact' check='none'>
> > > > > >      <model fallback='forbid'>SandyBridge</model>
> > > > > >      <feature policy='require' name='hypervisor'/>
> > > > > >      <feature policy='require' name='vmx'/>
> > > > > >    </cpu>
> > > > > >    <clock offset='localtime'>
> > > > > >      <timer name='rtc' tickpolicy='catchup'/>
> > > > > >      <timer name='pit' tickpolicy='delay'/>
> > > > > >      <timer name='hpet' present='no'/>
> > > > > >      <timer name='hypervclock' present='yes'/>
> > > > > >    </clock>
> > > > > >    <on_poweroff>destroy</on_poweroff>
> > > > > >    <on_reboot>restart</on_reboot>
> > > > > >    <on_crash>destroy</on_crash>
> > > > > >    <pm>
> > > > > >      <suspend-to-mem enabled='no'/>
> > > > > >      <suspend-to-disk enabled='no'/>
> > > > > >    </pm>
> > > > > >    <devices>
> > > > > >      <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator>
> > > > > >      <disk type='file' device='disk'>
> > > > > >        <driver name='qemu' type='qcow2'/>
> > > > > >        <source file='/images/nst105.qcow2'/>
> > > > > >        <target dev='sda' bus='sata'/>
> > > > > >        <boot order='2'/>
> > > > > >        <address type='drive' controller='0' bus='0' target='0'
> > > > > > unit='0'/>
> > > > > >      </disk>
> > > > > >      <controller type='usb' index='0' model='ich9-ehci1'>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x7'/>
> > > > > >      </controller>
> > > > > >      <controller type='usb' index='0' model='ich9-uhci1'>
> > > > > >        <master startport='0'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x0' multifunction='on'/>
> > > > > >      </controller>
> > > > > >      <controller type='usb' index='0' model='ich9-uhci2'>
> > > > > >        <master startport='2'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x1'/>
> > > > > >      </controller>
> > > > > >      <controller type='usb' index='0' model='ich9-uhci3'>
> > > > > >        <master startport='4'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> > > > > > function='0x2'/>
> > > > > >      </controller>
> > > > > >      <controller type='sata' index='0'>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> > > > > > function='0x2'/>
> > > > > >      </controller>
> > > > > >      <controller type='pci' index='0' model='pcie-root'/>
> > > > > >      <controller type='pci' index='1' model='pcie-root-port'>
> > > > > >        <model name='pcie-root-port'/>
> > > > > >        <target chassis='1' port='0x10'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > function='0x0' multifunction='on'/>
> > > > > >      </controller>
> > > > > >      <controller type='pci' index='2' model='pcie-root-port'>
> > > > > >        <model name='pcie-root-port'/>
> > > > > >        <target chassis='2' port='0x11'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > function='0x1'/>
> > > > > >      </controller>
> > > > > >      <controller type='pci' index='3' model='pcie-root-port'>
> > > > > >        <model name='pcie-root-port'/>
> > > > > >        <target chassis='3' port='0x12'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > > > > > function='0x2'/>
> > > > > >      </controller>
> > > > > >      <controller type='pci' index='4' model='dmi-to-pci-bridge'>
> > > > > >        <model name='i82801b11-bridge'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> > > > > > function='0x0'/>
> > > > > >      </controller>
> > > > > >      <controller type='pci' index='5' model='pci-bridge'>
> > > > > >        <model name='pci-bridge'/>
> > > > > >        <target chassisNr='5'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > > > > > function='0x0'/>
> > > > > >      </controller>
> > > > > >      <interface type='bridge'>
> > > > > >        <mac address='00:50:56:1b:dd:6f'/>
> > > > > >        <source bridge='br0'/>
> > > > > >        <model type='virtio'/>
> > > > > >        <boot order='1'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> > > > > > function='0x0'/>
> > > > > >      </interface>
> > > > > >      <serial type='pty'>
> > > > > >        <target port='0'/>
> > > > > >      </serial>
> > > > > >      <console type='pty'>
> > > > > >        <target type='serial' port='0'/>
> > > > > >      </console>
> > > > > >      <input type='tablet' bus='usb'>
> > > > > >        <address type='usb' bus='0' port='1'/>
> > > > > >      </input>
> > > > > >      <input type='mouse' bus='ps2'/>
> > > > > >      <input type='keyboard' bus='ps2'/>
> > > > > >      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'
> > > > > > keymap='en-us'>
> > > > > >        <listen type='address' address='127.0.0.1'/>
> > > > > >      </graphics>
> > > > > >      <video>
> > > > > >        <model type='vga' vram='16384' heads='1' primary='yes'/>
> > > > > >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > > > > > function='0x0'/>
> > > > > >      </video>
> > > > > >      <memballoon model='virtio'>
> > > > > >        <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> > > > > > function='0x0'/>
> > > > > >      </memballoon>
> > > > > >    </devices>
> > > > > >    <seclabel type='none' model='none'/>
> > > > > >    <seclabel type='dynamic' model='dac' relabel='yes'/>
> > > > > >    <qemu:commandline>
> > > > > >      <qemu:arg value='-device'/>
> > > > > >      <qemu:arg value='pcie-root-port,pref64-
> > > > > > reserve=500M,slot=0,id=pcie_port.1'/>
> > > > > >      <qemu:arg value='-netdev'/>
> > > > > >      <qemu:arg
> > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/>
> > > > > >      <qemu:arg value='-device'/>
> > > > > >      <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/>
> > > > > >    </qemu:commandline>
> > > > > > </domain>
> > > > > > "
> > > > > > ______________________
> > > > > > General info:
> > > > > > Host OS: RH7.0 (Kernel: 4.14.13)
> > > > > > QEMU version: 2.11
> > > > > > libvirt version: 3.2.0
> > > > > > Running the following on the host shows that both nested and IOMMU are
> > > > > > enabled:
> > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested
> > > > > > Y
> > > > > > ~]# dmesg | grep -e DMAR -e IOMMU
> > > > > > [    0.000000] DMAR: IOMMU enabled
> > > > > > 
> > > > > > Thanks,
> > > > > > Elijah
> > > > > > 
> > > > 
> > > >