poor performance of passed-through VGA

poor performance of passed-through VGA

[rndbit]

Hello,
I am one of those early tinkerers with VGA passthrough. This is very exciting feature
and as soon as i saw it i thought to myself that i want that "95% speed of bare metal"
as various sources claimed. So i invested into some harware only to find out that
actual 3d performance is far lower. I am not sure why it is the case so maybe someone
could either clarify if it is known defect or something to be expected. Or if this is
actually a bug and someone someone cared enough to tackle it i would gladly be a
test-monkey.

Hardware:
Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
CPU: AMD FX(tm)-8350
GPU1 (host): GeForce GTX 550 Ti
GPU2 (guest): Radeon R9 270X
QEMU 1.7.90 (from git)

qemu command line:
> /usr/local/bin/qemu-system-x86_64 \
> -enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \
> -smp 8,sockets=1,cores=8,threads=1 \
> -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \
> -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \
> -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \
> -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \
> -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \
> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
> -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
> -device vfio-pci,host=00:13.0,bus=pcie.0 \
> -device vfio-pci,host=00:13.2,bus=pcie.0 \
> -bios /home/novist/opt/src/seabios/out/bios.bin \
> -vga none

Kernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont).
Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1

> 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
> 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X]
> 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
Actual problem is that GPU seems to be underused. For example in passmark benchmark i
score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is
quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.

Another rather naive test i performed was checking gpu load with process hacker
while playing actual recent game (arma3 in this case). GPU utilization sits at about
20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat
playable. On bare metal though very high settings run smooth. Its worth noting that
directx10/11 bencmarks of passmark perform much better than directx9 ones.

Here are benchmark results and GPU load graphs:
GPU load: http://imgur.com/M70CnIV
Benchmark: http://imgur.com/IeYh4Zc

As you see while directx11 performs at about acceptable rate (considering others preach
glorious 95% of bare metal performance) directx10 test is quite slower and directx9
performs terribly.

Not sure what other information i could provide. If there is any - i would be happy to
look it up. So any idea where this mythical "95% of bare metal performance" can be found?
Cant wait for some insight from a professional.

P.S. i am also adding 2d benchmark results if anyone cares.
GPU load: http://imgur.com/28x5DN8
Benchmark: http://imgur.com/562PJYl

Re: poor performance of passed-through VGA

[Alex Williamson]

On Tue, 2014-04-08 at 17:01 +0300, rndbit wrote:
> Hello,
> I am one of those early tinkerers with VGA passthrough. This is very exciting feature
> and as soon as i saw it i thought to myself that i want that "95% speed of bare metal"
> as various sources claimed. So i invested into some harware only to find out that
> actual 3d performance is far lower. I am not sure why it is the case so maybe someone
> could either clarify if it is known defect or something to be expected. Or if this is
> actually a bug and someone someone cared enough to tackle it i would gladly be a
> test-monkey.
> 
> Hardware:
> Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
> CPU: AMD FX(tm)-8350
> GPU1 (host): GeForce GTX 550 Ti
> GPU2 (guest): Radeon R9 270X
> QEMU 1.7.90 (from git)
> 
> qemu command line:
> > /usr/local/bin/qemu-system-x86_64 \
> > -enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \
> > -smp 8,sockets=1,cores=8,threads=1 \
> > -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \
> > -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \
> > -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \
> > -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \
> > -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \
> > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
> > -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
> > -device vfio-pci,host=00:13.0,bus=pcie.0 \
> > -device vfio-pci,host=00:13.2,bus=pcie.0 \
> > -bios /home/novist/opt/src/seabios/out/bios.bin \
> > -vga none
> 
> Kernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont).
> Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
> 
> > 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> > 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
> > 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X]
> > 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> Actual problem is that GPU seems to be underused. For example in passmark benchmark i
> score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is
> quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
> 
> Another rather naive test i performed was checking gpu load with process hacker
> while playing actual recent game (arma3 in this case). GPU utilization sits at about
> 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat
> playable. On bare metal though very high settings run smooth. Its worth noting that
> directx10/11 bencmarks of passmark perform much better than directx9 ones.
> 
> Here are benchmark results and GPU load graphs:
> GPU load: http://imgur.com/M70CnIV
> Benchmark: http://imgur.com/IeYh4Zc
> 
> As you see while directx11 performs at about acceptable rate (considering others preach
> glorious 95% of bare metal performance) directx10 test is quite slower and directx9
> performs terribly.
> 
> Not sure what other information i could provide. If there is any - i would be happy to
> look it up. So any idea where this mythical "95% of bare metal performance" can be found?
> Cant wait for some insight from a professional.
> 
> P.S. i am also adding 2d benchmark results if anyone cares.
> GPU load: http://imgur.com/28x5DN8
> Benchmark: http://imgur.com/562PJYl

Have you tried assigning the Nvidia GPU?  If you doubt that good
performance is possible, see these images of the same tests:

http://imgur.com/a/7nELz

This is with an Nvidia Quadro K4000 assigned as a secondary device,
without VGA quirks on an Intel host.  It's entirely possible that VGA
quirks are getting in the way in your test and may still with the
GTX550Ti using VGA quirks.  I guarantee that if the quirks are hit in
any sort of hot path, performance will plummet.

It's also important to tune the setup, the above tests use hv-time on
the vCPU and a kernel that supports it, as well as large pages for the
guest and vCPU pinning.

There's potentially still work to do, the above tests show some
significant hits in the 2D test, but 3D is quite good.

If you'd like to help improve the situation, you can enable debug when
compiling qemu vfio, uncomment hw/misc/vfio.c:#define DEBUG_VFIO.  This
will produce a lot of output, especially on guest boot, but it should
settle down as the drivers get loaded and direct access is used.  If it
doesn't settle down and it's through the quirks, we may never be able to
achieve good performance since we don't own the driver.  If it does
settle down, you can use tools like perf and trace to analyze where time
is being spent.  Thanks,

Alex

Re: poor performance of passed-through VGA

[rndbit]

On 04/08/2014 08:00 PM, Alex Williamson wrote:
> On Tue, 2014-04-08 at 17:01 +0300, rndbit wrote:
>> Hello,
>> I am one of those early tinkerers with VGA passthrough. This is very exciting feature
>> and as soon as i saw it i thought to myself that i want that "95% speed of bare metal"
>> as various sources claimed. So i invested into some harware only to find out that
>> actual 3d performance is far lower. I am not sure why it is the case so maybe someone
>> could either clarify if it is known defect or something to be expected. Or if this is
>> actually a bug and someone someone cared enough to tackle it i would gladly be a
>> test-monkey.
>>
>> Hardware:
>> Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
>> CPU: AMD FX(tm)-8350
>> GPU1 (host): GeForce GTX 550 Ti
>> GPU2 (guest): Radeon R9 270X
>> QEMU 1.7.90 (from git)
>>
>> qemu command line:
>>> /usr/local/bin/qemu-system-x86_64 \
>>> -enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \
>>> -smp 8,sockets=1,cores=8,threads=1 \
>>> -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \
>>> -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \
>>> -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \
>>> -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \
>>> -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \
>>> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
>>> -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
>>> -device vfio-pci,host=00:13.0,bus=pcie.0 \
>>> -device vfio-pci,host=00:13.2,bus=pcie.0 \
>>> -bios /home/novist/opt/src/seabios/out/bios.bin \
>>> -vga none
>>
>> Kernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont).
>> Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
>>
>>> 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
>>> 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
>>> 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X]
>>> 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
>> Actual problem is that GPU seems to be underused. For example in passmark benchmark i
>> score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is
>> quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
>>
>> Another rather naive test i performed was checking gpu load with process hacker
>> while playing actual recent game (arma3 in this case). GPU utilization sits at about
>> 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat
>> playable. On bare metal though very high settings run smooth. Its worth noting that
>> directx10/11 bencmarks of passmark perform much better than directx9 ones.
>>
>> Here are benchmark results and GPU load graphs:
>> GPU load: http://imgur.com/M70CnIV
>> Benchmark: http://imgur.com/IeYh4Zc
>>
>> As you see while directx11 performs at about acceptable rate (considering others preach
>> glorious 95% of bare metal performance) directx10 test is quite slower and directx9
>> performs terribly.
>>
>> Not sure what other information i could provide. If there is any - i would be happy to
>> look it up. So any idea where this mythical "95% of bare metal performance" can be found?
>> Cant wait for some insight from a professional.
>>
>> P.S. i am also adding 2d benchmark results if anyone cares.
>> GPU load: http://imgur.com/28x5DN8
>> Benchmark: http://imgur.com/562PJYl
> 
> Have you tried assigning the Nvidia GPU?  If you doubt that good
> performance is possible, see these images of the same tests:
> 
> http://imgur.com/a/7nELz
> 
> This is with an Nvidia Quadro K4000 assigned as a secondary device,
> without VGA quirks on an Intel host.  It's entirely possible that VGA
> quirks are getting in the way in your test and may still with the
> GTX550Ti using VGA quirks.  I guarantee that if the quirks are hit in
> any sort of hot path, performance will plummet.
> 
> It's also important to tune the setup, the above tests use hv-time on
> the vCPU and a kernel that supports it, as well as large pages for the
> guest and vCPU pinning.
> 
> There's potentially still work to do, the above tests show some
> significant hits in the 2D test, but 3D is quite good.
> 
> If you'd like to help improve the situation, you can enable debug when
> compiling qemu vfio, uncomment hw/misc/vfio.c:#define DEBUG_VFIO.  This
> will produce a lot of output, especially on guest boot, but it should
> settle down as the drivers get loaded and direct access is used.  If it
> doesn't settle down and it's through the quirks, we may never be able to
> achieve good performance since we don't own the driver.  If it does
> settle down, you can use tools like perf and trace to analyze where time
> is being spent.  Thanks,
> 
> Alex
> 
> 

I have not tried passing-through nvidia gpu as i have read some scary
tales of nvidia gpus being problematic. Besides now it would be a little
bit pointless (unless its only for test) as i got more powerful GPU for
use in VM. I will definitely try figuring out where time is spent. Some
of your mentioned stuff is really all new to me and might take bit of
time to figure out.

Im not 100% sure what vga quirks are so please forgive for possibly
stupid question but are they enabled by x-vga=on? Something at the back
of my head is telling me that you are probably not using that flag since
your card is passed-through as >secondary< device. I also tried to
achieve same thing with no luck. What i tried was basically -vga std and
removing x-vga=on bit. Result was device manager indicating non-working
device. I believe error was 12 (not enough resources) or something very
similar. If you have ideas regarding this bit i would love to hear them.
In the meantime ill snoop around stuff you mentioned, thank you for hints.

Re: poor performance of passed-through VGA

[Alex Williamson]

On Tue, 2014-04-08 at 22:24 +0300, rndbit wrote:
> On 04/08/2014 08:00 PM, Alex Williamson wrote:
> > On Tue, 2014-04-08 at 17:01 +0300, rndbit wrote:
> >> Hello,
> >> I am one of those early tinkerers with VGA passthrough. This is very exciting feature
> >> and as soon as i saw it i thought to myself that i want that "95% speed of bare metal"
> >> as various sources claimed. So i invested into some harware only to find out that
> >> actual 3d performance is far lower. I am not sure why it is the case so maybe someone
> >> could either clarify if it is known defect or something to be expected. Or if this is
> >> actually a bug and someone someone cared enough to tackle it i would gladly be a
> >> test-monkey.
> >>
> >> Hardware:
> >> Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
> >> CPU: AMD FX(tm)-8350
> >> GPU1 (host): GeForce GTX 550 Ti
> >> GPU2 (guest): Radeon R9 270X
> >> QEMU 1.7.90 (from git)
> >>
> >> qemu command line:
> >>> /usr/local/bin/qemu-system-x86_64 \
> >>> -enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \
> >>> -smp 8,sockets=1,cores=8,threads=1 \
> >>> -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \
> >>> -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \
> >>> -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \
> >>> -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \
> >>> -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \
> >>> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
> >>> -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
> >>> -device vfio-pci,host=00:13.0,bus=pcie.0 \
> >>> -device vfio-pci,host=00:13.2,bus=pcie.0 \
> >>> -bios /home/novist/opt/src/seabios/out/bios.bin \
> >>> -vga none
> >>
> >> Kernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont).
> >> Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
> >>
> >>> 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> >>> 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
> >>> 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X]
> >>> 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> >> Actual problem is that GPU seems to be underused. For example in passmark benchmark i
> >> score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is
> >> quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
> >>
> >> Another rather naive test i performed was checking gpu load with process hacker
> >> while playing actual recent game (arma3 in this case). GPU utilization sits at about
> >> 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat
> >> playable. On bare metal though very high settings run smooth. Its worth noting that
> >> directx10/11 bencmarks of passmark perform much better than directx9 ones.
> >>
> >> Here are benchmark results and GPU load graphs:
> >> GPU load: http://imgur.com/M70CnIV
> >> Benchmark: http://imgur.com/IeYh4Zc
> >>
> >> As you see while directx11 performs at about acceptable rate (considering others preach
> >> glorious 95% of bare metal performance) directx10 test is quite slower and directx9
> >> performs terribly.
> >>
> >> Not sure what other information i could provide. If there is any - i would be happy to
> >> look it up. So any idea where this mythical "95% of bare metal performance" can be found?
> >> Cant wait for some insight from a professional.
> >>
> >> P.S. i am also adding 2d benchmark results if anyone cares.
> >> GPU load: http://imgur.com/28x5DN8
> >> Benchmark: http://imgur.com/562PJYl
> > 
> > Have you tried assigning the Nvidia GPU?  If you doubt that good
> > performance is possible, see these images of the same tests:
> > 
> > http://imgur.com/a/7nELz
> > 
> > This is with an Nvidia Quadro K4000 assigned as a secondary device,
> > without VGA quirks on an Intel host.  It's entirely possible that VGA
> > quirks are getting in the way in your test and may still with the
> > GTX550Ti using VGA quirks.  I guarantee that if the quirks are hit in
> > any sort of hot path, performance will plummet.
> > 
> > It's also important to tune the setup, the above tests use hv-time on
> > the vCPU and a kernel that supports it, as well as large pages for the
> > guest and vCPU pinning.
> > 
> > There's potentially still work to do, the above tests show some
> > significant hits in the 2D test, but 3D is quite good.
> > 
> > If you'd like to help improve the situation, you can enable debug when
> > compiling qemu vfio, uncomment hw/misc/vfio.c:#define DEBUG_VFIO.  This
> > will produce a lot of output, especially on guest boot, but it should
> > settle down as the drivers get loaded and direct access is used.  If it
> > doesn't settle down and it's through the quirks, we may never be able to
> > achieve good performance since we don't own the driver.  If it does
> > settle down, you can use tools like perf and trace to analyze where time
> > is being spent.  Thanks,
> > 
> > Alex
> > 
> > 
> 
> I have not tried passing-through nvidia gpu as i have read some scary
> tales of nvidia gpus being problematic. Besides now it would be a little
> bit pointless (unless its only for test) as i got more powerful GPU for
> use in VM. I will definitely try figuring out where time is spent. Some
> of your mentioned stuff is really all new to me and might take bit of
> time to figure out.

IME, Nvidia actually works better than AMD.  The Nvidia quirks mostly
come from the reverse engineering that the nouveau project has done
while the AMD quirks come solely from my reverse engineering.  Nvidia is
also interested in supporting the professional class cards in VMs
whereas I haven't seen any interest in that from AMD.  Note that the
professional class cards are the Quadro, GRID, and Tesla boards and
they're supported in a secondary display configuration.  Using x-vga=on
with consumer Geforce or Radeon boards is equally not supported by
either vendor.

> Im not 100% sure what vga quirks are so please forgive for possibly
> stupid question but are they enabled by x-vga=on? Something at the back
> of my head is telling me that you are probably not using that flag since
> your card is passed-through as >secondary< device. I also tried to
> achieve same thing with no luck. What i tried was basically -vga std and
> removing x-vga=on bit. Result was device manager indicating non-working
> device. I believe error was 12 (not enough resources) or something very
> similar. If you have ideas regarding this bit i would love to hear them.
> In the meantime ill snoop around stuff you mentioned, thank you for hints.

Yes, secondary mode is to provide the guest with a -vga device in
addition to the assigned GPU, and not use the x-vga=on option.  Whether
this works often depends on the driver in the guest.

The quirks that I'm talking about are fixes for hardware backdoors.  For
instance PCI config space is often mirrored in MMIO config space on
graphics cards.  Running in a virtual machine requires that we emulate
portions of config space, so we need to trap accesses to the MMIO
mirror.  If the driver uses the mirror, or anything within PAGE_SIZE of
the mirror, for performance critical access, the traps required for the
quirk will seriously affect the performance.

The Q35 QEMU model could also have something to do with the performance
issues you see.  There have been unsubstantiated claims that VMs are
slower with Q35.  At the same time, it provides a topology that looks a
lot more like a modern system than the default machine.  The numbers I
reported were using the standard 440FX machine type.  This is another
case where the driver in the guest may be more or less accommodating of
what we expose in the VM, so YMMV with consumer cards and different
drivers.  Thanks,

Alex

Re: poor performance of passed-through VGA

[rndbit]

On 04/08/2014 10:47 PM, Alex Williamson wrote:
> On Tue, 2014-04-08 at 22:24 +0300, rndbit wrote:
>> On 04/08/2014 08:00 PM, Alex Williamson wrote:
>>> On Tue, 2014-04-08 at 17:01 +0300, rndbit wrote:
>>>> Hello,
>>>> I am one of those early tinkerers with VGA passthrough. This is very exciting feature
>>>> and as soon as i saw it i thought to myself that i want that "95% speed of bare metal"
>>>> as various sources claimed. So i invested into some harware only to find out that
>>>> actual 3d performance is far lower. I am not sure why it is the case so maybe someone
>>>> could either clarify if it is known defect or something to be expected. Or if this is
>>>> actually a bug and someone someone cared enough to tackle it i would gladly be a
>>>> test-monkey.
>>>>
>>>> Hardware:
>>>> Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
>>>> CPU: AMD FX(tm)-8350
>>>> GPU1 (host): GeForce GTX 550 Ti
>>>> GPU2 (guest): Radeon R9 270X
>>>> QEMU 1.7.90 (from git)
>>>>
>>>> qemu command line:
>>>>> /usr/local/bin/qemu-system-x86_64 \
>>>>> -enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \
>>>>> -smp 8,sockets=1,cores=8,threads=1 \
>>>>> -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \
>>>>> -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \
>>>>> -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \
>>>>> -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \
>>>>> -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \
>>>>> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
>>>>> -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
>>>>> -device vfio-pci,host=00:13.0,bus=pcie.0 \
>>>>> -device vfio-pci,host=00:13.2,bus=pcie.0 \
>>>>> -bios /home/novist/opt/src/seabios/out/bios.bin \
>>>>> -vga none
>>>>
>>>> Kernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont).
>>>> Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
>>>>
>>>>> 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
>>>>> 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
>>>>> 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X]
>>>>> 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
>>>> Actual problem is that GPU seems to be underused. For example in passmark benchmark i
>>>> score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is
>>>> quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
>>>>
>>>> Another rather naive test i performed was checking gpu load with process hacker
>>>> while playing actual recent game (arma3 in this case). GPU utilization sits at about
>>>> 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat
>>>> playable. On bare metal though very high settings run smooth. Its worth noting that
>>>> directx10/11 bencmarks of passmark perform much better than directx9 ones.
>>>>
>>>> Here are benchmark results and GPU load graphs:
>>>> GPU load: http://imgur.com/M70CnIV
>>>> Benchmark: http://imgur.com/IeYh4Zc
>>>>
>>>> As you see while directx11 performs at about acceptable rate (considering others preach
>>>> glorious 95% of bare metal performance) directx10 test is quite slower and directx9
>>>> performs terribly.
>>>>
>>>> Not sure what other information i could provide. If there is any - i would be happy to
>>>> look it up. So any idea where this mythical "95% of bare metal performance" can be found?
>>>> Cant wait for some insight from a professional.
>>>>
>>>> P.S. i am also adding 2d benchmark results if anyone cares.
>>>> GPU load: http://imgur.com/28x5DN8
>>>> Benchmark: http://imgur.com/562PJYl
>>>
>>> Have you tried assigning the Nvidia GPU?  If you doubt that good
>>> performance is possible, see these images of the same tests:
>>>
>>> http://imgur.com/a/7nELz
>>>
>>> This is with an Nvidia Quadro K4000 assigned as a secondary device,
>>> without VGA quirks on an Intel host.  It's entirely possible that VGA
>>> quirks are getting in the way in your test and may still with the
>>> GTX550Ti using VGA quirks.  I guarantee that if the quirks are hit in
>>> any sort of hot path, performance will plummet.
>>>
>>> It's also important to tune the setup, the above tests use hv-time on
>>> the vCPU and a kernel that supports it, as well as large pages for the
>>> guest and vCPU pinning.
>>>
>>> There's potentially still work to do, the above tests show some
>>> significant hits in the 2D test, but 3D is quite good.
>>>
>>> If you'd like to help improve the situation, you can enable debug when
>>> compiling qemu vfio, uncomment hw/misc/vfio.c:#define DEBUG_VFIO.  This
>>> will produce a lot of output, especially on guest boot, but it should
>>> settle down as the drivers get loaded and direct access is used.  If it
>>> doesn't settle down and it's through the quirks, we may never be able to
>>> achieve good performance since we don't own the driver.  If it does
>>> settle down, you can use tools like perf and trace to analyze where time
>>> is being spent.  Thanks,
>>>
>>> Alex
>>>
>>>
>>
>> I have not tried passing-through nvidia gpu as i have read some scary
>> tales of nvidia gpus being problematic. Besides now it would be a little
>> bit pointless (unless its only for test) as i got more powerful GPU for
>> use in VM. I will definitely try figuring out where time is spent. Some
>> of your mentioned stuff is really all new to me and might take bit of
>> time to figure out.
> 
> IME, Nvidia actually works better than AMD.  The Nvidia quirks mostly
> come from the reverse engineering that the nouveau project has done
> while the AMD quirks come solely from my reverse engineering.  Nvidia is
> also interested in supporting the professional class cards in VMs
> whereas I haven't seen any interest in that from AMD.  Note that the
> professional class cards are the Quadro, GRID, and Tesla boards and
> they're supported in a secondary display configuration.  Using x-vga=on
> with consumer Geforce or Radeon boards is equally not supported by
> either vendor.
> 
>> Im not 100% sure what vga quirks are so please forgive for possibly
>> stupid question but are they enabled by x-vga=on? Something at the back
>> of my head is telling me that you are probably not using that flag since
>> your card is passed-through as >secondary< device. I also tried to
>> achieve same thing with no luck. What i tried was basically -vga std and
>> removing x-vga=on bit. Result was device manager indicating non-working
>> device. I believe error was 12 (not enough resources) or something very
>> similar. If you have ideas regarding this bit i would love to hear them.
>> In the meantime ill snoop around stuff you mentioned, thank you for hints.
> 
> Yes, secondary mode is to provide the guest with a -vga device in
> addition to the assigned GPU, and not use the x-vga=on option.  Whether
> this works often depends on the driver in the guest.
> 
> The quirks that I'm talking about are fixes for hardware backdoors.  For
> instance PCI config space is often mirrored in MMIO config space on
> graphics cards.  Running in a virtual machine requires that we emulate
> portions of config space, so we need to trap accesses to the MMIO
> mirror.  If the driver uses the mirror, or anything within PAGE_SIZE of
> the mirror, for performance critical access, the traps required for the
> quirk will seriously affect the performance.
> 
> The Q35 QEMU model could also have something to do with the performance
> issues you see.  There have been unsubstantiated claims that VMs are
> slower with Q35.  At the same time, it provides a topology that looks a
> lot more like a modern system than the default machine.  The numbers I
> reported were using the standard 440FX machine type.  This is another
> case where the driver in the guest may be more or less accommodating of
> what we expose in the VM, so YMMV with consumer cards and different
> drivers.  Thanks,
> 
> Alex
> 
> 

I see. From what you were saying i got an impression that trying 440fx
machine type would help. I actually managed to do it by simply defining
vm on libvirt. It passed through both GPU and USB ports just fine. On
boot VM properly switches to physical GPU and emulated one is disabled
by windows. So far so good. However there was no performance increase
whatsoever. At best it was just the same. I tried qemu with DEBUG_VFIO
enabled too. After OS is done booting log gets appended only with log
lines like these:
> vfio: vfio_bar_read(0000:00:13.2:BAR0+0x24, 4) = 0x4000
> vfio: vfio_bar_write(0000:00:13.2:BAR0+0x28, 0x3f, 4)
00:13.2 is USB port. I also tried running dx9 complex test and did not
see any more spam relating to GPU (06:00.0). I also checked vfio output
for config i posted earlier (with addition of hv-time) but result is
identical to libvirt 440fx test. Not sure what to make of this.

I also had not really successful attempt to enable hugepages but i think
it really should not be the case of 3d slowdown so i choose to ignore
this failure for now.
Tinkering around cpu pinning also resulted in few fps more or few less
in dx9 complex benchmark so really not significant to be the real
culprit here.