Re: graphics card pci passthrough success report

[Vince Scott <c0d3x42@gmail.com>]

Lennert Buytenhek <buytenh <at> wantstofly.org> writes:

> 
> Hi!
> 
> Some time ago already, we got graphics card pci passthrough working
> on a friend's computer, an ASRock Z68 Extreme4 Gen3 based PC with
> an Intel i7-2600, running on a Linux 3.2 kernel, with two Gigabyte
> GV-R677SL-1GD (AMD Radeon HD 6770) graphics cards in CrossFireX mode.
> 
> To get things to work we had to:
> 
> - Boot with intel_iommu=yes.  This broke the onboard Marvell
>   88SE9120 controller (an AHCI controller) with IOMMU related
>   kernel errors, and we had to move some drives around.
> 
> - Blacklist the radeon audio driver so that the graphics cards
>   aren't in use on the host.
> 
> - Apply the patches at the end of this mail to kvm and SeaBIOS to
>   allow for more BAR space under 4G.  (The relevant BARs on the
>   graphics cards _are_ 64 bit BARs, but kvm seemed to turn those
>   into 32 bit BARs in the guest.)
> 
> - Apply the hacky patch at the end of this mail to SeaBIOS to
>   always skip initialising the Radeon's option ROMs, or the VM
>   would hang inside the Radeon option ROM if you boot the VM
>   without the default cirrus video.
> 
> With these patches, the Windows 7 guest seemed to work OK on basic
> testing, allowing playing of all 3d games we tried, including in
> CrossFireX mode.  We ran into the following issues:
> 
> - The AMD Catalyst graphics card drivers bluescreen the VM on win7
>   boot if you boot with a CPU family/model/stepping that it does not
>   like and one or more Radeon cards passed through to the VM.
>   (Setting the guest family/model/stepping to be equal to the host's
>   CPU's fixes the bluescreen and allows booting to complete.)
> 
> - AMD Catalyst 12.4 worked okay, but the 12.6 beta would bluescreen
>   the VM.
> 
> - Playing video in Flash (e.g. youtube) and XBMC gives a green
>   rectangle where the video is supposed to be.  This was fixable in
>   both cases by turning off hardware acceleration (in Flash) or
>   "dxva2" (in xbmc).  A problem with these symptoms is noted in the
>   AMD Catalyst driver release notes, but the same problem does not
>   seem to appear when running Windows 7 directly on the host.
>   (Could this be related to skipping the option ROM?)
> 
> We also tried passing through an NVidia graphics card (GTX 260)
> instead of the Radeons, but that would crash the VM the couple of
> times we tried, and we didn't look into this further.
> 
> The setup (display on the right assigned to the Linux host, display
> on the left assigned to the Windows guest): http://i.imgur.com/idtVH.jpg
> 
> 3DMark running on Windows host: http://i.imgur.com/nvbQr.jpg
> 
> 3DMark running on Windows as a VM: http://i.imgur.com/vuL7H.jpg
> 
> (Running with only one graphics card gives about half the 3dmark
> score, so CrossFireX does appear to be working in both cases.)
> 
> cheers,
> Lennert
> 
> KVM patch for more <4G PCI BAR space:
> 
> diff -ur qemu-kvm-1.0+noroms/hw/pc_piix.c ../qemu-kvm-1.0+noroms/hw/pc_piix.c
> --- qemu-kvm-1.0+noroms/hw/pc_piix.c    2011-12-04 11:38:06.000000000 +0100
> +++ ../qemu-kvm-1.0+noroms/hw/pc_piix.c 2012-06-09 16:24:14.186327304 +0200
> @@ -112,9 +112,9 @@
>           kvmclock_create();
>       }
> 
> -    if (ram_size >= 0xe0000000 ) {
> -        above_4g_mem_size = ram_size - 0xe0000000;
> -        below_4g_mem_size = 0xe0000000;
> +    if (ram_size >= 0xc0000000 ) {
> +        above_4g_mem_size = ram_size - 0xc0000000;
> +        below_4g_mem_size = 0xc0000000;
>       } else {
>           above_4g_mem_size = 0;
>           below_4g_mem_size = ram_size;
> 
> Corresponding SeaBIOS patch:
> 
> diff --git a/src/acpi-dsdt.dsl b/src/acpi-dsdt.dsl
> index 15299ee..31e7102 100644
> --- a/src/acpi-dsdt.dsl
> +++ b/src/acpi-dsdt.dsl
> @@ -170,10 +170,10 @@ DefinitionBlock (
>                       ,, , AddressRangeMemory, TypeStatic)
>                   DWordMemory (ResourceProducer, PosDecode, MinFixed, 
MaxFixed, NonCacheable, ReadWrite,
>                       0x00000000,         // Address Space Granularity
> -                    0xE0000000,         // Address Range Minimum
> +                    0xC0000000,         // Address Range Minimum
>                       0xFEBFFFFF,         // Address Range Maximum
>                       0x00000000,         // Address Translation Offset
> -                    0x1EC00000,         // Address Length
> +                    0x3EC00000,         // Address Length
>                       ,, PW32, AddressRangeMemory, TypeStatic)
>               })
>               Name (CR64, ResourceTemplate ()
> diff --git a/src/config.h b/src/config.h
> index 3a70867..b305d9a 100644
> --- a/src/config.h
> +++ b/src/config.h
> @@ -44,9 +44,9 @@
>   #define BUILD_SMM_INIT_ADDR       0x38000
>   #define BUILD_SMM_ADDR            0xa8000
>   #define BUILD_SMM_SIZE            0x8000
> -#define BUILD_MAX_HIGHMEM         0xe0000000
> +#define BUILD_MAX_HIGHMEM         0xc0000000
> 
> -#define BUILD_PCIMEM_START        0xe0000000
> +#define BUILD_PCIMEM_START        0xc0000000
>   #define BUILD_PCIMEM_END          0xfec00000    /* IOAPIC is mapped at */
>   #define BUILD_PCIMEM64_START      0x8000000000ULL
>   #define BUILD_PCIMEM64_END        0x10000000000ULL
> 
> SeaBIOS patch to always skip Radeon option ROM initialization:
> 
> diff --git a/src/optionroms.c b/src/optionroms.c
> index 00697b2..857956a 100644
> --- a/src/optionroms.c
> +++ b/src/optionroms.c
> @@ -440,6 +440,8 @@ vga_setup(void)
>           foreachpci(pci) {
>               if (!is_pci_vga(pci))
>                   continue;
> +            if (pci->vendor == 0x1002 && pci->device == 0x68ba)
> +                continue;
>               vgahook_setup(pci);
>               init_pcirom(pci, 1, NULL);
>               break;
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 


Hi There,

I just wanted to post a follow up to this to say I've also managed to get a 
reasonably stable system with a similar setup.

Hardware:
 - asrock q77m vPro (with vt-d enabled)
 - core i7 3770
 - sapphire radeon 7850 2gb

Software inside guest
 - windows 8 64bit - standard/home edition
 - radeon catalyst drivers 12-10_vista_win7_win8_64_dd_cc_whql_net4

host OS:
 - ubuntu 12.10, booted with intel_iommu=yes
 - blacklisted radeon
 - stock kvm, libvirt, qemu
 - seabios recompiled to latest and including the below optionroms.c patch 
(modified for me device id), not sure if the patch has 
made any difference though.
 - sapphire rom downloaded from http://www.techpowerup.com/vgabios/
 - libvirt / qemu config updated to include the rom in the PCI passthrough 
section:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/etc/libvirt/roms/Sapphire.HD7850.2048.120306_1.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
    </hostdev>


What i've observed:
 * my original configuration used the stock seabios and without the rom file 
directive, this would mostly startup windows 8 fine 
from a cold boot of the host.  restarting win8 would lead to the host locking
 up completely most of the time, nothing reported on the console 
for the kernel.
 * as above, but shutting down windows 8 and then restarting it would work for 
about the first three attempts, the fourth would 
cause the vm to goto 100% cpu, powering off the vm and restarting would 
lock the host

After applying the seabios change and including the rom file the system appears 
to be much more stable.  I have managed to 
shutdown and restart win8 a number of times without rebooting the host.  
Probably so far tried about 6 or 7 shutdown/starts in a 
row.

again with the seabios and rom file changes, restarting windows8 seems to cause 
lockups.

HTH,
-vince