Code 12 with VGA passthrough, even with qemu-traditional

Code 12 with VGA passthrough, even with qemu-traditional

[Peter Kay]

Hi,

Despite a lot of fiddling I am still unable to pass through either an Nvidia Quadro 6000 (actually a soft modded GTX480) or a Radeon HD6950 in Xen 4.3, 4.4 or -unstable. It fails in Windows 8 with code 12.

The hardware is capable - I have had the Radeon pass through in KVM using the 'nosnoop' patch (prevents PCIe device using NoSnoop) and supplying the BIOS in a file. That was with the same VM. I would much prefer to get Xen working, however.

Normal passthrough works fine, I.e. LAN. This is also running under Debian Wheezy

Any advice welcome. I've tried unpatched unstable, unpatched 4.4 and 4.3 patched with all the ATI/Nvidia and 4GB memory patches. No difference. I am definitely using xen-traditional here

Hardware : S3210SHLC motherboard with Q6700 CPU (I.e. pre SLAT/EPT). Graphics card must be on the southbridge 4x PCIe slot (all other slots have their speed restricted). Motherboard include an embedded G200e, used as the console.

The same behaviour occurs with the 3.2.0.4 kernel (xen as modules) and 3.16 (built in)

Config files follow :

Xl log (01:00.0 is Quadro 6000)

Waiting for domain Windows (domid 1) to die [pid 3889]
Domain 1 has shut down, reason code 0 0x0
Action for shutdown reason code 0 is destroy
Domain 1 needs to be cleaned up: destroying the domain
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0
Done. Exiting now

Qemu log
domid: 1
Strip off blktap sub-type prefix to /home/peter/windows8.img (drv 'aio')
Using file /home/peter/windows8.img in read-write mode
Strip off blktap sub-type prefix to /home/peter/Windows8Pro.iso (drv 'aio')
Using file /home/peter/Windows8Pro.iso in read-only mode
Watching /local/domain/0/device-model/1/logdirty/cmd
Watching /local/domain/0/device-model/1/command
Watching /local/domain/1/cpu
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = 28bcc5e1-baba-490d-a4f3-a85159eeeb92
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/1/xen_extended_power_mgmt): read error
xs_read(): vncpasswd get error. /vm/28bcc5e1-baba-490d-a4f3-a85159eeeb92/vncpasswd.
medium change watch on `hdb' (index: 1): aio:/home/peter/Windows8Pro.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
vcpu-set: watch node error.
[xenstore_process_vcpu_set_event]: /local/domain/1/cpu has no CPU!
xs_read(/local/domain/1/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/1/log-throttling'
medium change watch on `/local/domain/1/log-throttling' - unknown device, ignored
vga s->lfb_addr = f1000000 s->lfb_end = f1800000 
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
mapping vram to f1000000 - f1800000
vga s->lfb_addr = f1000000 s->lfb_end = f1800000 
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Disable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x02000000 base_addr=0xdc000000)
pt_register_regions: IO region registered (size=0x08000000 base_addr=0xd000000c)
pt_register_regions: IO region registered (size=0x04000000 base_addr=0xd800000c)
pt_register_regions: IO region registered (size=0x00000080 base_addr=0x00002001)
pt_register_regions: Expansion ROM registered (size=0x00080000 base_addr=0xde080000)
pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
IRQ type = INTx
generate a sci for PHP.
deassert due to disable GPE bit.
shutdown requested in cpu_handle_ioreq
Issued domain 1 poweroff

Xen dmesg
 __  __            _  _    _____  ___  
 \ \/ /___ _ __   | || |  |___ / / _ \ 
  \  // _ \ '_ \  | || |_   |_ \| | | |
  /  \  __/ | | | |__   _| ___) | |_| |
 /_/\_\___|_| |_|    |_|(_)____(_)___/ 
                                       
(XEN) Xen version 4.3.0 (root@syllopsium.com) (gcc (Debian 4.7.2-5) 4.7.2) debug=n Thu Aug 21 06:47:13 BST 2014
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 1.99-27+deb7u2
(XEN) Command line: placeholder com1=115200,8n1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009ec00 (usable)
(XEN)  000000000009ec00 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfbf0000 (usable)
(XEN)  00000000cfbf0000 - 00000000cfc96000 (ACPI NVS)
(XEN)  00000000cfc96000 - 00000000cfcfa000 (usable)
(XEN)  00000000cfcfa000 - 00000000cfd5f000 (reserved)
(XEN)  00000000cfd5f000 - 00000000cfd69000 (usable)
(XEN)  00000000cfd69000 - 00000000cfddf000 (ACPI NVS)
(XEN)  00000000cfddf000 - 00000000cfde4000 (usable)
(XEN)  00000000cfde4000 - 00000000cfdff000 (ACPI data)
(XEN)  00000000cfdff000 - 00000000cfe00000 (usable)
(XEN)  00000000cfe00000 - 00000000cff00000 (reserved)
(XEN)  00000000f0000000 - 00000000f4000000 (reserved)
(XEN)  00000000fff80000 - 00000000fff8c000 (reserved)
(XEN)  0000000100000000 - 0000000230000000 (usable)
(XEN) ACPI: RSDP 000F03C0, 0024 (r2 INTEL )
(XEN) ACPI: XSDT CFDFE120, 00B4 (r1 INTEL  S3200SHC        0 INTL  1000013)
(XEN) ACPI: SLIC CFDFC000, 0176 (r1 INTEL  S3200SHC        2 INTL  1000013)
(XEN) ACPI: FACP CFDFA000, 00F4 (r3 INTEL  S3200SHC        0 MSFT  1000013)
(XEN) ACPI: DSDT CFDF4000, 56BC (r1 INTEL  S3200SHC        0 MSFT  1000013)
(XEN) ACPI: FACS CFD69000, 0040
(XEN) ACPI: APIC CFDF3000, 0084 (r1 INTEL  S3200SHC        0 MSFT  1000013)
(XEN) ACPI: WDDT CFDF2000, 0040 (r1 INTEL  S3200SHC        0 MSFT  1000013)
(XEN) ACPI: MCFG CFDF1000, 003C (r1 INTEL  S3200SHC        0 MSFT  1000013)
(XEN) ACPI: HPET CFDF0000, 0038 (r1 INTEL  S3200SHC        1 MSFT  1000013)
(XEN) ACPI: SPCR CFDEF000, 0050 (r1 INTEL  S3200SHC        0 MSFT  1000013)
(XEN) ACPI: SSDT CFDEE000, 0175 (r1 INTEL   Cpu0Ist       10 MSFT  1000013)
(XEN) ACPI: SSDT CFDED000, 0175 (r1 INTEL   Cpu1Ist       10 MSFT  1000013)
(XEN) ACPI: SSDT CFDEC000, 0175 (r1 INTEL   Cpu2Ist       10 MSFT  1000013)
(XEN) ACPI: SSDT CFDEB000, 0175 (r1 INTEL   Cpu3Ist       10 MSFT  1000013)
(XEN) ACPI: SSDT CFDEA000, 01BC (r1 INTEL     CpuPm       10 MSFT  1000013)
(XEN) ACPI: HEST CFDE9000, 00A8 (r1 INTEL  S3200SHC        1 INTL        1)
(XEN) ACPI: BERT CFDE8000, 0030 (r1 INTEL  S3200SHC        1 INTL        1)
(XEN) ACPI: ERST CFDE7000, 0230 (r1 INTEL  S3200SHC        1 INTL        1)
(XEN) ACPI: EINJ CFDE6000, 0130 (r1 INTEL  S3200SHC        1 INTL        1)
(XEN) ACPI: DMAR CFDE5000, 0128 (r1 INTEL  S3200SHC        1 MSFT        0)
(XEN) ACPI: DMAR CFDE4000, 0128 (r1 INTEL  S3200SHC        1 MSFT        0)
(XEN) System RAM: 8188MB (8384520kB)
(XEN) Domain heap initialised
(XEN) Processor #0 6:15 APIC version 20
(XEN) Processor #2 6:15 APIC version 20
(XEN) Processor #1 6:15 APIC version 20
(XEN) Processor #3 6:15 APIC version 20
(XEN) IOAPIC[0]: apic_id 5, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2660.070 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation not enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) not detected
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1ea1000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000208000000->000000020c000000 (1908782 pages to be allocated)
(XEN)  Init. ramdisk: 0000000215525000->000000022ffffc00
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81ea1000
(XEN)  Init. ramdisk: ffffffff81ea1000->ffffffff9c97bc00
(XEN)  Phys-Mach map: ffffffff9c97c000->ffffffff9d901848
(XEN)  Start info:    ffffffff9d902000->ffffffff9d9024b4
(XEN)  Page tables:   ffffffff9d903000->ffffffff9d9f4000
(XEN)  Boot stack:    ffffffff9d9f4000->ffffffff9d9f5000
(XEN)  TOTAL:         ffffffff80000000->ffffffff9dc00000
(XEN)  ENTRY ADDRESS: ffffffff818d61f0
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
 (XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 268kB init memory.
(XEN) physdev.c:161: dom0: wrong map_pirq type 4



-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Code 12 with VGA passthrough, even with qemu-traditional

[Sander Eikelenboom]

Thursday, August 21, 2014, 11:14:30 AM, you wrote:

> Hi,
> Despite a lot of fiddling I am still unable to pass through either an Nvidia Quadro 6000 (actually a soft modded GTX480) or a Radeon HD6950 in Xen 4.3, 4.4 or -unstable. It fails in Windows 8 with code 12.
> The hardware is capable - I have had the Radeon pass through in KVM using the 'nosnoop' patch (prevents PCIe device using NoSnoop) and supplying the BIOS in a file. That was with the same VM. I would much prefer to get Xen working, however.
> Normal passthrough works fine, I.e. LAN. This is also running under Debian Wheezy
> Any advice welcome. I've tried unpatched unstable, unpatched 4.4 and 4.3 patched with all the ATI/Nvidia and 4GB memory patches. No difference. I am definitely using xen-traditional here
> Hardware : S3210SHLC motherboard with Q6700 CPU (I.e. pre SLAT/EPT). Graphics card must be on the southbridge 4x PCIe slot (all other slots have their speed restricted). Motherboard include an embedded G200e, used as the console.
> The same behaviour occurs with the 3.2.0.4 kernel (xen as modules) and 3.16 (built in)

Have you tried with a linux guest, although it's perhaps not you end goal .. 
if that works you can at least say you should have a fair change of getting 
things to work (and i think more verbose and easy to debug then windows and the 
windows radeon drivers).

I have a system (amd based) with radeon HD6950 that is able to be passed through 
to a linux HVM guest (although you only get output after the kernel loads the 
radeon KMS/DRM driver since it's secondary passthrough). This is _not_ the 
primary graphics card from the host (got another radeon for dom0).

There are still issues  with rebooting the vm etc. 
I'm using xen-unstable, linux kernel 3.16 and qemu-xen (not traditional).


--
Sander

Re: Code 12 with VGA passthrough, even with qemu-traditional

[Richie]

On 8/21/2014 6:01 PM, Sander Eikelenboom wrote:
> I have a system (amd based) with radeon HD6950 that is able to be 
> passed through to a linux HVM guest (although you only get output 
> after the kernel loads the radeon KMS/DRM driver since it's secondary 
> passthrough). This is _not_ the primary graphics card from the host 
> (got another radeon for dom0).

Just chiming in here with a side note.

I decided to test out KVM + vfio on 3.16 with acs + intel igd patches 
and passed through my secondary card.   It was successful and I had the 
Seabios screen show up on my secondary VGA card (GTX 660ti).   It 
maintained video throughout the boot process including the Windows 7 
installer.  The only catch really was that since the host was on qemu 
2.0.0 and did not have the -cpu kvm=off switch (to hide kvm) I had to 
use Nvidia driver 335.x (or earlier).  Basic 3D tests have proved 
sucessfull and pt continues to work after subsequent guest reboots.

Now to be clear, I am partial to Xen.   My thinking is that at some 
point (or perhaps even now) that using the qemu upstream device model 
with Xen will yield this type of capability.  I am not so familiar with 
vfio or even pt with KVM.  There just happened to be the right guide out 
there on the net that could fill in the blanks.

Re: Code 12 with VGA passthrough, even with qemu-traditional

[Peter Kay]

On 21 August 2014 23:42:13 BST, Richie <listmail@triad.rr.com> wrote:

On 8/21/2014 6:01 PM, Sander Eikelenboom wrote:

I have a system (amd based) with radeon HD6950 that is able to be 
passed through to a linux HVM guest (although you only get output 
after the kernel loads the radeon KMS/DRM driver since it's secondary


passthrough). This is _not_ the primary graphics card from the host 
(got another radeon for dom0).


Just chiming in here with a side note.

I decided to test out KVM + vfio on 3.16 with acs + intel igd patches 
and passed through my secondary card. It was successful

...snip...

Now to be clear, I am partial to Xen. My thinking is that at some 
point (or perhaps even now) that using the qemu upstream device model 
with Xen will yield this type of capability. 


Hopefully the VGA passthrough will improve on Xen (non VGA passthrough is fine). I'm curious too as to whether Xen could use VFIO at some point - there's been a lot of work to decouple it from KVM and remove assumptions about PC architecture (like Xen, KVM is busily supporting ARM).

Would it be possible or do licensing/coding issues apply?

Re: Code 12 with VGA passthrough, even with qemu-traditional

[Konrad Rzeszutek Wilk]

On Fri, Aug 22, 2014 at 12:25:08PM +0100, Peter Kay wrote:
> 
> 
> On 21 August 2014 23:42:13 BST, Richie <listmail@triad.rr.com> wrote:
> 
> On 8/21/2014 6:01 PM, Sander Eikelenboom wrote:
> 
> I have a system (amd based) with radeon HD6950 that is able to be 
> passed through to a linux HVM guest (although you only get output 
> after the kernel loads the radeon KMS/DRM driver since it's secondary
> 
> 
> passthrough). This is _not_ the primary graphics card from the host 
> (got another radeon for dom0).
> 
> 
> Just chiming in here with a side note.
> 
> I decided to test out KVM + vfio on 3.16 with acs + intel igd patches 
> and passed through my secondary card. It was successful
> 
> ...snip...
> 
> Now to be clear, I am partial to Xen. My thinking is that at some 
> point (or perhaps even now) that using the qemu upstream device model 
> with Xen will yield this type of capability. 
> 
> 
> Hopefully the VGA passthrough will improve on Xen (non VGA passthrough is fine). I'm curious too as to whether Xen could use VFIO at some point - there's been a lot of work to decouple it from KVM and remove assumptions about PC architecture (like Xen, KVM is busily supporting ARM).
> 
> Would it be possible or do licensing/coding issues apply?

No license issues. The issues that exists are just bugs.
There are some issues in QEMU that make passthrough with
radeon and intel cards a bit iffy.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Code 12 with VGA passthrough, even with qemu-traditional

[Peter Kay]

[-- Attachment #1.1: Type: text/plain, Size: 2323 bytes --]

Yep - I've now managed to get this working. The answer was simple - I
assumed it would work right from bootup but it needs the Nvidia Quadro
drivers installed. The built in Microsoft ones won't do, and hotplug does
not work - the device must be there from the start. It looks like Nouveau
doesn't work either - but I haven't rebuilt my Linux VM yet to confirm that
only works with the official Nvidia drivers too.

Also, using -current it does work fine using qemu-xen. I have a VM with 5GB
available running without an issue.

Thanks for the help, all. It was optimistic to hope hotplug would work, I'm
just happy it works at all


On 25 August 2014 14:26, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
wrote:

> On Fri, Aug 22, 2014 at 12:25:08PM +0100, Peter Kay wrote:
> >
> >
> > On 21 August 2014 23:42:13 BST, Richie <listmail@triad.rr.com> wrote:
> >
> > On 8/21/2014 6:01 PM, Sander Eikelenboom wrote:
> >
> > I have a system (amd based) with radeon HD6950 that is able to be
> > passed through to a linux HVM guest (although you only get output
> > after the kernel loads the radeon KMS/DRM driver since it's secondary
> >
> >
> > passthrough). This is _not_ the primary graphics card from the host
> > (got another radeon for dom0).
> >
> >
> > Just chiming in here with a side note.
> >
> > I decided to test out KVM + vfio on 3.16 with acs + intel igd patches
> > and passed through my secondary card. It was successful
> >
> > ...snip...
> >
> > Now to be clear, I am partial to Xen. My thinking is that at some
> > point (or perhaps even now) that using the qemu upstream device model
> > with Xen will yield this type of capability.
> >
> >
> > Hopefully the VGA passthrough will improve on Xen (non VGA passthrough
> is fine). I'm curious too as to whether Xen could use VFIO at some point -
> there's been a lot of work to decouple it from KVM and remove assumptions
> about PC architecture (like Xen, KVM is busily supporting ARM).
> >
> > Would it be possible or do licensing/coding issues apply?
>
> No license issues. The issues that exists are just bugs.
> There are some issues in QEMU that make passthrough with
> radeon and intel cards a bit iffy.
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 3407 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Code 12 with VGA passthrough, even with qemu-traditional

[Peter Kay]

On 21 August 2014 23:01:46 BST, Sander Eikelenboom <linux@eikelenboom.it> wrote:
>
>Thursday, August 21, 2014, 11:14:30 AM, you wrote:
>
>> Hi,
>> Despite a lot of fiddling I am still unable to pass through either an
>Nvidia Quadro 6000 (actually a soft modded GTX480) or a Radeon HD6950
...snippy...
>Have you tried with a linux guest, although it's perhaps not you end
>goal .. 
>if that works you can at least say you should have a fair change of
>getting 
>things to work (and i think more verbose and easy to debug then windows
>and the 
>windows radeon drivers).
>
>I have a system (amd based) with radeon HD6950 that is able to be
>passed through 
>to a linux HVM guest (although you only get output after the kernel
>loads the 
>radeon KMS/DRM driver since it's secondary passthrough). This is _not_
>the 
>primary graphics card from the host (got another radeon for dom0).
>
>There are still issues  with rebooting the vm etc. 
>I'm using xen-unstable, linux kernel 3.16 and qemu-xen (not
>traditional).
I do have a debian VM - it booted and assigned an IRQ etc when I hot attached the device but black screened the primary stdvga VNC display when I tried to poke X onto the Quadro. I've not had chance to get SSH/serial console going yet so I can see the logs. It worked fine under KVM with a 6950 and didn't need the NoSnoop patch.

Certainly I'm very willing to diagnose the problem in a Linux VM. Now I'm running a fully supported configuration I'm hopeful of getting this working.

Thanks for confirming that -traditional is not required for passthrough. That'll remove the need for me to much around with memory patches

PK
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.