Re: Struggling with PCI-Passthrough

[Alexander Brychcy <bober@brychcy.net>]

Am 11.03.2014 11:49, schrieb Andrew Cooper:
> On 11/03/14 10:26, Alexander Brychcy wrote:
>> Am 10.03.2014 12:09, schrieb Andrew Cooper:
>>> On 10/03/14 09:59, Alexander Brychcy wrote:
>>>> Hi,
>>>>
>>>> I'm currently trying to create a Windows 7 HVM with GPU Passthrough.
>>>>
>>>> My setup is the following:
>>>> CPU:	Xeon E3-1230v3
>>>> Board:	Asus P9D-E/4L
>>>> RAM:	16GB Kingston from Board QVL
>>>> GPU:	Sapphire R9 270X 4GB
>>>>
>>>> The passthrough itself is working (Tested with Debian Wheezy dom0 on Xen
>>>> 4.1, then XenServer 6.2 and now Debian Jessie dom0 on Xen 4.3)
>>>>
>>>> Under Debian Jessie i was not able to passthrough the USB Controller,
>>>> although it appeard at xl pci-assignable-list and i cannot assign more
>>>> than 2GB of memory to the HVM.
>>>>
>>>> The overall problem is, that the HVM starts up and is running fine. But
>>>> after a unspecific amount of time the hole system freezes. The cursor in
>>>> dom0 still blinking, but no input possible - not even in xen serial
>>>> console. I had one run, without a freeze, but the GPU performance was
>>>> very poor, so i tried a manual reset. After the HVM reboot the
>>>> performance was as expected, but soon the system froze again.
>>>>
>>>> I'm not sure, if this problem is related to Xen, because i tried to
>>>> create the virtual machine with KVM and had the same issue. But i
>>>> switched to Xen to get the best performace.
>>>>
>>>> I will add my config and log files and hope you can help me to get this
>>>> setup fully working.
>>>>
>>>> Kind regards,
>>>> Alexander
>>> Looking at the serial log, it would appear that Xen has decided to
>>> schedule no vcpus whatsoever, and is completely idle.
>>>
>>> Looking at the 'Q' output, only two vcpus (one from each domain) are not
>>> in a blocked state, but no obvious reason for them to be blocked; they
>>> are not blocked waiting for events.
>>>
>>> Are you able to try this with a debug version of Xen?  Perhaps an
>>> assertion will be tripped which might give more of a clue.
>>>
>>> It is certainly interesting that the same behaviour exists with KVM,
>>> suggesting it is something to do with hardware interaction, but I can
>>> see no obvious reason why Xen is as idle as it is.
>>>
>>> ~Andrew
>>>
>> I will try everything to get this machine running, but i won't be able
>> test again before the weekend. The nullmodem cable was only borrowed and
>> it takes some time for my own to arrive.
>>
>> Which switches do I have to set during configure to get a debug version?
>>
>> I will start a new thread on xen-users when everything is set up. Just
>> clicked on the wrong mailto when starting this conversation.
> 
> I would keep this on Xen-devel, especially if building a debug version
> of xen.  My gut feel is that it isn't a xen-users@ problem.
> 
> Debug is not a configure option.  from the root of the Xen source tree,
> `make -C xen debug=y` (with appropriate -j and cross compile options for
> non-standard environments) should make a debug xen/xen.gz.
> 
> If this is the same source that your tools are built from, it is save to
> just swap out Xen and reboot, to save rebuilding and installing
> everything else.
> 
> ~Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 
It took some time, because by bachelor thesis is due next month and i
don't have a lot of spare time to troubleshoot this problem. I just
installed a fresh hdd, so i don't have to switch drive images back and
forth until it is fully working.
Status quo:
Xen 4.4.0 incl. debug=y
dom0: Debian 7.4 with vanilla kernel 3.13.6
domU #1: untouched Debian 7.4 startet by pygrub and kernel 3.2
domU #2: HVM Windows 7 with pci passthrough

As ever there are only problems with domU #2. The first one, that i
could not passthrough an USB controller resolved. No matter at which
port you plug an USB device on the mainboard, it is always assigned to
the XHCI controller. My workaround is to add another PCIe USB controller
for the vm, so i'm still able to have usb connection in dom0 for backup
or other purposes.
Tried to passthrough a NIC to verify if it is a bios related bug. This
works without any problems and nothing freezes at all.
So last but not least the VGA Passthrough. Issue is still present. HVM
boots and GPU is assigned and working. But after some time respectively
an application tries to access some kind of function the whole system
freezes.
To keep the log outputs small, this should be the most interesting lines:

dmesg:
...
(d3) Scan for VGA option rom
(d3) WARNING! Found unaligned PCI rom (vd=1013:00b8)
(d3) Running option rom at c000:0003
(XEN) stdvga.c:147:d3 entering stdvga and caching modes
(d3) Turning on vga text mode console
...

qemu-dm-win7x64-tv.log:
[00:05.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:05.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)
[00:06.0] xen_pt_pci_config_access_check: Error: Failed to access
register with invalid access size alignment. (addr: 0x0e, len: 4)

Nearly forgot the RAM problem. The maximum amount of RAM i can assign to
the HVM when i passthrough the GPU is 3GB. Haven't tried it yet but may
this patch still be usable with 4.4.0?
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram